*goredo* redo implementation on pure Go

OVERVIEW                                               *goredo-overview*

This is pure Go implementation of DJB's redo (http://cr.yp.to/redo.html)
build system proposal. Originally it was just a rewrite of redo-c
(https://github.com/leahneukirchen/redo-c), but later most features of
apenwarr/redo (https://redo.readthedocs.io/en/latest/) were also
implemented. Why yet another implementation? It is feature full and has
better performance comparing to shell and Python implementation.

It passes tests from dieweltistgarnichtso.net's redo-sh.tests and
implementation-neutral from apenwarr/redo.

goredo is free software: see the file COPYING for copying conditions.
Home page: http://www.goredo.cypherpunks.ru/

INSTALL                                                 *goredo-install*

Hopefully it should work on all POSIX systems.
 >
    $ go get go.cypherpunks.ru/goredo
    $ goredo -symlinks
    $ export PATH=`pwd`:$PATH

If you have problems with *.golang.org's inability to verify
authenticity of go.cypherpunks.ru TLS connection, then you can disable
their usage by setting GOPRIVATE=go.cypherpunks.ru. If you still have
problems with the authenticity on your side, then build it manually: >

    $ git clone git://git.cypherpunks.ru/goredo.git
    $ cd goredo
    $ git tag -v v0.8.0
    $ git clone git://git.cypherpunks.ru/gorecfile.git
    $ ( cd gorecfile ; git tag -v v0.4.0 )
    $ echo "replace go.cypherpunks.ru/recfile => `pwd`/gorecfile" >> go.mod
    $ git clone git://git.cypherpunks.ru/gotai64n.git
    $ ( cd gotai64n ; git tag -v v0.2.0 )
    $ echo "replace go.cypherpunks.ru/tai64n => `pwd`/gotai64n" >> go.mod
    $ go build
    $ ./goredo -symlinks
    $ export PATH=`pwd`:$PATH

NOTES                                                     *goredo-notes*

* "all" target is default
* stdout is always captured, but no target is created if it was empty
* empty targets are considered always out of date
* .do's $3 is relative path to the file in target directory
* .do search goes up to / by default, but can be limited with either
  $REDO_TOP_DIR environment variable, or by having .redo/top file in it
* target's completion messages are written after they finish
* executable .do is run as is, non-executable is run with /bin/sh -e[x]
* tracing (-x) can be obviously done only for non-executable .do
* parallizable build is done only during redo-ifchange for human
  convenience: you can globally enable REDO_JOBS, but still do for
  example: redo htmls infos index upload

FEATURES                                               *goredo-features*

* explicit useful and convenient checks from apenwarr/redo:
    * check that $1 was not touched during .do execution
    * check that stdout and $3 are not written simultaneously
    * check that generated target was not modified "externally" outside
      the redo, preventing its overwriting, but continuing the build
* targets, dependency information and their directories are explicitly
  synced (can be disabled, should work faster)
* file's change is detected by comparing its ctime and BLAKE2b hash
* files creation is umask-friendly (unlike mkstemp() used in redo-c)
* parallel build with jobs limit, optionally in infinite mode
* coloured messages (can be disabled)
* verbose debug messages, including out-of-date determination, PIDs,
  lock and jobserver acquirings/releases
* displaying of each target's execution time
* each target's stderr can be prefixed with the PID
* optional statusline with currently running/waiting/done jobs
* target's stderr can be stored on the disk with TAI64N timestamp
  prefixes for each line. To convert them to localtime you can use either
  tai64nlocal utility from daemontools (http://cr.yp.to/daemontools.html),
  or similar one: >
    $ go get go.cypherpunks.ru/tai64n/cmd/tai64nlocal

COMMANDS                                               *goredo-commands*

* redo-ifchange, redo-ifcreate, redo-always
* redo -- same as redo-ifchange, but forcefully and sequentially run
  specified targets
* redo-log -- display TAI64N timestamped last stderr of the target
* redo-stamp -- record stamp dependency. Nothing more, dummy
* redo-cleanup -- removes either temporary, log files, or everything
  related to goredo
* redo-whichdo -- .do search paths for specified target (similar to
  apenwarr/redo): >
    $ redo-whichdo x/y/a.b.o
    x/y/a.b.o.do
    x/y/default.b.o.do
    x/y/default.o.do
    x/y/default.do
    x/default.b.o.do
    x/default.o.do
    x/default.do
    default.b.o.do
    default.o.do
    default.do
    ../default.b.o.do
    ../default.o.do
    ../default.do
* redo-dot -- dependency DOT graph generator. For example to visualize
  your dependencies with GraphViz: >
    $ redo target [...] # to assure that **/.redo/*.dep are filled up
    $ redo-dot target [...] > whatever.dot
    $ dot -Tpng whatever.dot > whatever.png # possibly add -Gsplines=ortho

FAQ                                                         *goredo-faq*

    Hashing and stamping~

All targets are checksummed if their ctime differs from the previous
one. apenwarr/redo gives many reasons why every time checksumming is
bad, but in my opinion in practice all of them do not apply.

* Aggregate targets and willing to be out-of-date ones just must not
  produce empty output files. apenwarr/*, redo-c and goredo
  implementations consider non existing file as an out-of-date target
* If you really wish to produce an empty target file, just touch $3

DJB's proposal with both stdout and $3 gives that ability to control
your desired behaviour. Those who does not capture stdout -- failed.
Those who creates an empty file if no stdout was written -- failed.

redo is a tool to help people. Literally all targets can be safely
"redo-stamp < $3"-ed, reducing false positive out-of-dates. Of course,
with the correct stdout/$3 working and placing necessary results in $3,
instead of just silently feeding them in redo-stamp.

redo implementations are already automatically record -ifchange on .do
files and -ifcreate on non-existing .do files. So why they can not
record redo-stamp the same way implicitly? No, Zen of Python does not
applicable there, because -ifchange/-ifcreate contradict it already.

Modern cryptographic hash algorithms and CPUs are so fast, that even all
read and writes to or from hard drive arrays can be easily checksummed
and transparently compressed, as ZFS with LZ4 and Skein/BLAKE[23]
algorithms demonstrate us.

goredo includes redo-stamp, that really records the stamp in the .dep
file, but it does not play any role later. It is stayed just for
compatibility.

    Can removed .do lead to permanent errors of its non existence?~

Yes, because dependency on it was recorded previously. Is it safe to
assume that .do-less target now is an ordinary source-file? I have no
confidence in such behaviour. So it is user's decision how to deal with
it, probably it was just his inaccuracy mistake. If you really want to
get rid of that dependency knowledge for foo/bar target, then just
remove foo/.redo/bar.dep.

    Does redo-always always rebuilds target?~

goredo, together with apenwarr/redo, rebuilds target once per run.
Always rebuilds during every redo invocation, but only once during it
building. http://news.dieweltistgarnichtso.net/bin/redo-sh.html#why-built-twice
has other opinion, that is why its redo-sh.tests/always_rebuild_1 will
fail. Rebuilding of always-ed targets even during the same build process
ruins any redo's usability in practice.

For example if my .h file contains source code's version number, that is
git-describe's output and all my other files depends on that header,
then any redo-ifchange of .o will lead to git-describe execution, that
is rather heavy. Of course, because of either hashing or possible
redo-stamp-ing its dependants won't be rebuilt further, but build time
will be already ruined. If you need to rebuild TeX documents (case
mentioned in redo-sh's FAQ) until all references and numbers are ready,
then you must naturally expectedly explicitly use while cycle in your
.do, as apenwarr/redo already suggests.

STATE                                                     *goredo-state*

Dependency and build state is kept inside .redo subdirectory in each
directory related the build. Each corresponding target has its own,
recreated with every rebuild, .dep file. It is recfile
(https://www.gnu.org/software/recutils/), that could have various
dependency information (dep.rec with the schema included): >

    Build: 80143f04-bfff-4673-950c-081d712f573d

    Type: ifcreate
    Target: foo.o.do

    Type: ifchange
    Target: default.o.do
    Ctime: 1605721341.253305000
    Hash: f4929732f96f11e6d4ebe94536b5edef426d00ed0146853e37a87f4295e18eda

    Type: always

    Type: stamp
    Hash: 5bbdf635932cb16b9127e69b6f3872577efed338f0a4ab6f2c7ca3df6ce50cc9

USAGE                                                     *goredo-usage*

Run any of the command above with the -help.

LICENCE~

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, version 3 of the License.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

 vim: filetype=help