1 *goredo* redo implementation on pure Go
3 OVERVIEW *goredo-overview*
5 This is pure Go implementation of DJB's redo (http://cr.yp.to/redo.html)
6 build system proposal. Originally it was just a rewrite of redo-c
7 (https://github.com/leahneukirchen/redo-c), but later most features of
8 apenwarr/redo (https://redo.readthedocs.io/en/latest/) were also
9 implemented. Why yet another implementation? It is feature full and has
10 better performance comparing to shell and Python implementation.
12 goredo is free software: see the file COPYING for copying conditions.
13 Home page: http://www.goredo.cypherpunks.ru/
15 INSTALL *goredo-install*
17 $ go get go.cypherpunks.ru/goredo
19 $ export PATH=`pwd`:$PATH
21 If you have problems with *.golang.org's unability to verify
22 authenticity of go.cypherpunks.ru TLS connection, then you can disable
23 their usage by setting GOPRIVATE=go.cypherpunks.ru. If you still have
24 problems with the authenticity on your side, then build it manually: >
26 $ git clone git://git.cypherpunks.ru/goredo.git
29 $ git clone git://git.cypherpunks.ru/gorecfile.git
30 $ ( cd gorecfile ; git tag -v v0.3.0 )
31 $ echo "replace go.cypherpunks.ru/recfile => `pwd`/gorecfile" >> go.mod
32 $ git clone git://git.cypherpunks.ru/gotai64n.git
33 $ ( cd gotai64n ; git tag -v v0.1.0 )
34 $ echo "replace go.cypherpunks.ru/tai64n => `pwd`/gotai64n" >> go.mod
37 $ export PATH=`pwd`:$PATH
41 * "all" target is default
42 * stdout is always captured, but no target is created if it was empty
43 * empty targets are considered always out of date
44 * .do's $3 is relative path to the file in same directory
45 * .do search goes up to / by default, but can be limited with either
46 $REDO_TOP_DIR environment variable, or by having .redo/top file in it
47 * target's completion messages are written after they finish
48 * executable .do is run as is, non-executable is run with /bin/sh -e[x]
49 * tracing (-x) can be done only for non-executable .do
51 FEATURES *goredo-features*
53 * explicit useful and convenient checks from apenwarr/redo:
54 * check that $1 was not touched during .do execution
55 * check that stdout and $3 are not written simultaneously
56 * check that generated target was not modified "externally" outside
57 the redo, preventing its overwriting, but continuing the build
58 * targets, dependency information and their directories are explicitly
59 synced (can be disabled, should work faster)
60 * file's change is detected by comparing its ctime and BLAKE2b hash
61 * files creation is umask-friendly (unlike mkstemp() used in redo-c)
62 * parallel build with jobs limit, optionally in infinite mode
63 * coloured messages (can be disabled)
64 * verbose debug messages, including out-of-date determination, PIDs,
65 lock and jobserver acquirings/releases
66 * displaying of each target's execution time
67 * each target's stderr can be prefixed with the PID
68 * optional statusline with currently running/waiting/done jobs
69 * target's stderr can be stored on the disk with TAI64N timestamp
70 prefixes for each line. To convert them to localtime you can use either
71 tai64nlocal utility from daemontools (http://cr.yp.to/daemontools.html)
72 or make a symlink, to use built-in slower decoder: >
73 $ ln -s goredo tai64nlocal
75 COMMANDS *goredo-commands*
77 * redo-ifchange, redo-ifcreate, redo-always
78 * redo -- same as redo-ifchange, but forcefully and sequentially run
80 * redo-log -- display TAI64N timestamped last stderr of the target
81 * redo-stamp -- record stamp dependency. Nothing more, dummy
82 * redo-cleanup -- removes either temporary, log files, or everything
84 * redo-whichdo -- .do search paths for specified target (similar to
86 $ redo-whichdo x/y/a.b.o
99 * redo-dot -- dependency DOT graph generator. For example to visualize
100 your dependencies with GraphViz: >
101 $ redo target [...] # to assure that **/.redo/*.dep are filled up
102 $ redo-dot target [...] > whatever.dot
103 $ dot -Tpng whatever.dot > whatever.png # possibly add -Gsplines=ortho
107 Hashing and stamping~
109 All targets are checksummed if their ctime differs from the previous
110 one. apenwarr/redo gives many reasons why every time checksumming is
111 bad, but in my opinion in practice all of them do not apply.
113 * Aggregate targets and willing to be out-of-date ones just must not
114 produce empty output files. apenwarr/*, redo-c and goredo
115 implementations consider non existing file as an out-of-date target
116 * If you really wish to produce an empty target file, just touch $3
118 DJB's proposal with both stdout and $3 gives that ability to control
119 your desired behaviour. Those who does not capture stdout -- failed.
120 Those who creates an empty file if no stdout was written -- failed.
122 redo is a tool to help people. Literally all targets can be safely
123 "redo-stamp < $3"-ed, reducing false positive out-of-dates. Of course,
124 with the correct stdout/$3 working and placing necessary results in $3,
125 instead of just silently feeding them in redo-stamp.
127 redo implementations are already automatically record -ifchange on .do
128 files and -ifcreate on non-existing .do files. So why they can not
129 record redo-stamp the same way implicitly? No, Zen of Python does not
130 applicable there, because -ifchange/-ifcreate contradict it already.
132 Modern cryptographic hash algorithms and CPUs are so fast, that even all
133 read and writes to or from hard drive arrays can be easily checksummed
134 and transparently compressed, as ZFS with LZ4 and Skein/BLAKE[23]
135 algorithms demonstrate us.
137 goredo includes redo-stamp, that really records the stamp in the .dep
138 file, but it does not play any role later. It is stayed just for
141 Removed .do can lead to permanent errors of its non existence~
143 That is true, because dependency on it was recorded previously. Is it
144 safe to assume that .do-less target now is an ordinary source-file? I
145 have no confidence in such behaviour. So it is user's decision how to
146 deal with it, probably it was just his inaccuracy mistake. If you really
147 want to get rid of that dependency knowledge for foo/bar target, then
148 just remove foo/.redo/bar.dep.
152 Dependency and build state is kept inside .redo subdirectory in each
153 directory related the build. Each corresponding target has its own,
154 recreated with every rebuild, .dep file. It is recfile
155 (https://www.gnu.org/software/recutils/), that could have various
156 dependency information (dep.rec with the schema included): >
158 Build: 80143f04-bfff-4673-950c-081d712f573d
165 Ctime: 1605721341.253305000
166 Hash: f4929732f96f11e6d4ebe94536b5edef426d00ed0146853e37a87f4295e18eda
171 Hash: 5bbdf635932cb16b9127e69b6f3872577efed338f0a4ab6f2c7ca3df6ce50cc9
175 Run any of the command above with the -help.
179 This program is free software: you can redistribute it and/or modify
180 it under the terms of the GNU General Public License as published by
181 the Free Software Foundation, version 3 of the License.
183 This program is distributed in the hope that it will be useful,
184 but WITHOUT ANY WARRANTY; without even the implied warranty of
185 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
186 GNU General Public License for more details.