2 @unnumbered Integration with existing software
4 Here is some examples of how you can solve popular tasks with NNCP,
5 making them store-and-forward friendly.
8 * Index files for freqing: FreqIndex.
12 * BitTorrent and huge files: BitTorrent.
13 * Downloading service: DownloadService.
15 * Multimedia streaming: Multimedia.
19 @section Index files for freqing
21 In many cases you do not know exact files list on remote machine you
22 want to freq from. Because files can be updated there. It is useful to
23 run cron-ed job on it to create files listing you can freq and search
27 0 4 * * * cd /storage ; tmp=`mktemp` ; \
28 tree -f -h -N --du --timefmt \%Y-\%m-\%d |
29 zstdmt -19 > $tmp && chmod 644 $tmp && mv $tmp TREE.txt.zst ; \
30 tree -J -f --timefmt \%Y-\%m-\%d |
31 zstdmt -19 > $tmp && chmod 644 $tmp && mv $tmp TREE.json.zst
35 @section Integration with Postfix
37 This section is taken from @url{http://www.postfix.org/UUCP_README.html,
38 Postfix and UUCP} manual and just replaces UUCP-related calls with NNCP
41 @strong{Setting up a Postfix Internet to NNCP gateway}
43 Here is how to set up a machine that sits on the Internet and that forwards
44 mail to a LAN that is connected via NNCP.
48 @item You need an @ref{nncp-exec} program that extracts the sender
49 address from mail that arrives via NNCP, and that feeds the mail into
50 the Postfix @command{sendmail} command.
52 @item Define a @command{pipe(8)} based mail delivery transport for
55 /usr/local/etc/postfix/master.cf:
56 nncp unix - n n - - pipe
57 flags=F user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
60 This runs the @command{nncp-exec} command to place outgoing mail into
61 the NNCP queue after replacing @var{$nexthop} by the the receiving NNCP
62 node and after replacing @var{$recipient} by the recipients. The
63 @command{pipe(8)} delivery agent executes the @command{nncp-exec}
64 command without assistance from the shell, so there are no problems with
65 shell meta characters in command-line parameters.
67 @item Specify that mail for @emph{example.com}, should be delivered via
68 NNCP, to a host named @emph{nncp-host}:
71 /usr/local/etc/postfix/transport:
72 example.com nncp:nncp-host
73 .example.com nncp:nncp-host
76 See the @command{transport(5)} manual page for more details.
78 @item Execute the command @command{postmap /etc/postfix/transport}
79 whenever you change the @file{transport} file.
81 @item Enable @file{transport} table lookups:
84 /usr/local/etc/postfix/main.cf:
85 transport_maps = hash:$config_directory/transport
88 @item Add @emph{example.com} to the list of domains that your site is
89 willing to relay mail for.
92 /usr/local/etc/postfix/main.cf:
93 relay_domains = example.com ...other relay domains...
96 See the @option{relay_domains} configuration parameter description for
99 @item Execute the command @command{postfix reload} to make the changes
104 @strong{Setting up a Postfix LAN to NNCP gateway}
106 Here is how to relay mail from a LAN via NNCP to the Internet.
110 @item You need an @ref{nncp-exec} program that extracts the sender
111 address from mail that arrives via NNCP, and that feeds the mail into
112 the Postfix @command{sendmail} command.
114 @item Specify that all remote mail must be sent via the @command{nncp}
115 mail transport to your NNCP gateway host, say, @emph{nncp-gateway}:
118 /usr/local/etc/postfix/main.cf:
119 relayhost = nncp-gateway
120 default_transport = nncp
123 Postfix 2.0 and later also allows the following more succinct form:
126 /usr/local/etc/postfix/main.cf:
127 default_transport = nncp:nncp-gateway
130 @item Define a @command{pipe(8)} based message delivery transport for
131 mail delivery via NNCP:
134 /usr/local/etc/postfix/master.cf:
135 nncp unix - n n - - pipe
136 flags=F user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
139 This runs the @command{nncp-exec} command to place outgoing mail into
140 the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or
141 whatever you specified) and the recipients before execution of the
142 command. The @command{nncp-exec} command is executed without assistance
143 from the shell, so there are no problems with shell meta characters.
145 @item Execute the command @command{postfix reload} to make the changes
151 @section Integration with Web feeds
153 RSS and Atom feeds could be collected using
154 @url{https://github.com/wking/rss2email, rss2email} program. It
155 converts all incoming feed entries to email messages. Read about how to
156 integration @ref{Postfix} with email. @command{rss2email} could be run
157 in a cron, to collect feeds without any user interaction. Also this
158 program supports ETags and won't pollute the channel if remote server
161 After installing @command{rss2email}, create configuration file:
164 $ r2e new rss-robot@@address.com
167 and add feeds you want to retrieve:
170 $ r2e add https://git.cypherpunks.ru/cgit.cgi/nncp.git/atom/?h=master
180 @section Integration with Web pages
182 Simple HTML web page can be downloaded very easily for sending and
183 viewing it offline after:
186 $ wget http://www.example.com/page.html
189 But most web pages contain links to images, CSS and JavaScript files,
190 required for complete rendering.
191 @url{https://www.gnu.org/software/wget/, GNU Wget} supports that
192 documents parsing and understanding page dependencies. You can download
193 the whole page with dependencies the following way:
200 --restrict-file-names=ascii \
203 --execute robots=off \
204 http://www.example.com/page.html
207 that will create @file{www.example.com} directory with all files
208 necessary to view @file{page.html} web page. You can create single file
209 compressed tarball with that directory and send it to remote node:
212 $ tar cf - www.example.com | zstd |
213 nncp-file - remote.node:www.example.com-page.tar.zst
216 But there are multi-paged articles, there are the whole interesting
217 sites you want to get in a single package. You can mirror the whole web
218 site by utilizing @command{wget}'s recursive feature:
225 --no-remove-listing \
228 http://www.example.com/
231 There is a standard for creating
232 @url{https://en.wikipedia.org/wiki/Web_ARChive, Web ARChives}:
233 @strong{WARC}. Fortunately again, @command{wget} supports it as an
238 --warc-file www.example_com-$(date '+%Y%M%d%H%m%S') \
239 --no-warc-compression \
242 http://www.example.com/
245 That command will create uncompressed @file{www.example_com-XXX.warc}
246 web archive. By default, WARCs are compressed using
247 @url{https://en.wikipedia.org/wiki/Gzip, gzip}, but, in example above,
248 we have disabled it to compress with stronger and faster
249 @url{https://en.wikipedia.org/wiki/Zstd, zstd}, before sending via
252 There are plenty of software acting like HTTP proxy for your browser,
253 allowing to view that WARC files. However you can extract files from
254 that archive using @url{https://pypi.python.org/pypi/Warcat, warcat}
255 utility, producing usual directory hierarchy:
258 $ python3 -m warcat extract \
259 www.example_com-XXX.warc \
260 --output-dir www.example.com-XXX \
265 @section BitTorrent and huge files
267 If dealing with @ref{Git}, @ref{Feeds, web feeds} and @ref{Multimedia,
268 multimedia} goes relatively fast, then BitTorrent and huge files
269 consumes much time. You can not wait for downloads finish, but want to
272 @url{http://aria2.github.io/, aria2} multi-protocol download utility
273 could be used for solving that issue conveniently. It supports HTTP,
274 HTTPS, FTP, SFTP and BitTorrent protocols, together with
275 @url{http://tools.ietf.org/html/rfc5854, Metalink} format. BitTorrent
276 support is fully-featured: UDP trackers, DHT, PEX, encryption, magnet
277 URIs, Web-seeding, selective downloads, LPD. @command{aria2} can
278 accelerate HTTP*/*FTP downloads by segmented multiple parallel
281 You can queue you files after they are completely downloaded.
282 @file{aria2-downloaded.sh} contents:
284 @verbatiminclude aria2-downloaded.sh
287 @url{http://aria2.github.io/manual/en/html/aria2c.html#files, input file}
288 with the jobs you want to download:
292 http://www.nncpgo.org/download/nncp-0.11.tar.xz
294 http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig
297 --on-download-complete aria2-downloaded.sh \
301 and all that downloaded (@file{nncp.txz}, @file{nncp.txz.sig}) files
302 will be sent to @file{remote.node} when finished.
304 @node DownloadService
305 @section Downloading service
307 Previous sections tell about manual downloading and sending results to
308 remote node. But one wish to remotely initiate downloading. That can be
309 easily solved with @ref{CfgExec, exec} handles.
313 warcer: ["/bin/sh", "/path/to/warcer.sh"]
314 wgeter: ["/bin/sh", "/path/to/wgeter.sh"]
316 "/usr/local/bin/aria2c",
317 "--on-download-complete", "aria2-downloaded.sh",
318 "--on-bt-download-complete", "aria2-downloaded.sh"
323 @file{warcer.sh} contents:
325 @verbatiminclude warcer.sh
327 @file{wgeter.sh} contents:
329 @verbatiminclude wgeter.sh
331 Now you can queue that node to send you some website's page, file or
335 $ echo http://www.nncpgo.org/Postfix.html |
336 nncp-exec remote.node warcer postfix-whole-page
337 $ echo http://www.nncpgo.org/Postfix.html |
338 nncp-exec remote.node wgeter postfix-html-page
340 http://www.nncpgo.org/download/nncp-0.11.tar.xz
341 http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig |
342 nncp-exec remote.node aria2c
346 @section Integration with Git
348 @url{https://git-scm.com/, Git} version control system already has all
349 necessary tools for store-and-forward networking.
350 @url{https://git-scm.com/docs/git-bundle, git-bundle} command is
353 Use it to create bundles containing all required blobs/trees/commits and tags:
356 $ git bundle create repo-initial.bundle master --tags --branches
357 $ git tag -f last-bundle
358 $ nncp-file repo-initial.bundle remote.node:repo-$(date % '+%Y%M%d%H%m%S').bundle
361 Do usual working with the Git: commit, add, branch, checkout, etc. When
362 you decide to queue your changes for sending, create diff-ed bundle and
366 $ git bundle create repo-$(date '+%Y%M%d%H%m%S').bundle last-bundle..master
368 $ git bundle create repo-$(date '+%Y%M%d').bundle --since=10.days master
371 Received bundle on remote machine acts like usual remote:
374 $ git clone -b master repo-XXX.bundle
377 overwrite @file{repo.bundle} file with newer bundles you retrieve and
378 fetch all required branches and commits:
381 $ git pull # assuming that origin remote points to repo.bundle
382 $ git fetch repo.bundle master:localRef
383 $ git ls-remote repo.bundle
386 Bundles are also useful when cloning huge repositories (like Linux has).
387 Git's native protocol does not support any kind of interrupted download
388 resuming, so you will start from the beginning if connection is lost.
389 Bundles, being an ordinary files, can be downloaded with native
390 HTTP/FTP/NNCP resuming capabilities. After you fetch repository via the
391 bundle, you can add an ordinary @file{git://} remote and fetch the
394 Also you can find the following exec-handler useful:
396 @verbatiminclude git-bundler.sh
398 And it allows you to request for bundles like that:
399 @code{echo some-old-commit..master | nncp-exec REMOTE bundler REPONAME}.
402 @section Integration with multimedia streaming
404 Many video and audio streams could be downloaded using
405 @url{http://yt-dl.org/, youtube-dl} program.
406 @url{https://rg3.github.io/youtube-dl/supportedsites.html, Look} how
407 many of them are supported, including @emph{Dailymotion}, @emph{Vimeo}
410 When you multimedia becomes an ordinary file, you can transfer it easily.
414 --exec 'nncp-file @{@} remote.node:' \
415 'https://www.youtube.com/watch?list=PLd2Cw8x5CytxPAEBwzilrhQUHt_UN10FJ'