2 @unnumbered Integration with existing software
4 Here is some examples of how you can solve popular tasks with NNCP,
5 making them store-and-forward friendly.
8 * Index files for freqing: FreqIndex.
13 * BitTorrent and huge files: BitTorrent.
14 * Downloading service: DownloadService.
16 * Multimedia streaming: Multimedia.
20 @section Index files for freqing
22 In many cases you do not know exact files list on remote machine you
23 want to freq from. Because files can be updated there. It is useful to
24 run cron-ed job on it to create files listing you can freq and search
28 0 4 * * * cd /storage ; tmp=`mktemp` ; \
29 tree -f -h -N --du --timefmt \%Y-\%m-\%d |
30 zstdmt -19 > $tmp && chmod 644 $tmp && mv $tmp TREE.txt.zst ; \
31 tree -J -f --timefmt \%Y-\%m-\%d |
32 zstdmt -19 > $tmp && chmod 644 $tmp && mv $tmp TREE.json.zst
36 @section Integration with Postfix
38 This section is taken from @url{http://www.postfix.org/UUCP_README.html,
39 Postfix and UUCP} manual and just replaces UUCP-related calls with NNCP
42 @strong{Setting up a Postfix Internet to NNCP gateway}
44 Here is how to set up a machine that sits on the Internet and that forwards
45 mail to a LAN that is connected via NNCP.
49 @item You need an @ref{nncp-exec} program that extracts the sender
50 address from mail that arrives via NNCP, and that feeds the mail into
51 the Postfix @command{sendmail} command.
53 @item Define a @command{pipe(8)} based mail delivery transport for
56 /usr/local/etc/postfix/master.cf:
57 nncp unix - n n - - pipe
58 flags=Rqhu user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
61 This runs the @command{nncp-exec} command to place outgoing mail into
62 the NNCP queue after replacing @var{$nexthop} by the receiving NNCP
63 node and after replacing @var{$recipient} by the recipients. The
64 @command{pipe(8)} delivery agent executes the @command{nncp-exec}
65 command without assistance from the shell, so there are no problems with
66 shell meta characters in command-line parameters.
68 Pay attention to @code{flags}, containing @code{R}, telling Postfix to
69 include @code{Return-Path:} header. Otherwise that envelope sender
70 information may be lost. Possibly you will also need somehow to
71 preserve that header on the receiving side, because @command{sendmail}
72 command will replace it. For example you can rename it before feeding to
73 @command{sendmail} with
74 @code{reformail -R Return-Path: X-Original-Return-Path: | sendmail}, or
77 @verbatiminclude sendmail.sh
79 Also pay attention that @command{maildrop} does not like @code{From_}
80 mbox-style header, so you possibly want:
83 mailbox_command = reformail -f0 | maildrop -d $@{USER@}
86 @item Specify that mail for @emph{example.com}, should be delivered via
87 NNCP, to a host named @emph{nncp-host}:
90 /usr/local/etc/postfix/transport:
91 example.com nncp:nncp-host
92 .example.com nncp:nncp-host
95 See the @command{transport(5)} manual page for more details.
97 @item Execute the command @command{postmap /etc/postfix/transport}
98 whenever you change the @file{transport} file.
100 @item Enable @file{transport} table lookups:
103 /usr/local/etc/postfix/main.cf:
104 transport_maps = hash:$config_directory/transport
107 @item Add @emph{example.com} to the list of domains that your site is
108 willing to relay mail for.
111 /usr/local/etc/postfix/main.cf:
112 relay_domains = example.com ...other relay domains...
115 See the @option{relay_domains} configuration parameter description for
118 @item Execute the command @command{postfix reload} to make the changes
123 @strong{Setting up a Postfix LAN to NNCP gateway}
125 Here is how to relay mail from a LAN via NNCP to the Internet.
129 @item You need an @ref{nncp-exec} program that extracts the sender
130 address from mail that arrives via NNCP, and that feeds the mail into
131 the Postfix @command{sendmail} command.
133 @item Specify that all remote mail must be sent via the @command{nncp}
134 mail transport to your NNCP gateway host, say, @emph{nncp-gateway}:
137 /usr/local/etc/postfix/main.cf:
138 relayhost = nncp-gateway
139 default_transport = nncp
142 Postfix 2.0 and later also allows the following more succinct form:
145 /usr/local/etc/postfix/main.cf:
146 default_transport = nncp:nncp-gateway
149 @item Define a @command{pipe(8)} based message delivery transport for
150 mail delivery via NNCP:
153 /usr/local/etc/postfix/master.cf:
154 nncp unix - n n - - pipe
155 flags=Fqhu user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
158 This runs the @command{nncp-exec} command to place outgoing mail into
159 the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or
160 whatever you specified) and the recipients before execution of the
161 command. The @command{nncp-exec} command is executed without assistance
162 from the shell, so there are no problems with shell meta characters.
164 @item Execute the command @command{postfix reload} to make the changes
170 @section Integration with Exim
172 This section is unaltered copy-paste of
173 @url{https://changelog.complete.org/archives/10165-asynchronous-email-exim-over-nncp-or-uucp, Asynchronous Email: Exim over NNCP (or UUCP)}
174 article by John Goerzen, with his permission.
176 @strong{Sending from Exim to a smarthost}
178 One common use for async email is from a satellite system: one that
179 doesn't receive mail, or have local mailboxes, but just needs to get
180 email out to the Internet. This is a common situation even for
181 conventionally-connected systems; in Exim speak, this is a "satellite
182 system that routes mail via a smarthost". That is, every outbound
183 message goes to a specific target, which then is responsible for
184 eventual delivery (over the Internet, LAN, whatever).
186 This is fairly simple in Exim.
188 We actually have two choices for how to do this: bsmtp or rmail mode.
189 bsmtp (batch SMTP) is the more modern way, and is essentially a
190 derivative of SMTP that explicitly can be queued asynchronously.
191 Basically it's a set of SMTP commands that can be saved in a file. The
192 alternative is "rmail" (which is just an alias for sendmail these days),
193 where the data is piped to rmail/sendmail with the recipients given on
194 the command line. Both can work with Exim and NNCP, but because we're
195 doing shiny new things, we'll use bsmtp.
197 These instructions are loosely based on the
198 @url{https://people.debian.org/~jdg/bsmtp.html, Using outgoing BSMTP with Exim HOWTO}.
199 Some of these may assume Debianness in the configuration, but should be
200 easily enough extrapolated to other configs as well.
202 First, configure Exim to use satellite mode with minimal DNS lookups
203 (assuming that you may not have working DNS anyhow).
205 Then, in the Exim primary router section for smarthost
206 (@file{router/200_exim4-config_primary} in Debian split configurations),
207 just change @code{transport = remote_smtp_smarthost to transport = nncp}.
209 Now, define the NNCP transport. If you are on Debian, you might name this
210 @file{transports/40_exim4-config_local_nncp}:
214 debug_print = "T: nncp transport for $local_part@@$domain"
219 command = /usr/local/nncp/bin/nncp-exec -noprogress -quiet hostname_goes_here rsmtp
220 .ifdef REMOTE_SMTP_HEADERS_REWRITE
221 headers_rewrite = REMOTE_SMTP_HEADERS_REWRITE
223 .ifdef REMOTE_SMTP_RETURN_PATH
224 return_path = REMOTE_SMTP_RETURN_PATH
228 This is pretty straightforward. We pipe to @command{nncp-exec}, run it
229 as the nncp user. @command{nncp-exec} sends it to a target node and runs
230 whatever that node has called @command{rsmtp} (the command to receive
231 bsmtp data). When the target node processes the request, it will run the
232 configured command and pipe the data in to it.
234 @strong{More complicated: Routing to various NNCP nodes}
236 Perhaps you would like to be able to send mail directly to various NNCP
237 nodes. There are a lot of ways to do that.
239 Fundamentally, you will need a setup similar to the UUCP example in
240 @url{https://www.exim.org/exim-html-current/doc/html/spec_html/ch-the_manualroute_router.html,
241 Exim's manualroute manual}, which lets you define how to reach various
242 hosts via UUCP/NNCP. Perhaps you have a star topology (every NNCP node
243 exchanges email with a central hub). In the NNCP world, you have two
244 choices of how you do this. You could, at the Exim level, make the
245 central hub the smarthost for all the side nodes, and let it
246 redistribute mail. That would work, but requires decrypting messages at
247 the hub to let Exim process. The other alternative is to configure NNCP
248 to just send to the destinations via the central hub; that takes
249 advantage of onion routing and doesn't require any Exim processing at
250 the central hub at all.
252 @strong{Receiving mail from NNCP}
254 On the receiving side, first you need to configure NNCP to authorize the
255 execution of a mail program. In the section of your receiving host where
256 you set the permissions for the client, include something like this:
260 rsmtp: ["/usr/sbin/sendmail", "-bS"]
264 The -bS option is what tells Exim to receive BSMTP on @code{stdin}.
266 Now, you need to tell Exim that nncp is a trusted user (able to set From
267 headers arbitrarily). Assuming you are running NNCP as the nncp user,
268 then add @code{MAIN_TRUSTED_USERS = nncp} to a file such as
269 @file{/etc/exim4/conf.d/main/01_exim4-config_local-nncp}. That's it!
271 Some hosts, of course, both send and receive mail via NNCP and will need
272 configurations for both.
275 @section Integration with Web feeds
277 RSS and Atom feeds could be collected using
278 @url{https://github.com/wking/rss2email, rss2email} program. It converts
279 all incoming feed entries to email messages. Read about how to integrate
280 @ref{Postfix}/@ref{Exim} with email. @command{rss2email} could be run in
281 a cron, to collect feeds without any user interaction. Also this program
282 supports ETags and won't pollute the channel if remote server supports
285 After installing @command{rss2email}, create configuration file:
288 $ r2e new rss-robot@@address.com
291 and add feeds you want to retrieve:
294 $ r2e add http://www.git.cypherpunks.ru/?p=nncp.git;a=atom
304 @section Integration with Web pages
306 Simple HTML web page can be downloaded very easily for sending and
307 viewing it offline after:
310 $ wget http://www.example.com/page.html
313 But most web pages contain links to images, CSS and JavaScript files,
314 required for complete rendering.
315 @url{https://www.gnu.org/software/wget/, GNU Wget} supports that
316 documents parsing and understanding page dependencies. You can download
317 the whole page with dependencies the following way:
324 --restrict-file-names=ascii \
327 --execute robots=off \
328 http://www.example.com/page.html
331 that will create @file{www.example.com} directory with all files
332 necessary to view @file{page.html} web page. You can create single file
333 compressed tarball with that directory and send it to remote node:
336 $ tar cf - www.example.com | zstd |
337 nncp-file - remote.node:www.example.com-page.tar.zst
340 But there are multi-paged articles, there are the whole interesting
341 sites you want to get in a single package. You can mirror the whole web
342 site by utilizing @command{wget}'s recursive feature:
349 --no-remove-listing \
352 http://www.example.com/
355 There is a standard for creating
356 @url{https://en.wikipedia.org/wiki/Web_ARChive, Web ARChives}:
357 @strong{WARC}. Fortunately again, @command{wget} supports it as an
362 --warc-file www.example_com-$(date '+%Y%M%d%H%m%S') \
363 --no-warc-compression \
366 http://www.example.com/
369 That command will create uncompressed @file{www.example_com-XXX.warc}
370 web archive. By default, WARCs are compressed using
371 @url{https://en.wikipedia.org/wiki/Gzip, gzip}, but, in example above,
372 we have disabled it to compress with stronger and faster
373 @url{https://en.wikipedia.org/wiki/Zstd, zstd}, before sending via
376 There are plenty of software acting like HTTP proxy for your browser,
377 allowing to view that WARC files. However you can extract files from
378 that archive using @url{https://pypi.python.org/pypi/Warcat, warcat}
379 utility, producing usual directory hierarchy:
382 $ python3 -m warcat extract \
383 www.example_com-XXX.warc \
384 --output-dir www.example.com-XXX \
389 @section BitTorrent and huge files
391 If dealing with @ref{Git}, @ref{Feeds, web feeds} and @ref{Multimedia,
392 multimedia} goes relatively fast, then BitTorrent and huge files
393 consumes much time. You can not wait for downloads finish, but want to
396 @url{http://aria2.github.io/, aria2} multi-protocol download utility
397 could be used for solving that issue conveniently. It supports HTTP,
398 HTTPS, FTP, SFTP and BitTorrent protocols, together with
399 @url{http://tools.ietf.org/html/rfc5854, Metalink} format. BitTorrent
400 support is fully-featured: UDP trackers, DHT, PEX, encryption, magnet
401 URIs, Web-seeding, selective downloads, LPD. @command{aria2} can
402 accelerate HTTP*/*FTP downloads by segmented multiple parallel
405 You can queue you files after they are completely downloaded.
406 @file{aria2-downloaded.sh} contents:
408 @verbatiminclude aria2-downloaded.sh
411 @url{http://aria2.github.io/manual/en/html/aria2c.html#files, input file}
412 with the jobs you want to download:
416 http://www.nncpgo.org/download/nncp-0.11.tar.xz
418 http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig
421 --on-download-complete aria2-downloaded.sh \
425 and all that downloaded (@file{nncp.txz}, @file{nncp.txz.sig}) files
426 will be sent to @file{remote.node} when finished.
428 @node DownloadService
429 @section Downloading service
431 Previous sections tell about manual downloading and sending results to
432 remote node. But one wish to remotely initiate downloading. That can be
433 easily solved with @ref{CfgExec, exec} handles.
437 warcer: ["/bin/sh", "/path/to/warcer.sh"]
438 wgeter: ["/bin/sh", "/path/to/wgeter.sh"]
440 "/usr/local/bin/aria2c",
441 "--on-download-complete", "aria2-downloaded.sh",
442 "--on-bt-download-complete", "aria2-downloaded.sh"
447 @file{warcer.sh} contents:
449 @verbatiminclude warcer.sh
451 @file{wgeter.sh} contents:
453 @verbatiminclude wgeter.sh
455 Now you can queue that node to send you some website's page, file or
459 $ echo http://www.nncpgo.org/Postfix.html |
460 nncp-exec remote.node warcer postfix-whole-page
461 $ echo http://www.nncpgo.org/Postfix.html |
462 nncp-exec remote.node wgeter postfix-html-page
464 http://www.nncpgo.org/download/nncp-0.11.tar.xz
465 http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig |
466 nncp-exec remote.node aria2c
470 @section Integration with Git
472 @url{https://git-scm.com/, Git} version control system already has all
473 necessary tools for store-and-forward networking.
474 @url{https://git-scm.com/docs/git-bundle, git-bundle} command is
477 Use it to create bundles containing all required blobs/trees/commits and tags:
480 $ git bundle create repo-initial.bundle master --tags --branches
481 $ git tag -f last-bundle
482 $ nncp-file repo-initial.bundle remote.node:repo-$(date % '+%Y%M%d%H%m%S').bundle
485 Do usual working with the Git: commit, add, branch, checkout, etc. When
486 you decide to queue your changes for sending, create diff-ed bundle and
490 $ git bundle create repo-$(date '+%Y%M%d%H%m%S').bundle last-bundle..master
492 $ git bundle create repo-$(date '+%Y%M%d').bundle --since=10.days master
495 Received bundle on remote machine acts like usual remote:
498 $ git clone -b master repo-XXX.bundle
501 overwrite @file{repo.bundle} file with newer bundles you retrieve and
502 fetch all required branches and commits:
505 $ git pull # assuming that origin remote points to repo.bundle
506 $ git fetch repo.bundle master:localRef
507 $ git ls-remote repo.bundle
510 Bundles are also useful when cloning huge repositories (like Linux has).
511 Git's native protocol does not support any kind of interrupted download
512 resuming, so you will start from the beginning if connection is lost.
513 Bundles, being an ordinary files, can be downloaded with native
514 HTTP/FTP/NNCP resuming capabilities. After you fetch repository via the
515 bundle, you can add an ordinary @file{git://} remote and fetch the
518 Also you can find the following exec-handler useful:
520 @verbatiminclude git-bundler.sh
522 And it allows you to request for bundles like that:
523 @code{echo some-old-commit..master | nncp-exec REMOTE bundler REPONAME}.
526 @section Integration with multimedia streaming
528 Many video and audio streams could be downloaded using
529 @url{http://yt-dl.org/, youtube-dl} program.
530 @url{https://rg3.github.io/youtube-dl/supportedsites.html, Look} how
531 many of them are supported, including @emph{Dailymotion}, @emph{Vimeo}
534 When you multimedia becomes an ordinary file, you can transfer it easily.
538 --exec 'nncp-file @{@} remote.node:' \
539 'https://www.youtube.com/watch?list=PLd2Cw8x5CytxPAEBwzilrhQUHt_UN10FJ'