2 @unnumbered Integration with existing software
4 Here is some examples of how you can solve popular tasks with NNCP,
5 making them store-and-forward friendly.
11 * BitTorrent and huge files: BitTorrent.
13 * Multimedia streaming: Multimedia.
17 @section Integration with Postfix
19 This section is taken from @url{http://www.postfix.org/nncp_README.html,
20 Postfix and UUCP} manual and just replaces UUCP-related calls with NNCP
23 @strong{Setting up a Postfix Internet to NNCP gateway}
25 Here is how to set up a machine that sits on the Internet and that forwards
26 mail to a LAN that is connected via NNCP.
30 @item You need an @ref{nncp-mail} program that extracts the sender
31 address from mail that arrives via NNCP, and that feeds the mail into
32 the Postfix @command{sendmail} command.
34 @item Define a @command{pipe(8)} based mail delivery transport for
37 /usr/local/etc/postfix/master.cf:
38 nncp unix - n n - - pipe
39 flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient
42 This runs the @command{nncp-mail} command to place outgoing mail into
43 the NNCP queue after replacing @var{$nexthop} by the the receiving NNCP
44 node and after replacing @var{$recipient} by the recipients. The
45 @command{pipe(8)} delivery agent executes the @command{nncp-mail}
46 command without assistance from the shell, so there are no problems with
47 shell meta characters in command-line parameters.
49 @item Specify that mail for @emph{example.com}, should be delivered via
50 NNCP, to a host named @emph{nncp-host}:
53 /usr/local/etc/postfix/transport:
54 example.com nncp:nncp-host
55 .example.com nncp:nncp-host
58 See the @command{transport(5)} manual page for more details.
60 @item Execute the command @command{postmap /etc/postfix/transport}
61 whenever you change the @file{transport} file.
63 @item Enable @file{transport} table lookups:
66 /usr/local/etc/postfix/main.cf:
67 transport_maps = hash:$config_directory/transport
70 @item Add @emph{example.com} to the list of domains that your site is
71 willing to relay mail for.
74 /usr/local/etc/postfix/main.cf:
75 relay_domains = example.com ...other relay domains...
78 See the @option{relay_domains} configuration parameter description for
81 @item Execute the command @command{postfix reload} to make the changes
86 @strong{Setting up a Postfix LAN to NNCP gateway}
88 Here is how to relay mail from a LAN via NNCP to the Internet.
92 @item You need an @ref{nncp-mail} program that extracts the sender
93 address from mail that arrives via NNCP, and that feeds the mail into
94 the Postfix @command{sendmail} command.
96 @item Specify that all remote mail must be sent via the @command{nncp}
97 mail transport to your NNCP gateway host, say, @emph{nncp-gateway}:
100 /usr/local/etc/postfix/main.cf:
101 relayhost = nncp-gateway
102 default_transport = nncp
105 Postfix 2.0 and later also allows the following more succinct form:
108 /usr/local/etc/postfix/main.cf:
109 default_transport = nncp:nncp-gateway
112 @item Define a @command{pipe(8)} based message delivery transport for
113 mail delivery via NNCP:
116 /usr/local/etc/postfix/master.cf:
117 nncp unix - n n - - pipe
118 flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient
121 This runs the @command{nncp-mail} command to place outgoing mail into
122 the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or
123 whatever you specified) and the recipients before executing the command.
124 The @command{nncp-mail} command is executed without assistance from the
125 shell, so there are no problems with shell meta characters.
127 @item Execute the command @command{postfix reload} to make the changes
133 @section Integration with Web feeds
135 RSS and Atom feeds could be collected using
136 @url{https://github.com/wking/rss2email, rss2email} program. It
137 converts all incoming feed entries to email messages. Read about how to
138 integration @ref{Postfix} with email. @command{rss2email} could be run
139 in a cron, to collect feeds without any user interaction. Also this
140 program supports ETags and won't pollute the channel if remote server
143 After installing @command{rss2email}, create configuration file:
145 % r2e new rss-robot@address.com
147 and add feeds you want to retrieve:
149 % r2e add https://git.cypherpunks.ru/cgit.cgi/nncp.git/atom/?h=master
157 @section Integration with Web pages
159 Simple HTML web page can be downloaded very easily for sending and
160 viewing it offline after:
162 % wget http://www.example.com/page.html
165 But most web pages contain links to images, CSS and JavaScript files,
166 required for complete rendering.
167 @url{https://www.gnu.org/software/wget/, GNU Wget} supports that
168 documents parsing and understanding page dependencies. You can download
169 the whole page with dependencies the following way:
175 --restrict-file-names=ascii \
178 --execute robots=off \
179 http://www.example.com/page.html
181 that will create @file{www.example.com} directory with all files
182 necessary to view @file{page.html} web page. You can create single file
183 compressed tarball with that directory and send it to remote node:
185 % tar cf - www.example.com | xz -9 |
186 nncp-file - remote.node:www.example.com-page.tar.xz
189 But there are multi-paged articles, there are the whole interesting
190 sites you want to get in a single package. You can mirror the whole web
191 site by utilizing @command{wget}'s recursive feature:
197 --no-remove-listing \
200 http://www.example.com/
203 There is a standard for creating
204 @url{https://en.wikipedia.org/wiki/Web_ARChive, Web ARChives}:
205 @strong{WARC}. Fortunately again, @command{wget} supports it as an
209 --warc-file www.example_com-$(date '+%Y%M%d%H%m%S') \
210 --no-warc-compression \
213 http://www.example.com/
215 That command will create uncompressed @file{www.example_com-XXX.warc}
216 web archive. By default, WARCs are compressed using
217 @url{https://en.wikipedia.org/wiki/Gzip, gzip}, but, in example above,
218 we have disabled it to compress with stronger @command{xz}, before
219 sending via @command{nncp-file}.
221 There are plenty of software acting like HTTP proxy for your browser,
222 allowing to view that WARC files. However you can extract files from
223 that archive using @url{https://pypi.python.org/pypi/Warcat, warcat}
224 utility, producing usual directory hierarchy:
226 % python3 -m warcat extract \
227 www.example_com-XXX.warc \
228 --output-dir www.example.com-XXX \
233 @section BitTorrent and huge files
235 If dealing with @ref{Git}, @ref{Feeds, web feeds} and @ref{Multimedia,
236 multimedia} goes relatively fast, then BitTorrent and huge files
237 consumes much time. You can not wait for downloads finish, but want to
240 @url{http://aria2.github.io/, aria2} multi-protocol download utility
241 could be used for solving that issue conveniently. It supports HTTP,
242 HTTPS, FTP, SFTP and BitTorrent protocols, together with
243 @url{http://tools.ietf.org/html/rfc5854, Metalink} format. BitTorrent
244 support is fully-featured: UDP trackers, DHT, PEX, encryption, magnet
245 URIs, Web-seeding, selective downloads, LPD. @command{aria2} can
246 accelerate HTTP*/*FTP downloads by segmented multiple parallel
249 You can queue you files after they are completely downloaded:
251 % cat send-downloaded.sh
253 nncp-file -chunked $(( 1024 * 100 )) "$3" remote.node
256 --on-download-complete send-downloaded.sh \
257 http://example.org/file.iso \
258 http://example.org/file.iso.asc
262 @url{http://aria2.github.io/manual/en/html/aria2c.html#files, input file}
263 with the jobs you want to download:
266 http://www.nncpgo.org/download/nncp-0.11.tar.xz
268 http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig
271 --on-download-complete send-downloaded.sh \
274 and all that downloaded (@file{nncp.txz}, @file{nncp.txz.sig}) files
275 will be sent to @file{remote.node} when finished.
278 @section Integration with Git
280 @url{https://git-scm.com/, Git} version control system already has all
281 necessary tools for store-and-forward networking.
282 @url{https://git-scm.com/docs/git-bundle, git-bundle} command is
285 Use it to create bundles containing all required blobs/trees/commits and tags:
287 % git bundle create repo-initial.bundle master --tags --branches
288 % git tag -f last-bundle
289 % nncp-file repo-initial.bundle remote.node:repo-$(date % '+%Y%M%d%H%m%S').bundle
292 Do usual working with the Git: commit, add, branch, checkout, etc. When
293 you decide to queue your changes for sending, create diff-ed bundle and
296 % git bundle create repo-$(date '+%Y%M%d%H%m%S').bundle last-bundle..master
298 % git bundle create repo-$(date '+%Y%M%d').bundle --since=10.days master
301 Received bundle on remote machine acts like usual remote:
303 % git clone -b master repo-XXX.bundle
305 overwrite @file{repo.bundle} file with newer bundles you retrieve and
306 fetch all required branches and commits:
308 % git pull # assuming that origin remote points to repo.bundle
309 % git fetch repo.bundle master:localRef
310 % git ls-remote repo.bundle
313 Bundles are also useful when cloning huge repositories (like Linux has).
314 Git's native protocol does not support any kind of interrupted download
315 resuming, so you will start from the beginning if connection is lost.
316 Bundles, being an ordinary files, can be downloaded with native
317 HTTP/FTP/NNCP resuming capabilities. After you fetch repository via the
318 bundle, you can add an ordinary @file{git://} remote and fetch the
322 @section Integration with multimedia streaming
324 Many video and audio streams could be downloaded using
325 @url{http://yt-dl.org/, youtube-dl} program.
326 @url{https://rg3.github.io/youtube-dl/supportedsites.html, Look} how
327 many of them are supported, including @emph{Dailymotion}, @emph{Vimeo}
330 When you multimedia becomes an ordinary file, you can transfer it easily.
333 --exec 'nncp-file {} remote.node:' \
334 'https://www.youtube.com/watch?list=PLd2Cw8x5CytxPAEBwzilrhQUHt_UN10FJ'