* Web feeds: Feeds.
* Web pages: WARCs.
* BitTorrent and huge files: BitTorrent.
+* Downloading service: DownloadService.
* Git::
* Multimedia streaming: Multimedia.
@end menu
@node Postfix
@section Integration with Postfix
-This section is taken from @url{http://www.postfix.org/nncp_README.html,
+This section is taken from @url{http://www.postfix.org/UUCP_README.html,
Postfix and UUCP} manual and just replaces UUCP-related calls with NNCP
ones.
@itemize
-@item You need an @ref{nncp-mail} program that extracts the sender
+@item You need an @ref{nncp-exec} program that extracts the sender
address from mail that arrives via NNCP, and that feeds the mail into
the Postfix @command{sendmail} command.
@verbatim
/usr/local/etc/postfix/master.cf:
nncp unix - n n - - pipe
- flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient
+ flags=F user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
@end verbatim
-This runs the @command{nncp-mail} command to place outgoing mail into
+This runs the @command{nncp-exec} command to place outgoing mail into
the NNCP queue after replacing @var{$nexthop} by the the receiving NNCP
node and after replacing @var{$recipient} by the recipients. The
-@command{pipe(8)} delivery agent executes the @command{nncp-mail}
+@command{pipe(8)} delivery agent executes the @command{nncp-exec}
command without assistance from the shell, so there are no problems with
shell meta characters in command-line parameters.
@itemize
-@item You need an @ref{nncp-mail} program that extracts the sender
+@item You need an @ref{nncp-exec} program that extracts the sender
address from mail that arrives via NNCP, and that feeds the mail into
the Postfix @command{sendmail} command.
@verbatim
/usr/local/etc/postfix/master.cf:
nncp unix - n n - - pipe
- flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient
+ flags=F user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
@end verbatim
-This runs the @command{nncp-mail} command to place outgoing mail into
+This runs the @command{nncp-exec} command to place outgoing mail into
the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or
-whatever you specified) and the recipients before executing the command.
-The @command{nncp-mail} command is executed without assistance from the
-shell, so there are no problems with shell meta characters.
+whatever you specified) and the recipients before execution of the
+command. The @command{nncp-exec} command is executed without assistance
+from the shell, so there are no problems with shell meta characters.
@item Execute the command @command{postfix reload} to make the changes
effective.
After installing @command{rss2email}, create configuration file:
@verbatim
-% r2e new rss-robot@address.com
+$ r2e new rss-robot@address.com
@end verbatim
and add feeds you want to retrieve:
@verbatim
-% r2e add https://git.cypherpunks.ru/cgit.cgi/nncp.git/atom/?h=master
+$ r2e add https://git.cypherpunks.ru/cgit.cgi/nncp.git/atom/?h=master
@end verbatim
and run the process:
@verbatim
-% r2e run
+$ r2e run
@end verbatim
@node WARCs
Simple HTML web page can be downloaded very easily for sending and
viewing it offline after:
@verbatim
-% wget http://www.example.com/page.html
+$ wget http://www.example.com/page.html
@end verbatim
But most web pages contain links to images, CSS and JavaScript files,
documents parsing and understanding page dependencies. You can download
the whole page with dependencies the following way:
@verbatim
-% wget \
+$ wget \
--page-requisites \
--convert-links \
--adjust-extension \
necessary to view @file{page.html} web page. You can create single file
compressed tarball with that directory and send it to remote node:
@verbatim
-% tar cf - www.example.com | xz -9 |
+$ tar cf - www.example.com | xz -9 |
nncp-file - remote.node:www.example.com-page.tar.xz
@end verbatim
sites you want to get in a single package. You can mirror the whole web
site by utilizing @command{wget}'s recursive feature:
@verbatim
-% wget \
+$ wget \
--recursive \
--timestamping \
-l inf \
@strong{WARC}. Fortunately again, @command{wget} supports it as an
output format.
@verbatim
-% wget \
+$ wget \
--warc-file www.example_com-$(date '+%Y%M%d%H%m%S') \
--no-warc-compression \
--no-warc-keep-log \
that archive using @url{https://pypi.python.org/pypi/Warcat, warcat}
utility, producing usual directory hierarchy:
@verbatim
-% python3 -m warcat \
- extract www.example_com-XXX.warc \
+$ python3 -m warcat extract \
+ www.example_com-XXX.warc \
--output-dir www.example.com-XXX \
--progress
@end verbatim
accelerate HTTP*/*FTP downloads by segmented multiple parallel
connections.
-You can queue you files after they are completely downloaded:
-@verbatim
-% cat send-downloaded.sh
-#!/bin/sh
-nncp-file -chunked $(( 1024 * 100 )) "$3" remote.node
-
-% aria2c \
- --on-download-complete send-downloaded.sh \
- http://example.org/file.iso \
- http://example.org/file.iso.asc
-@end verbatim
+You can queue you files after they are completely downloaded.
+@file{aria2-downloaded.sh} contents:
+@verbatiminclude aria2-downloaded.sh
Also you can prepare
@url{http://aria2.github.io/manual/en/html/aria2c.html#files, input file}
with the jobs you want to download:
@verbatim
-% cat jobs
+$ cat jobs
http://www.nncpgo.org/download/nncp-0.11.tar.xz
out=nncp.txz
http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig
out=nncp.txz.sig
-% aria2c \
- --on-download-complete send-downloaded.sh \
+$ aria2c \
+ --on-download-complete aria2-downloaded.sh \
--input-file jobs
@end verbatim
and all that downloaded (@file{nncp.txz}, @file{nncp.txz.sig}) files
will be sent to @file{remote.node} when finished.
+@node DownloadService
+@section Downloading service
+
+Previous sections tell about manual downloading and sending results to
+remote node. But one wish to remotely initiate downloading. That can be
+easily solved with @ref{CfgExec, exec} handles.
+
+@verbatim
+exec: {
+ warcer: ["/bin/sh", "/path/to/warcer.sh"]
+ wgeter: ["/bin/sh", "/path/to/wgeter.sh"]
+ aria2c: [
+ "/usr/local/bin/aria2c",
+ "--on-download-complete", "aria2-downloaded.sh",
+ "--on-bt-download-complete", "aria2-downloaded.sh"
+ ]
+}
+@end verbatim
+
+@file{warcer.sh} contents:
+@verbatiminclude warcer.sh
+
+@file{wgeter.sh} contents:
+@verbatiminclude wgeter.sh
+
+Now you can queue that node to send you some website's page, file or
+BitTorrents:
+
+@verbatim
+$ echo http://www.nncpgo.org/Postfix.html |
+ nncp-exec remote.node warcer postfix-whole-page
+$ echo http://www.nncpgo.org/Postfix.html |
+ nncp-exec remote.node wgeter postfix-html-page
+$ echo \
+ http://www.nncpgo.org/download/nncp-0.11.tar.xz
+ http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig |
+ nncp-exec remote.node aria2c
+@end verbatim
+
@node Git
@section Integration with Git
Use it to create bundles containing all required blobs/trees/commits and tags:
@verbatim
-% git bundle create repo-initial.bundle master
-% git tag -f last-bundle master
-% nncp-file repo-initial.bundle remote.node:repo-$(date % '+%Y%M%d%H%m%S').bundle
+$ git bundle create repo-initial.bundle master --tags --branches
+$ git tag -f last-bundle
+$ nncp-file repo-initial.bundle remote.node:repo-$(date % '+%Y%M%d%H%m%S').bundle
@end verbatim
Do usual working with the Git: commit, add, branch, checkout, etc. When
you decide to queue your changes for sending, create diff-ed bundle and
transfer them:
@verbatim
-% git bundle create repo-$(date '+%Y%M%d%H%m%S').bundle last-bundle..master
+$ git bundle create repo-$(date '+%Y%M%d%H%m%S').bundle last-bundle..master
or maybe
-% git bundle create repo-$(date '+%Y%M%d').bundle --since=10.days master
+$ git bundle create repo-$(date '+%Y%M%d').bundle --since=10.days master
@end verbatim
Received bundle on remote machine acts like usual remote:
@verbatim
-% git clone -b master repo-XXX.bundle
+$ git clone -b master repo-XXX.bundle
@end verbatim
overwrite @file{repo.bundle} file with newer bundles you retrieve and
fetch all required branches and commits:
@verbatim
-% git pull # assuming that origin remote points to repo.bundle
-% git fetch repo.bundle master:localRef
-% git ls-remote repo.bundle
+$ git pull # assuming that origin remote points to repo.bundle
+$ git fetch repo.bundle master:localRef
+$ git ls-remote repo.bundle
@end verbatim
Bundles are also useful when cloning huge repositories (like Linux has).
bundle, you can add an ordinary @file{git://} remote and fetch the
difference.
+Also you can find the following exec-handler useful:
+@verbatiminclude git-bundler.sh
+And it allows you to request for bundles like that:
+@code{echo some-old-commit..master | nncp-exec REMOTE bundler REPONAME}.
+
@node Multimedia
@section Integration with multimedia streaming
When you multimedia becomes an ordinary file, you can transfer it easily.
@verbatim
-% youtube-dl \
+$ youtube-dl \
--exec 'nncp-file {} remote.node:' \
'https://www.youtube.com/watch?list=PLd2Cw8x5CytxPAEBwzilrhQUHt_UN10FJ'
@end verbatim