* Web feeds: Feeds.
* Web pages: WARCs.
* BitTorrent and huge files: BitTorrent.
+* Downloading service: DownloadService.
* Git::
* Multimedia streaming: Multimedia.
@end menu
@itemize
-@item You need an @ref{nncp-mail} program that extracts the sender
+@item You need an @ref{nncp-exec} program that extracts the sender
address from mail that arrives via NNCP, and that feeds the mail into
the Postfix @command{sendmail} command.
@verbatim
/usr/local/etc/postfix/master.cf:
nncp unix - n n - - pipe
- flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient
+ flags=F user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
@end verbatim
-This runs the @command{nncp-mail} command to place outgoing mail into
+This runs the @command{nncp-exec} command to place outgoing mail into
the NNCP queue after replacing @var{$nexthop} by the the receiving NNCP
node and after replacing @var{$recipient} by the recipients. The
-@command{pipe(8)} delivery agent executes the @command{nncp-mail}
+@command{pipe(8)} delivery agent executes the @command{nncp-exec}
command without assistance from the shell, so there are no problems with
shell meta characters in command-line parameters.
@itemize
-@item You need an @ref{nncp-mail} program that extracts the sender
+@item You need an @ref{nncp-exec} program that extracts the sender
address from mail that arrives via NNCP, and that feeds the mail into
the Postfix @command{sendmail} command.
@verbatim
/usr/local/etc/postfix/master.cf:
nncp unix - n n - - pipe
- flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient
+ flags=F user=nncp argv=nncp-exec -quiet $nexthop sendmail $recipient
@end verbatim
-This runs the @command{nncp-mail} command to place outgoing mail into
+This runs the @command{nncp-exec} command to place outgoing mail into
the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or
whatever you specified) and the recipients before executing the command.
-The @command{nncp-mail} command is executed without assistance from the
+The @command{nncp-exec} command is executed without assistance from the
shell, so there are no problems with shell meta characters.
@item Execute the command @command{postfix reload} to make the changes
@verbatim
% cat send-downloaded.sh
#!/bin/sh
-nncp-file -chunked $(( 1024 * 100 )) "$3" remote.node
+
+if [ "$2" -eq 0 ]; then
+ # This could be downloaded .torrent file itself
+ exit 0
+fi
+
+if [ "$2" -gt 1 ]; then
+ # This is directory downloaded with BitTorrent
+ wholedir="$(dirname "$3")"
+ name=$(basename "$wholedir")
+ cd "$wholedir"/..
+ tartmp=$(mktemp ./finished.XXXXXX)
+ tar cf $tartmp "$name"
+ nncp-file -chunked $(( 1024 * 100 )) $tartmp remote:"$name".tar
+ rm $tartmp
+else
+ nncp-file -chunked $(( 1024 * 100 )) "$3" remote:
+fi
% aria2c \
--on-download-complete send-downloaded.sh \
http://example.org/file.iso \
http://example.org/file.iso.asc
+% aria2c \
+ --on-bt-download-complete send-downloaded.sh \
+ http://example.org/file.torrent
@end verbatim
Also you can prepare
and all that downloaded (@file{nncp.txz}, @file{nncp.txz.sig}) files
will be sent to @file{remote.node} when finished.
+@node DownloadService
+@section Downloading service
+
+Previous sections tell about manual downloading and sending results to
+remote node. But one wish to remotely initiate downloading. That can be
+easily solved with @ref{CfgExec, exec} handles.
+
+@verbatim
+exec:
+ warcer: ["/bin/sh", "/path/to/warcer.sh"]
+ wgeter: ["/bin/sh", "/path/to/wgeter.sh"]
+ aria2c: [
+ "/usr/local/bin/aria2c",
+ "--on-download-complete", "send-downloaded.sh",
+ "--on-bt-download-complete", "send-downloaded.sh"
+ ]
+@end verbatim
+
+@file{warcer.sh} contents:
+
+@verbatim
+#!/bin/sh -ex
+
+user_agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27"
+
+name="$1"
+read cmdline
+
+tmp=$(mktemp -d)
+cd $tmp
+warc_name=$name-$(date '+%Y%M%d%H%m%S')
+wget \
+ --page-requisites \
+ --convert-links \
+ --adjust-extension \
+ --restrict-file-names=ascii \
+ --span-hosts \
+ --random-wait \
+ --execute robots=off \
+ --user-agent "$user_agent" \
+ --reject '*.woff*,*.ttf,*.eot,*.js' \
+ --tries 10 \
+ --warc-file $warc_name \
+ --no-warc-compression \
+ --no-warc-keep-log \
+ $cmdline || :
+xz -9 "$warc_name".warc
+nncp-file -nice $NNCP_NICE "$warc_name".warc.xz $NNCP_SENDER:
+rm -r $tmp
+@end verbatim
+
+@file{wgeter.sh} contents:
+
+@verbatim
+#!/bin/sh -ex
+
+name="$1"
+read cmdline
+tmp=$(mktemp)
+wget --output-document=$tmp $cmdline
+xz -9 $tmp
+nncp-file -nice $NNCP_NICE $tmp.xz $NNCP_SENDER:$name.xz
+rm $tmp.xz
+@end verbatim
+
+Now you can queue that node to send you some website's page, file or
+BitTorrents:
+
+@verbatim
+% echo http://www.nncpgo.org/Postfix.html |
+ nncp-exec remote.node warcer postfix-whole-page
+% echo http://www.nncpgo.org/Postfix.html |
+ nncp-exec remote.node wgeter postfix-html-page
+% echo \
+ http://www.nncpgo.org/download/nncp-0.11.tar.xz
+ http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig |
+ nncp-exec remote.node aria2c
+@end verbatim
+
@node Git
@section Integration with Git