From e3f150979baac5165e81e418baacb07ace950175 Mon Sep 17 00:00:00 2001 From: Sergey Matveev Date: Sun, 10 Sep 2017 18:13:24 +0300 Subject: [PATCH] Various integration examples --- doc/about.ru.texi | 6 +- doc/about.texi | 2 +- doc/index.texi | 4 +- doc/integration.texi | 335 +++++++++++++++++++++++++++++++++++++++++++ doc/postfix.texi | 115 --------------- 5 files changed, 341 insertions(+), 121 deletions(-) create mode 100644 doc/integration.texi delete mode 100644 doc/postfix.texi diff --git a/doc/about.ru.texi b/doc/about.ru.texi index e9a2dce..c285e16 100644 --- a/doc/about.ru.texi +++ b/doc/about.ru.texi @@ -34,9 +34,9 @@ NNCP это @url{https://www.gnu.org/philosophy/pragmatic.ru.html, программное обеспечение}: лицензировано под условиями @url{https://www.gnu.org/licenses/gpl-3.0.ru.html, GNU GPLv3+}. Оно должно работать на всех @url{https://ru.wikipedia.org/wiki/POSIX, -POSIX}-совместимых системах. Лёгкая интеграция с существующими -@url{https://ru.wikipedia.org/wiki/SMTP, SMTP} серверами. Единственный -конфигурационный @url{http://yaml.org/, YAML} файл. +POSIX}-совместимых системах. Лёгкая @ref{Integration, интеграция} с +существующими @url{https://ru.wikipedia.org/wiki/SMTP, SMTP} серверами. +Единственный конфигурационный @url{http://yaml.org/, YAML} файл. Зачем создавать ещё одно решение с принципом сохранить-и-переслать когда уже существуют UUCP, FTN и даже SMTP? Посмотрите @ref{Сравнение, сравнение}! diff --git a/doc/about.texi b/doc/about.texi index eb306ad..cfd5321 100644 --- a/doc/about.texi +++ b/doc/about.texi @@ -27,7 +27,7 @@ NNCP is @url{https://www.gnu.org/philosophy/pragmatic.html, copylefted} @url{https://www.gnu.org/philosophy/free-sw.html, free software} licenced under @url{https://www.gnu.org/licenses/gpl-3.0.html, GNU GPLv3+}. It should work on all @url{https://en.wikipedia.org/wiki/POSIX, -POSIX}-compatible systems. Easy integration with existing +POSIX}-compatible systems. Easy @ref{Integration, integration} with existing @url{https://en.wikipedia.org/wiki/SMTP, SMTP} servers. Single @url{http://yaml.org/, YAML} configuration file. diff --git a/doc/index.texi b/doc/index.texi index 87c008e..4ce8613 100644 --- a/doc/index.texi +++ b/doc/index.texi @@ -33,7 +33,7 @@ A copy of the license is included in the section entitled "Copying conditions". * Installation:: * Configuration:: * Call configuration: Call. -* Integration with Postfix: Postfix. +* Integration:: * Commands:: * Niceness:: * Chunked files: Chunked. @@ -55,7 +55,7 @@ A copy of the license is included in the section entitled "Copying conditions". @include install.texi @include cfg.texi @include call.texi -@include postfix.texi +@include integration.texi @include cmds.texi @include niceness.texi @include chunked.texi diff --git a/doc/integration.texi b/doc/integration.texi new file mode 100644 index 0000000..d53423a --- /dev/null +++ b/doc/integration.texi @@ -0,0 +1,335 @@ +@node Integration +@unnumbered Integration with existing software + +Here is some examples of how you can solve popular tasks with NNCP, +making them store-and-forward friendly. + +@menu +* Postfix:: +* Web feeds: Feeds. +* Web pages: WARCs. +* BitTorrent and huge files: BitTorrent. +* Git:: +* Multimedia streaming: Multimedia. +@end menu + +@node Postfix +@section Integration with Postfix + +This section is taken from @url{http://www.postfix.org/nncp_README.html, +Postfix and UUCP} manual and just replaces UUCP-related calls with NNCP +ones. + +@strong{Setting up a Postfix Internet to NNCP gateway} + +Here is how to set up a machine that sits on the Internet and that forwards +mail to a LAN that is connected via NNCP. + +@itemize + +@item You need an @ref{nncp-mail} program that extracts the sender +address from mail that arrives via NNCP, and that feeds the mail into +the Postfix @command{sendmail} command. + +@item Define a @command{pipe(8)} based mail delivery transport for +delivery via NNCP: +@verbatim +/usr/local/etc/postfix/master.cf: +nncp unix - n n - - pipe + flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient +@end verbatim + +This runs the @command{nncp-mail} command to place outgoing mail into +the NNCP queue after replacing @var{$nexthop} by the the receiving NNCP +node and after replacing @var{$recipient} by the recipients. The +@command{pipe(8)} delivery agent executes the @command{nncp-mail} +command without assistance from the shell, so there are no problems with +shell meta characters in command-line parameters. + +@item Specify that mail for @emph{example.com}, should be delivered via +NNCP, to a host named @emph{nncp-host}: + +@verbatim +/usr/local/etc/postfix/transport: + example.com nncp:nncp-host + .example.com nncp:nncp-host +@end verbatim + +See the @command{transport(5)} manual page for more details. + +@item Execute the command @command{postmap /etc/postfix/transport} +whenever you change the @file{transport} file. + +@item Enable @file{transport} table lookups: + +@verbatim +/usr/local/etc/postfix/main.cf: + transport_maps = hash:$config_directory/transport +@end verbatim + +@item Add @emph{example.com} to the list of domains that your site is +willing to relay mail for. + +@verbatim +/usr/local/etc/postfix/main.cf: + relay_domains = example.com ...other relay domains... +@end verbatim + +See the @option{relay_domains} configuration parameter description for +details. + +@item Execute the command @command{postfix reload} to make the changes +effective. + +@end itemize + +@strong{Setting up a Postfix LAN to NNCP gateway} + +Here is how to relay mail from a LAN via NNCP to the Internet. + +@itemize + +@item You need an @ref{nncp-mail} program that extracts the sender +address from mail that arrives via NNCP, and that feeds the mail into +the Postfix @command{sendmail} command. + +@item Specify that all remote mail must be sent via the @command{nncp} +mail transport to your NNCP gateway host, say, @emph{nncp-gateway}: + +@verbatim +/usr/local/etc/postfix/main.cf: + relayhost = nncp-gateway + default_transport = nncp +@end verbatim + +Postfix 2.0 and later also allows the following more succinct form: + +@verbatim +/usr/local/etc/postfix/main.cf: + default_transport = nncp:nncp-gateway +@end verbatim + +@item Define a @command{pipe(8)} based message delivery transport for +mail delivery via NNCP: + +@verbatim +/usr/local/etc/postfix/master.cf: +nncp unix - n n - - pipe + flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient +@end verbatim + +This runs the @command{nncp-mail} command to place outgoing mail into +the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or +whatever you specified) and the recipients before executing the command. +The @command{nncp-mail} command is executed without assistance from the +shell, so there are no problems with shell meta characters. + +@item Execute the command @command{postfix reload} to make the changes +effective. + +@end itemize + +@node Feeds +@section Integration with Web feeds + +RSS and Atom feeds could be collected using +@url{https://github.com/wking/rss2email, rss2email} program. It +converts all incoming feed entries to email messages. Read about how to +integration @ref{Postfix} with email. @command{rss2email} could be run +in a cron, to collect feeds without any user interaction. Also this +program supports ETags and won't pollute the channel if remote server +supports them too. + +After installing @command{rss2email}, create configuration file: +@verbatim +% r2e new rss-robot@address.com +@end verbatim +and add feeds you want to retrieve: +@verbatim +% r2e add https://git.cypherpunks.ru/cgit.cgi/nncp.git/atom/?h=master +@end verbatim +and run the process: +@verbatim +% r2e run +@end verbatim + +@node WARCs +@section Integration with Web pages + +Simple HTML web page can be downloaded very easily for sending and +viewing it offline after: +@verbatim +% wget http://www.example.com/page.html +@end verbatim + +But most web pages contain links to images, CSS and JavaScript files, +required for complete rendering. +@url{https://www.gnu.org/software/wget/, GNU Wget} supports that +documents parsing and understanding page dependencies. You can download +the whole page with dependencies the following way: +@verbatim +% wget \ + --page-requisites \ + --convert-links \ + --adjust-extension \ + --restrict-file-names=ascii \ + --span-hosts \ + --random-wait \ + --execute robots=off \ + http://www.example.com/page.html +@end verbatim +that will create @file{www.example.com} directory with all files +necessary to view @file{page.html} web page. You can create single file +compressed tarball with that directory and send it to remote node: +@verbatim +% tar cf - www.example.com | xz -9 | + nncp-file - remote.node:www.example.com-page.tar.xz +@end verbatim + +But there are multi-paged articles, there are the whole interesting +sites you want to get in a single package. You can mirror the whole web +site by utilizing @command{wget}'s recursive feature: +@verbatim +% wget \ + --recursive \ + --timestamping \ + -l inf \ + --no-remove-listing \ + --no-parent \ + [...] + http://www.example.com/ +@end verbatim + +There is a standard for creating +@url{https://en.wikipedia.org/wiki/Web_ARChive, Web ARChives}: +@strong{WARC}. Fortunately again, @command{wget} supports it as an +output format. +@verbatim +% wget \ + --warc-file www.example_com-$(date '+%Y%M%d%H%m%S') \ + --no-warc-compression \ + --no-warc-keep-log \ + [...] + http://www.example.com/ +@end verbatim +That command will create uncompressed @file{www.example_com-XXX.warc} +web archive. By default, WARCs are compressed using +@url{https://en.wikipedia.org/wiki/Gzip, gzip}, but, in example above, +we have disabled it to compress with stronger @command{xz}, before +sending via @command{nncp-file}. + +There are plenty of software acting like HTTP proxy for your browser, +allowing to view that WARC files. However you can extract files from +that archive using @url{https://pypi.python.org/pypi/Warcat, warcat} +utility, producing usual directory hierarchy: +@verbatim +% python3 -m warcat \ + extract www.example_com-XXX.warc \ + --output-dir www.example.com-XXX \ + --progress +@end verbatim + +@node BitTorrent +@section BitTorrent and huge files + +If dealing with @ref{Git}, @ref{Feeds, web feeds} and @ref{Multimedia, +multimedia} goes relatively fast, then BitTorrent and huge files +consumes much time. You can not wait for downloads finish, but want to +queue them after. + +@url{http://aria2.github.io/, aria2} multi-protocol download utility +could be used for solving that issue conveniently. It supports HTTP, +HTTPS, FTP, SFTP and BitTorrent protocols, together with +@url{http://tools.ietf.org/html/rfc5854, Metalink} format. BitTorrent +support is fully-featured: UDP trackers, DHT, PEX, encryption, magnet +URIs, Web-seeding, selective downloads, LPD. @command{aria2} can +accelerate HTTP*/*FTP downloads by segmented multiple parallel +connections. + +You can queue you files after they are completely downloaded: +@verbatim +% cat send-downloaded.sh +#!/bin/sh +nncp-file "$3" remote.node + +% aria2c \ + --on-download-complete send-downloaded.sh \ + http://example.org/file.iso \ + http://example.org/file.iso.asc +@end verbatim + +Also you can prepare +@url{http://aria2.github.io/manual/en/html/aria2c.html#files, input file} +with the jobs you want to download: +@verbatim +% cat jobs +http://www.nncpgo.org/download/nncp-0.11.tar.xz + out=nncp.txz +http://www.nncpgo.org/download/nncp-0.11.tar.xz.sig + out=nncp.txz.sig +% aria2c \ + --on-download-complete send-downloaded.sh \ + --input-file jobs +@end verbatim +and all that downloaded (@file{nncp.txz}, @file{nncp.txz.sig}) files +will be sent to @file{remote.node} when finished. + +@node Git +@section Integration with Git + +@url{https://git-scm.com/, Git} version control system already has all +necessary tools for store-and-forward networking. +@url{https://git-scm.com/docs/git-bundle, git-bundle} command is +everything you need. + +Use it to create bundles containing all required blobs/trees/commits and tags: +@verbatim +% git bundle create repo-initial.bundle master +% git tag -f last-bundle master +% nncp-file repo-initial.bundle remote.node:repo-$(date % '+%Y%M%d%H%m%S').bundle +@end verbatim + +Do usual working with the Git: commit, add, branch, checkout, etc. When +you decide to queue your changes for sending, create diff-ed bundle and +transfer them: +@verbatim +% git bundle create repo-$(date '+%Y%M%d%H%m%S').bundle last-bundle..master +or maybe +% git bundle create repo-$(date '+%Y%M%d').bundle --since=10.days master +@end verbatim + +Received bundle on remote machine acts like usual remote: +@verbatim +% git clone -b master repo-XXX.bundle +@end verbatim +overwrite @file{repo.bundle} file with newer bundles you retrieve and +fetch all required branches and commits: +@verbatim +% git pull # assuming that origin remote points to repo.bundle +% git fetch repo.bundle master:localRef +% git ls-remote repo.bundle +@end verbatim + +Bundles are also useful when cloning huge repositories (like Linux has). +Git's native protocol does not support any kind of interrupted download +resuming, so you will start from the beginning if connection is lost. +Bundles, being an ordinary files, can be downloaded with native +HTTP/FTP/NNCP resuming capabilities. After you fetch repository via the +bundle, you can add an ordinary @file{git://} remote and fetch the +difference. + +@node Multimedia +@section Integration with multimedia streaming + +Many video and audio streams could be downloaded using +@url{http://yt-dl.org/, youtube-dl} program. +@url{https://rg3.github.io/youtube-dl/supportedsites.html, Look} how +many of them are supported, including @emph{Dailymotion}, @emph{Vimeo} +and @emph{YouTube}. + +When you multimedia becomes an ordinary file, you can transfer it easily. +@verbatim +% youtube-dl \ + --exec 'nncp-file {} remote.node:' \ + 'https://www.youtube.com/watch?list=PLd2Cw8x5CytxPAEBwzilrhQUHt_UN10FJ' +@end verbatim diff --git a/doc/postfix.texi b/doc/postfix.texi deleted file mode 100644 index 95ed9bb..0000000 --- a/doc/postfix.texi +++ /dev/null @@ -1,115 +0,0 @@ -@node Postfix -@unnumbered Integration with Postfix - -This section is taken from @url{http://www.postfix.org/nncp_README.html, -Postfix and UUCP} manual and just replaces UUCP-related calls with NNCP -ones. - -@strong{Setting up a Postfix Internet to NNCP gateway} - -Here is how to set up a machine that sits on the Internet and that forwards -mail to a LAN that is connected via NNCP. - -@itemize - -@item You need an @ref{nncp-mail} program that extracts the sender -address from mail that arrives via NNCP, and that feeds the mail into -the Postfix @command{sendmail} command. - -@item Define a @command{pipe(8)} based mail delivery transport for -delivery via NNCP: -@verbatim -/usr/local/etc/postfix/master.cf: -nncp unix - n n - - pipe - flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient -@end verbatim - -This runs the @command{nncp-mail} command to place outgoing mail into -the NNCP queue after replacing @var{$nexthop} by the the receiving NNCP -node and after replacing @var{$recipient} by the recipients. The -@command{pipe(8)} delivery agent executes the @command{nncp-mail} -command without assistance from the shell, so there are no problems with -shell meta characters in command-line parameters. - -@item Specify that mail for @emph{example.com}, should be delivered via -NNCP, to a host named @emph{nncp-host}: - -@verbatim -/usr/local/etc/postfix/transport: - example.com nncp:nncp-host - .example.com nncp:nncp-host -@end verbatim - -See the @command{transport(5)} manual page for more details. - -@item Execute the command @command{postmap /etc/postfix/transport} -whenever you change the @file{transport} file. - -@item Enable @file{transport} table lookups: - -@verbatim -/usr/local/etc/postfix/main.cf: - transport_maps = hash:$config_directory/transport -@end verbatim - -@item Add @emph{example.com} to the list of domains that your site is -willing to relay mail for. - -@verbatim -/usr/local/etc/postfix/main.cf: - relay_domains = example.com ...other relay domains... -@end verbatim - -See the @option{relay_domains} configuration parameter description for -details. - -@item Execute the command @command{postfix reload} to make the changes -effective. - -@end itemize - -@strong{Setting up a Postfix LAN to NNCP gateway} - -Here is how to relay mail from a LAN via NNCP to the Internet. - -@itemize - -@item You need an @ref{nncp-mail} program that extracts the sender -address from mail that arrives via NNCP, and that feeds the mail into -the Postfix @command{sendmail} command. - -@item Specify that all remote mail must be sent via the @command{nncp} -mail transport to your NNCP gateway host, say, @emph{nncp-gateway}: - -@verbatim -/usr/local/etc/postfix/main.cf: - relayhost = nncp-gateway - default_transport = nncp -@end verbatim - -Postfix 2.0 and later also allows the following more succinct form: - -@verbatim -/usr/local/etc/postfix/main.cf: - default_transport = nncp:nncp-gateway -@end verbatim - -@item Define a @command{pipe(8)} based message delivery transport for -mail delivery via NNCP: - -@verbatim -/usr/local/etc/postfix/master.cf: -nncp unix - n n - - pipe - flags=F user=nncp argv=nncp-mail -quiet $nexthop $recipient -@end verbatim - -This runs the @command{nncp-mail} command to place outgoing mail into -the NNCP queue. It substitutes the hostname (@emph{nncp-gateway}, or -whatever you specified) and the recipients before executing the command. -The @command{nncp-mail} command is executed without assistance from the -shell, so there are no problems with shell meta characters. - -@item Execute the command @command{postfix reload} to make the changes -effective. - -@end itemize -- 2.44.0