MTH memory usage optimization

[nncp.git] / doc / news.texi

@node News
@unnumbered News

See also this page @ref{Новости, on russian}.

@menu
* Release 7.3.1: Release 7_3_1.
* Release 7.3.0: Release 7_3_0.
* Release 7.2.1: Release 7_2_1.
* Release 7.2.0: Release 7_2_0.
* Release 7.1.1: Release 7_1_1.
* Release 7.1.0: Release 7_1_0.
* Release 7.0.0: Release 7_0_0.
* Release 6.6.0: Release 6_6_0.
* Release 6.5.0: Release 6_5_0.
* Release 6.4.0: Release 6_4_0.
* Release 6.3.0: Release 6_3_0.
* Release 6.2.1: Release 6_2_1.
* Release 6.2.0: Release 6_2_0.
* Release 6.1.0: Release 6_1_0.
* Release 6.0.0: Release 6_0_0.
* Release 5.6.0: Release 5_6_0.
* Release 5.5.1: Release 5_5_1.
* Release 5.5.0: Release 5_5_0.
* Release 5.4.1: Release 5_4_1.
* Release 5.4.0: Release 5_4_0.
* Release 5.3.3: Release 5_3_3.
* Release 5.3.2: Release 5_3_2.
* Release 5.3.1: Release 5_3_1.
* Release 5.3.0: Release 5_3_0.
* Release 5.2.1: Release 5_2_1.
* Release 5.2.0: Release 5_2_0.
* Release 5.1.2: Release 5_1_2.
* Release 5.1.1: Release 5_1_1.
* Release 5.1.0: Release 5_1_0.
* Release 5.0.0: Release 5_0_0.
* Release 4.1: Release 4_1.
* Release 4.0: Release 4_0.
* Release 3.4: Release 3_4.
* Release 3.3: Release 3_3.
* Release 3.2: Release 3_2.
* Release 3.1: Release 3_1.
* Release 3.0: Release 3_0.
* Release 2.0: Release 2_0.
* Release 1.0: Release 1_0.
* Release 0.12: Release 0_12.
* Release 0.11: Release 0_11.
* Release 0.10: Release 0_10.
* Release 0.9: Release 0_9.
* Release 0.8: Release 0_8.
* Release 0.7: Release 0_7.
* Release 0.6: Release 0_6.
* Release 0.5: Release 0_5.
* Release 0.4: Release 0_4.
* Release 0.3: Release 0_3.
* Release 0.2: Release 0_2.
@end menu

@node Release 7_3_1
@section Release 7.3.1
@itemize

@item
Fixed possibly left opened file descriptor in online commands.

@item
Severely decreased memory usage of MTH hashing.

@end itemize

@node Release 7_3_0
@section Release 7.3.0
@itemize

@item
Fixed some workability problems on 32-bit systems with big files.

@item
Ability to use directory with a bunch of files as a configuration.
@command{nncp-cfgdir} command appeared.

@end itemize

@node Release 7_2_1
@section Release 7.2.1
@itemize

@item
Small optimizations in online commands.

@end itemize

@node Release 7_2_0
@section Release 7.2.0
@itemize

@item
@command{nncp-trns} command appeared for manual transition packets creation.

@item
If destination node of transitional packet has non empty @option{via}
route, then do not ignore, but use it.

@item
Do not relay multicast packet to area message's originator, that
obviously has seen its own packet.

@item
Much less memory usage during MTH hashing when offset is zero: when
packet is not resumed, but for example checked with @command{nncp-check}
command.

@end itemize

@node Release 7_1_1
@section Release 7.1.1
@itemize

@item
Fixed failing directories fsync after @file{.seen} file creation.

@end itemize

@node Release 7_1_0
@section Release 7.1.0
@itemize

@item
Multicasting areas feature appeared. Implemented merely by an additional
plain packet type with @command{nncp-toss}, @command{nncp-file} and
@command{nncp-exec} commands modification.

@item
Fixed workability of @command{nncp-file} and @command{nncp-exec}
commands, that use temporary file (stdin and @option{-use-tmp}).

@item
Fixed disappearing bad return code in @command{nncp-exec} command.

@item
Fixed invalid @file{.hdr} generation when transitional packets are used.

@item
@option{-all} option appeared in @command{nncp-rm} command, applying to
all the nodes at once.

@item
@option{-cycle} option appeared in @command{nncp-check} command, looping
the check in infinite cycle.

@item
@command{nncp-rm} command can take node alias name.

@item
@command{nncp-pkt} can parse @file{.hdr} files.

@end itemize

@node Release 7_0_0
@section Release 7.0.0
@itemize

@item
Minimal required Go version 1.13.

@item
Merkle Tree-based Hashing with BLAKE3 (MTH) is used instead of BLAKE2b.
Because of that, there are backward @strong{incompatible} changes of
encrypted files (everything laying in the spool directory) and
@file{.meta} files of chunked transfer.

Current implementation is far from being optimal: it lacks
parallelizable calculations and has higher memory consumption: nearly
512 KiB for each 1 GiB of file's data. Future performance and memory
size optimizations should not lead to packet's format change. But it is
still several times faster than BLAKE2b.

@item
Resumed online downloads, because of MTH, require reading only of the
preceding part of file, not the whole one as was before.

@item
@command{nncp-hash} utility appeared for calculating file's MTH hash.

@item
BLAKE2 KDF and XOF functions are replaced with BLAKE3 in encrypted
packets. Lowering number of used primitives. Also, its encrypted
packet's header is used as an associated data during encryption.

@item
MultiCast Discovery uses
@verb{|ff02::4e4e:4350|} address instead of @verb{|ff02::1|}.

@item
@command{nncp-cfgenc} mistakenly asked passphrase three times during encryption.

@item
@command{nncp-stat} reports about partly downloaded packets.

@item
Updated dependencies.

@end itemize

@node Release 6_6_0
@section Release 6.6.0
@itemize

@item
@command{nncp-daemon}, @command{nncp-call} and @command{nncp-caller}
commands wait for all background checksummers completion after
connection is finished.

@item
Added possibility of address determining through multicast announcement
in local area network, so called MCD (MultiCast Discovery).

@end itemize

@node Release 6_5_0
@section Release 6.5.0
@itemize

@item
Fixed segfault in @command{nncp-daemon} when SP handshake did not succeed.

@item
Fixed possible bad return code ignoring in automatic tosser.

@item
Fixed race during file descriptors closing when online protocol call is
finished, that could lead to write error of received packet fragment.

@item
Kill all packet transmission progress bars in @command{nncp-daemon},
@command{nncp-call} and @command{nncp-caller} when call is finished.

@end itemize

@node Release 6_4_0
@section Release 6.4.0
@itemize

@item
Fixed possible race in online protocol, that lead to panic.

@end itemize

@node Release 6_3_0
@section Release 6.3.0
@itemize

@item
Fixed possible panic while showing progress during online protocol.

@end itemize

@node Release 6_2_1
@section Release 6.2.1
@itemize

@item
Three places in logs contained excess @code{%s}.

@end itemize

@node Release 6_2_0
@section Release 6.2.0
@itemize

@item
Returned @command{nncp-caller}'s @option{-autotoss*} options workability.

@item
Yet another logging refactoring and simplification.
Should be no visible differences to the end user.

@end itemize

@node Release 6_1_0
@section Release 6.1.0
@itemize

@item
Optimization: most commands do not keep opened file descriptors now.
Previously you can exceed maximal number of opened files if you have got
many packets in the spool directory.

@item
Optimization: do not close file descriptor of the file we download
online. Previously each chunk lead to expensive open/close calls.

@item
Online downloaded files are saved with @file{.nock} (non-checksummed)
suffix, waiting either for @command{nncp-check}, or online daemons to
perform integrity check.

@item
Optimization: files, that are not resumed, are checksummed immediately
during the online download, skipping @file{.nock}-intermediate step.

@item
Ability to store encrypted packet's header in @file{.hdr} file, close to
the packet itself. That can greatly increase performance of packets
listing on filesystems with big block's size.

@end itemize

@node Release 6_0_0
@section Release 6.0.0
@itemize

@item
Log uses human readable and easy machine parseable
@url{https://www.gnu.org/software/recutils/, recfile} format for the
records, instead of structured RFC 3339 lines. Old logs are not readable
by @command{nncp-log} anymore.

@item
@option{-autotoss*} option workability with @command{nncp-daemon}'s
@option{-inetd} mode.

@item
Call's @option{when-tx-exists} allows to make a call only when outbound
packets exists. Combined with seconds-aware cron expression that can be
used as some kind of auto dialler.

@item
@command{nncp-cronexpr} command allows you to check validity and
expectations of specified cron expression.

@end itemize

@node Release 5_6_0
@section Release 5.6.0
@itemize

@item
@option{-autotoss*} option runs tosser not after the call, but every
second while it is active.

@item
@option{autotoss}, @option{autotoss-doseen},
@option{autotoss-nofile}, @option{autotoss-nofreq},
@option{autotoss-noexec}, @option{autotoss-notrns} options available in
@option{calls} configuration section. You can configure per-call
automatic tosser options.

@item
Use vendoring, instead of @env{GOPATH} overriding during tarball
installation, because current minimal Go's version is 1.12 and it
supports modules.

@end itemize

@node Release 5_5_1
@section Release 5.5.1
@itemize

@item
Respect for @env{BINDIR}, @env{INFODIR} and @env{DOCDIR} environment
variables in @file{config} during installation.

@end itemize

@node Release 5_5_0
@section Release 5.5.0
@itemize

@item
Bugfixes in @command{nncp-call(er)}/@command{nncp-daemon},
@command{nncp-bundle} and @command{nncp-stat}.

@item
@command{nncp-rm} has @option{-dryrun} and @option{-older} options now.

@item
@command{nncp-exec} has @option{-use-tmp} and @option{-nocompress}
options now. Uncompressed packets are not compatible with previous NNCP
versions.

@item
@command{nncp-call}, @command{nncp-caller} and @command{nncp-daemon} commands
have @option{-autotoss*} options for running tosser after call is ended.

@item
Updated dependencies. Minimal required Go version is 1.12.

@end itemize

@node Release 5_4_1
@section Release 5.4.1
@itemize

@item
Fixed @code{SENDMAIL} variable usage during the build.

@end itemize

@node Release 5_4_0
@section Release 5.4.0
@itemize

@item
Updated dependencies.

@item
Build system is moved from Makefiles to @url{http://cr.yp.to/redo.html, redo}.
This should not influence package maintainers, because minimal @command{redo}
implementation is included in tarball.

@end itemize

@node Release 5_3_3
@section Release 5.3.3
@itemize

@item
More various error checks.

@item
Updated dependencies.

@end itemize

@node Release 5_3_2
@section Release 5.3.2
@itemize

@item
Fixed incorrect logic of @option{onlinedeadline} timeout, where
connection won't take into account incoming packets events and will
forcefully disconnect.

@end itemize

@node Release 5_3_1
@section Release 5.3.1
@itemize

@item
Fixed @option{onlinedeadline} workability with call addresses that use
external commands (@verb{#"|somecmd"#}).

@item
@command{nncp-stat} has @option{-pkt} option displaying information
about each packet in the spool.

@end itemize

@node Release 5_3_0
@section Release 5.3.0
@itemize

@item
Progress messages contain prefix, describing the running action.

@item
Fixed not occurring handshake messages padding.

@item
Finish all SP protocol related goroutines, less memory leak.

@item
SP protocol generates less socket write calls, thus generating less TCP
packets.

@item
Check @option{onlinedeadline} and @option{maxonlinetime} options every
second, independently from socket reads (up to 10 seconds).

@item
Once per minute, if no other traffic exists, PING packets are sent in
SP-connection. That allows faster determining of connection unworkability.

@item
@command{nncp-toss} uses lock-file to prevent simultaneous tossing.

@end itemize

@node Release 5_2_1
@section Release 5.2.1
@itemize

@item
Fixed SP protocol error handling, sometimes causing program panic.

@end itemize

@node Release 5_2_0
@section Release 5.2.0
@itemize

@item
Most commands by default show oneline operations progress.
@option{-progress}, @option{-noprogress} command line options,
@option{noprogress} configuration file option appeared.

@item
Fixed incorrect @command{nncp-check} command return code, that returned
bad code when everything is good.

@item
Free disk space check during @command{nncp-bundle -rx} call.

@end itemize

@node Release 5_1_2
@section Release 5.1.2
@itemize

@item
@strong{Critical} vulnerability: remote peers authentication could lead
to incorrect identification of remote side, allowing foreign encrypted
packets downloading.

@item
Bugfix: private and public Noise keys were swapped in newly created
configuration files, that lead to inability to authenticate online peers.

@item
Explicit directories fsync-ing for guaranteed files renaming.

@end itemize

@node Release 5_1_1
@section Release 5.1.1
@itemize

@item
Fixed workability of @command{nncp-file} with @option{-chunked 0} option.

@end itemize

@node Release 5_1_0
@section Release 5.1.0
@itemize

@item
@command{nncp-file} can send directories, automatically creating pax
archive on the fly.

@item
Free disk space is checked during outbound packets creation.

@item
@option{freq}, @option{freqminsize}, @option{freqchunked} configuration
file options replaced with the structure:
@option{freq: @{path: ..., minsize: ..., chunked: ...@}}.

@item
Added @option{freq.maxsize} configuration file option, forbidding of
freq sending larger than specified size.

@item
Ability to notify about successfully executed commands (exec) with
@option{notify.exec} configuration file option.

@end itemize

@node Release 5_0_0
@section Release 5.0.0
@itemize

@item
@strong{Incompatible} configuration file format change: YAML is
replaced with Hjson, due to its simplicity, without noticeable lack
of either functionality or convenience.

@item
@strong{Incompatible} plain packet format changes. Older versions are
not supported. @code{zlib} compression is replaced with
@code{Zstandard}, due to its speed and efficiency, despite library
version is not mature enough.

@item
Ability to call remote nodes via pipe call of external command, not only
through TCP.

@item
@command{nncp-cfgnew} generates configuration file with many
comments. @option{-nocomments} option can be used for an old
behaviour.

@item
Duplicate filenames have @file{.CTR} suffix, instead of @file{CTR}, to
avoid possible collisions with @file{.nncp.chunkCTR}.

@item
Ability to override process umask through configuration file option.

@item
Files and directories are created with 666/777 permissions by default,
allowing control with @command{umask}.

@item
Updated dependencies.

@item
Full usage of go modules for dependencies management
(@code{go.cypherpunks.ru/nncp/v5} namespace is used).

@item
Forbid any later GNU GPL version autousage
(project's licence now is GNU GPLv3-only).

@end itemize

@node Release 4_1
@section Release 4.1
@itemize
@item Workability on GNU/Linux systems and Go 1.10 is fixed.
@end itemize

@node Release 4_0
@section Release 4.0
@itemize

@item
@strong{Incompatible} encrypted and eblob packet format change: AEAD
encryption mode with 128 KiB blocks is used now, because previously
@command{nncp-toss} did not verify encrypted packet's MAC before feeding
decrypted data to external command. Older versions are not supported.

@item
Available free space checking before copying in @command{nncp-xfer},
@command{nncp-daemon}, @command{nncp-call(er)}.

@item
@command{nncp-call} has ability only to list packets on remote node,
without their transmission.

@item
@command{nncp-call} has ability to transfer only specified packets.

@item
Workability of @option{xxrate} preference in @option{calls}
configuration file section.

@item
Dependant libraries are updated.

@item
Minor bugfixes.

@item
Begin using of @code{go.mod} subsystem.

@end itemize

@node Release 3_4
@section Release 3.4
@itemize
@item @command{nncp-daemon} can be run as @command{inetd}-service.
@end itemize

@node Release 3_3
@section Release 3.3
@itemize

@item
@command{nncp-daemon}, @command{nncp-call}, @command{nncp-caller} check
if @file{.seen} exists and treat it like file was already downloaded.
Possibly it was transferred out-of-bound and remote side needs to be
notifier about that.

@item
If higher priority packet is spooled, then @command{nncp-daemon} will
queue its sending first, interrupting lower priority transmissions.

@item
Simple packet rate limiter added to online-related tools
(@command{nncp-daemon}, @command{nncp-call}, @command{nncp-caller}).

@item
Ability to specify niceness with symbolic notation:
@verb{|NORMAL|}, @verb{|BULK+10|}, @verb{|PRIORITY-5|}, etc.

@item
Changed default niceness levels:
for @command{nncp-exec} from 64 to 96,
for @command{nncp-freq} from 64 to 160,
for @command{nncp-file} from 196 to 224.

@end itemize

@node Release 3_2
@section Release 3.2
@itemize
@item
@strong{Incompatible} @emph{bundle} archive format changes and
@command{nncp-bundle} workability with Go 1.10+. Bundles must be
valid tar archives, but Go 1.9 made them invalid because of long paths
inside. NNCP accidentally was dependant on that bug. Explicit adding of
@file{NNCP/} directory in archive restores workability with valid tar
archives.
@end itemize

@node Release 3_1
@section Release 3.1
@itemize
@item
Ability to disable relaying at all using @verb{|-via -|} command line option.
@end itemize

@node Release 3_0
@section Release 3.0
@itemize

@item
@strong{Incompatible} plain packet format changes. Older versions are
not supported.

@item
Ability to queue remote command execution, by configuring @option{exec}
option in configuration file and using @command{nncp-exec} command:
    @itemize
    @item
    @command{nncp-mail} command is replaced with more flexible
    @command{nncp-exec}. Instead of @verb{|nncp-mail NODE RECIPIENT|}
    you must use @verb{|nncp-exec NODE sendmail RECIPIENT|}.
    @item
    @option{sendmail} configuration file option is replaced with
    @option{exec}. @verb{|sendmail: [...]|} must be replaced with
    @verb{|exec: sendmail: [...]|}.
    @end itemize

@item
Ability to override @option{via} configuration option for destination
node via @option{-via} command line option for following commands:
@command{nncp-file}, @command{nncp-freq}, @command{nncp-exec}.

@item
Chunked files, having size less than specified chunk size, will be sent
as an ordinary single file.

@item
Exec commands are invoked with additional @env{NNCP_NICE} and
@env{NNCP_SELF} environment variables.

@item
Files, that are sent as a reply to freq, have niceness level taken from
the freq packet. You can set desired niceness during @command{nncp-freq}
invocation using @option{-replynice} option.

@item
@command{nncp-toss} command can ignore specified packet types during
processing: @option{-nofile}, @option{-nofreq}, @option{-noexec},
@option{-notrns}.

@item
@command{nncp-file} command uses
@option{FreqMinSize}/@option{FreqChunked} configuration file options
for @option{-minsize}/@option{-chunked} by default. You can turn this
off by specifying zero value.

@end itemize

@node Release 2_0
@section Release 2.0
@itemize

@item
@strong{Incompatible} encrypted/eblob packet format changes. Older
versions are not supported.

@item
Twofish encryption algorithm is replaced with ChaCha20. It is much more
faster. One cryptographic primitive less.

@item
HKDF-BLAKE2b-256 KDF algorithm is replaced with BLAKE2Xb XOF. Yet
another cryptographic primitive less (assuming that BLAKE2X is nearly
identical to BLAKE2).

@end itemize

@node Release 1_0
@section Release 1.0
@itemize

@item
@strong{Incompatible} encrypted packet format changes. Older versions
are not supported.

@item
@command{nncp-bundle} command can either create stream of encrypted
packets, or digest it. It is useful when dealing with
@code{stdin}/@code{stdout} based transmission methods (like writing to
CD-ROM without intermediate prepared ISO image and working with tape
drives).

@item
@command{nncp-toss} is able to create @file{.seen} files preventing
duplicate packets receiving.

@item
Single background checksum verifier worker is allowed in
@command{nncp-call}. This is helpful when thousands of small inbound
packets could create many goroutines.

@item
Ability to override path to spool directory and logfile through either
command line argument, or environment variable.

@item
@command{nncp-rm} is able to delete outbound/inbound, @file{.seen},
@file{.part}, @file{.lock} and temporary files.

@end itemize

@node Release 0_12
@section Release 0.12
@itemize
@item Sendmail command is called with @env{NNCP_SENDER} environment variable.
@end itemize

@node Release 0_11
@section Release 0.11
@itemize
@item @command{nncp-stat}'s command output is sorted by node name.
@end itemize

@node Release 0_10
@section Release 0.10
@itemize
@item
@command{nncp-freq}'s @file{DST} argument is optional now. Last
@file{SRC} path's element will be used by default.
@end itemize

@node Release 0_9
@section Release 0.9
@itemize
@item
Fix @option{-rx}/@option{-tx} arguments processing in
@command{nncp-call} command. They were ignored.
@end itemize

@node Release 0_8
@section Release 0.8
@itemize
@item
Little bugfix in @command{nncp-file} command, where @option{-minsize}
option for unchunked transfer was not in KiBs, but in bytes.
@end itemize

@node Release 0_7
@section Release 0.7
@itemize

@item
Ability to feed @command{nncp-file} from @code{stdin}, that uses an
encrypted temporary file for that.

@item
Chunked files transmission appeared with corresponding
@command{nncp-reass} command and @option{freqchunked} configuration file
entry. Useful for transferring big files over small storage devices.

@item
@option{freqminsize} configuration file option, analogue to
@option{-minsize} one.

@item
@command{nncp-xfer}'s @option{-force} option is renamed to
@option{-mkdir} for clarity.

@item
@option{-minsize} option is specified in KiBs, not bytes, for
convenience.

@item
@command{nncp-newcfg} command is renamed to @command{nncp-cfgnew},
and @command{nncp-mincfg} to @command{nncp-cfgmin} -- now they have
common prefix and are grouped together for convenience.

@item
@command{nncp-cfgenc} command appeared, allowing configuration file
encryption/decryption, for keeping it safe without any either OpenPGP or
similar tools usage.

@item
Cryptographic libraries (dependencies) are updated.

@end itemize

@node Release 0_6
@section Release 0.6
@itemize
@item Small @command{nncp-rm} command appeared.
@item Cryptographic libraries (dependencies) are updated.
@end itemize

@node Release 0_5
@section Release 0.5
@itemize
@item
Trivial small fix in default niceness level of @command{nncp-file}
and @command{nncp-freq} commands.
@end itemize

@node Release 0_4
@section Release 0.4
@itemize

@item
Small fix in @command{nncp-call}, @command{nncp-caller},
@command{nncp-daemon}: they can segmentation fail sometimes (no data is
lost).

@item
@command{nncp-newnode} renamed to @command{nncp-newcfg} -- it is shorter
and more convenient to use.

@item
@command{nncp-mincfg} command appeared: helper allowing to create
minimalistic stripped down configuration file without private keys, that
is useful during @command{nncp-xfer} usage.

@end itemize

@node Release 0_3
@section Release 0.3
@itemize
@item Fixed compatibility with Go 1.6.
@end itemize

@node Release 0_2
@section Release 0.2
@itemize

@item
@strong{Incompatible} packet's format change (magic number is changed
too): size field is encrypted and is not send in plaintext anymore.

@item
@option{-minsize} option gives ability to automatically pad outgoing
packets to specified minimal size.

@item
@command{nncp-daemon} and @command{nncp-call}/@command{nncp-caller}
always check new @emph{tx} packets appearance in the background while
connected. Remote side is immediately notified.

@item
@option{-onlinedeadline} option gives ability to configure timeout of
inactivity of online connection, when it could be disconnected. It could
be used to keep connection alive for a long time.

@item
@option{-maxonlinetime} option gives ability to set maximal allowable
online connection aliveness time.

@item
@command{nncp-caller} command appeared: cron-ed TCP daemon caller.

@item
@command{nncp-pkt} command can decompress the data.

@end itemize