From 07a0344ed56852bf0570117e504c50e4152a3697 Mon Sep 17 00:00:00 2001 From: Sergey Matveev Date: Fri, 16 Feb 2018 12:43:15 +0300 Subject: [PATCH] Note about ZFS deduplication for chunked files --- doc/chunked.texi | 10 ++++++++++ doc/cmds.texi | 5 ++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/doc/chunked.texi b/doc/chunked.texi index 1682cc3..4ac50b7 100644 --- a/doc/chunked.texi +++ b/doc/chunked.texi @@ -43,3 +43,13 @@ size and their hash checksums. This is variable length array of 32 byte fixed length opaque data @tab BLAKE2b-256 checksum of each chunk @end multitable + +@anchor{ChunkedZFS} +It is strongly advisable to reassemble incoming chunked files on +@url{https://en.wikipedia.org/wiki/ZFS, ZFS} dataset with deduplication +feature enabled. It could be more CPU and memory hungry, but will save +your disk's IO and free space from pollution (although temporary). But +pay attention that you chunks must be either equal to, or divisible by +dataset's @option{recordsize} value for deduplication workability. +Default ZFS's @option{recordsize} is 128 KiBs, so it is advisable to +chunk your files on 128, 256, 384, 512, etc KiB blocks. diff --git a/doc/cmds.texi b/doc/cmds.texi index 2b1d37f..ce0855a 100644 --- a/doc/cmds.texi +++ b/doc/cmds.texi @@ -263,7 +263,8 @@ If @option{-chunked} is specified, then source file will be split @ref{Chunked, on chunks}. @option{INT} is the desired chunk size in KiBs. This mode is more CPU hungry. Pay attention that chunk is saved in spool directory immediately and it is not deleted if any error occurs. -@option{-minsize} option is applied per each chunk. +@option{-minsize} option is applied per each chunk. Do not forget about +@ref{ChunkedZFS, possible} ZFS deduplication issues. If @ref{CfgNotify, notification} is enabled on the remote side for file transmissions, then it will sent simple letter after successful @@ -402,6 +403,8 @@ Checksums: 3: 0e9e229501bf0ca42d4aa07393d19406d40b179f3922a3986ef12b41019b45a3 @end verbatim + Do not forget about @ref{ChunkedZFS, possible} ZFS deduplication issues. + @node nncp-rm @section nncp-rm -- 2.44.0