2 @unnumbered Chunked files
4 There is ability to transfer huge files with dividing them into smaller
5 chunks. Each chunk is treated like a separate file, producing separate
6 outbound packet unrelated with other ones.
8 This is useful when your removable storage device has smaller capacity
9 than huge file's size. You can transfer those chunks on different
10 storage devices, and/or at different time, reassembling the whole packet
11 on the destination node.
13 Splitting is done with @ref{nncp-file, nncp-file -chunked} command and
14 reassembling with @ref{nncp-reass} command.
16 Chunked @file{FILE} produces @file{FILE.nncp.meta},
17 @file{FILE.nncp.chunk0}, @file{FILE.nncp.chunk1}, @dots{} files. All
18 @file{.nncp.chunkXXX} can be concatenated together to produce original
21 @file{.nncp.meta} contains information about file/chunk
22 size and their hash checksums. This is
23 @url{https://tools.ietf.org/html/rfc4506, XDR}-encoded structure:
26 +------------------------------+---------------------+
27 | MAGIC | FILESIZE | CHUNKSIZE | HASH0 | HASH1 | ... |
28 +------------------------------+---------------------+
31 @multitable @columnfractions 0.2 0.3 0.5
32 @headitem @tab XDR type @tab Value
33 @item Magic number @tab
34 8-byte, fixed length opaque data @tab
35 @verb{|N N C P M 0x00 0x00 0x02|}
37 unsigned hyper integer @tab
38 Whole reassembled file's size
40 unsigned hyper integer @tab
41 Size of each chunk (except for the last one, that could be smaller)
43 variable length array of 32 byte fixed length opaque data @tab
44 @ref{MTH} checksum of each chunk
48 It is strongly advisable to reassemble incoming chunked files on
49 @url{https://en.wikipedia.org/wiki/ZFS, ZFS} dataset with deduplication
50 feature enabled. It could be more CPU and memory hungry, but will save
51 your disk's IO and free space from pollution (although temporary). But
52 pay attention that you chunks must be either equal to, or divisible by
53 dataset's @option{recordsize} value for deduplication workability.
54 Default ZFS's @option{recordsize} is 128 KiBs, so it is advisable to
55 chunk your files on 128, 256, 384, 512, etc KiB blocks.