The prune command can remove redundant chunks (chunks with the same chunk id
but at different subdirectory level). However, if the same chunk appears
mutliple times in the listing returned by the storage, it will be treated as
a redundant chunk and thus removed.
The version bit should not be set to 1 when encoding a snapshot. Instead,
it must be set to 1 on snapshot creation.
To correctly process old snapshots encoded incorrectly with version bit set
to 1, the first byte of the encoded file list is also checked. If the first
byte is `[`, then it must be an old snapshot, since the file list in the new
snapshot format always starts with a string encoded in msgpack, the first
byte of which can't be `[`.
This option is useful only when erasure coding is enabled. It will
download and re-upload chunks that contain corruption but are
generally recoverable. It can also be used to fix chunks that
are created by 3.0.1 on arm64 machines with wrong hashes.
Main changes:
* Change the listing order of files/directories so that the local and remote
snapshots can be compared on-the-fly.
* Introduce a new struct called EntryList that maintains a list of
files/directories, which are kept in memory when the number is lower, and
serialized into a file when there are too many.
* EntryList can also be turned into an on-disk incomplete snapshot quickly,
to support fast-resume on next run.
* ChunkOperator can now download and upload chunks, thus replacing original
ChunkDownloader and ChunkUploader. The new ChunkDownloader is only used
to prefetch chunks during the restore operation.
* Restore/check should report an error instead of a success at the end if there
were any errors and -persist is specified
* Don't compute the file hash before passing the file to the chunk maker; this is
redundant as the chunk maker will produce the file hash
* Add a LOG_WERROR function to switch between LOG_WARN and LOG_ERROR dynamically
This option will download and verify every chunk. Unlike the -files option,
this option only downloads each chunk once. There is also a new -threads
option to use multiple threads to download chunks.
You can now add a key 'filters' in the preferences file that points to the
path of the filters file. If this key is not found in the preferences,
the default location '.duplicacy/filters' is used.
There is a new option '-filters' for the set command that set this key in
the preferences, but you can also edit the file directly.
This is to support public key encryption in the backup operation. You can use
the -key option to supply the public key to the backup command, and then the
same option to supply the private key when restoring a previous revision.
The storage must be encrypted for this to work.
A chunk not in the chunk list may actually exists in two scenarios:
* the chunk may be a special snapshot chunk that contains the chunk sequence,
so it may be resurrected by the chunk downloader if it had been turned into
a fossil before
* if the API to list all chunks doesn't return the complete list due to some
bug
This additional lookup avoid reporting the missing chunk prematurely.
The call to GetSnapshotChunks in ShowStatisticsTabular sets keepChunkHashes to
true -- this can cause too much memory consumption with hundreds of revisions.
Directories containing a file with this name will not be backed up. I find it easier to drop a .nobackup file in directories I don't want backed up instead of maintaining a file of exclusions. This is also useful for scripts that create data in the repository but don't want it to be backed up.
