Basic support for BSD and Darwin style chflags (stat flags). Applies
these flags at the end of file restore.
Supports linux style ioctl_iflags(2) in a 2 step process. Flags that
need to be applied prior to writes such as compress and especially no-COW
are applied immediately upon file open.
The flags format is backwards compatible. An attribute starting with a
null byte is used to store flags in the entry attributes table. With
an old version of duplicacy the restore of this attribute should silently
fail (effectively be ignored).
Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink.
TODO: Tests, possible option to switch off mutable/append prior to
restore of existing file similar to rsync. Does not apply attributes
or flags to the top most directory.
This is useful for upgrading an existing storage to zstd compression or others.
Chunks need to be decompressed and re-compressed during copy anyway. Only
the bit-identical option requires the same compression level
Also fix a typo: compatiable -> compatible
Zstd compression can be enabled by providing `-zstd` or `-zstd-level <level>`
to `init`, `add`, or `backup`. With `-zstd` the compression level will be
`default`, and with `-zstd-level` the level can be any of `fastest`, `default`,
`better`, or `best`.
This option is useful only when erasure coding is enabled. It will
download and re-upload chunks that contain corruption but are
generally recoverable. It can also be used to fix chunks that
are created by 3.0.1 on arm64 machines with wrong hashes.
This commit fixed 2 bugs. The first bug occurs when an incomplete backup
contains a zero-byte file and no chunks. The second bug occurs when the
repository contains only zero-byte files.
Main changes:
* Change the listing order of files/directories so that the local and remote
snapshots can be compared on-the-fly.
* Introduce a new struct called EntryList that maintains a list of
files/directories, which are kept in memory when the number is lower, and
serialized into a file when there are too many.
* EntryList can also be turned into an on-disk incomplete snapshot quickly,
to support fast-resume on next run.
* ChunkOperator can now download and upload chunks, thus replacing original
ChunkDownloader and ChunkUploader. The new ChunkDownloader is only used
to prefetch chunks during the restore operation.
When restoring a file that doesn't exit locally, if the file is large (>100M)
Duplicacy will create an empty sparse file. But this newly created file will
be mistaken for a local copy and hence the restore will fail with a message
suggesting the -overwrite option.
* Restore/check should report an error instead of a success at the end if there
were any errors and -persist is specified
* Don't compute the file hash before passing the file to the chunk maker; this is
redundant as the chunk maker will produce the file hash
* Add a LOG_WERROR function to switch between LOG_WARN and LOG_ERROR dynamically
The fixed-size chunk reader may create 0-byte chunks from empty files. This
may cause validation errors when preparing the snapshot file as the last step
of a backup.
When copying to an RSA-encrypted storage, we relied on the RSA encryption
version to determine if a chunk is a snapshot chunk or a file chunk. This is
wrong when the source storage is not encrypted or not RSA-encrypted. There
is a more reliable to determine if a chunk is a snapshot chunk or not.
The field encryptionVersion in the Chunk struct is supposed to pass the status
of RSA encrytpion from a source chunk to a destination chunk in a copy command.
This field needs to be a 3-state boolean in order to pass the status correctly.
You can now add a key 'filters' in the preferences file that points to the
path of the filters file. If this key is not found in the preferences,
the default location '.duplicacy/filters' is used.
There is a new option '-filters' for the set command that set this key in
the preferences, but you can also edit the file directly.
This is to support public key encryption in the backup operation. You can use
the -key option to supply the public key to the backup command, and then the
same option to supply the private key when restoring a previous revision.
The storage must be encrypted for this to work.
Directories containing a file with this name will not be backed up. I find it easier to drop a .nobackup file in directories I don't want backed up instead of maintaining a file of exclusions. This is also useful for scripts that create data in the repository but don't want it to be backed up.
