Basic support for BSD and Darwin style chflags (stat flags). Applies
these flags at the end of file restore.
Supports linux style ioctl_iflags(2) in a 2 step process. Flags that
need to be applied prior to writes such as compress and especially no-COW
are applied immediately upon file open.
The flags format is backwards compatible. An attribute starting with a
null byte is used to store flags in the entry attributes table. With
an old version of duplicacy the restore of this attribute should silently
fail (effectively be ignored).
Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink.
TODO: Tests, possible option to switch off mutable/append prior to
restore of existing file similar to rsync. Does not apply attributes
or flags to the top most directory.
The exclude by attribute function is broken on non-Darwin POSIX: linux and freebsd.
This is because those xattrs must be prefixed by a legal namespace. The old xattr
library implicitly appended the user namespace to the xattr, but the current
official go pkg does not (which is just as well).
Also fix the test to remove the discordant old xattr dependency and provide
test cases for both darwin and non-darwin POSIX.
ListObjects has been deprecated since 2016 and ListObjectsV2 with use of
explicit pagination tokens is more performant for large listings as well.
This also mitigates an issue with iDrive E2 where the StartAfter/Marker
is included in the output, leading to duplicate entries. Right now this
causes an exhaustive prune to delete chunks erroneously flagged as
duplicate, destroying the storage.
The prune command can remove redundant chunks (chunks with the same chunk id
but at different subdirectory level). However, if the same chunk appears
mutliple times in the listing returned by the storage, it will be treated as
a redundant chunk and thus removed.
This is useful for upgrading an existing storage to zstd compression or others.
Chunks need to be decompressed and re-compressed during copy anyway. Only
the bit-identical option requires the same compression level
Also fix a typo: compatiable -> compatible
This is mainly to handle the case when a fossil needs to be downloaded.
This happens when a metadata chunk has been marked as a fossil while the
correpsonding snapshot must be reconstructed for determine referenced
chunks.
The storage url is smb://user@server[:port]/share/path. The password can be
set in the environment variable DUPLICACY_SMB_PASSWORD for default storage or
DUPLICACY_<STORAGE_NAME>_SMB_PASSWORD.
This backend is based on https://github.com/hirochachacha/go-smb2. The
previous samba:// backend is just an alias for the disk-based backend with
caching enabled.
This is needed when a metadata chunk has been turned into a fossil. In B2,
the fossil file is the last version of the file with an 'upload' action, so
to download the file, the 'b2_list_file_versions' api is called to find the
file id and then download it using 'b2_download_file_by_id'.
This is needed when a metadata chunk has been turned into a fossil
Zstd compression can be enabled by providing `-zstd` or `-zstd-level <level>`
to `init`, `add`, or `backup`. With `-zstd` the compression level will be
`default`, and with `-zstd-level` the level can be any of `fastest`, `default`,
`better`, or `best`.
This bug leaks a chunk every time files in a revision are listed. Not a big
deal for backup and restore, but it becomes problematic when listing files in
many revisions for commands such check and history.
The version bit should not be set to 1 when encoding a snapshot. Instead,
it must be set to 1 on snapshot creation.
To correctly process old snapshots encoded incorrectly with version bit set
to 1, the first byte of the encoded file list is also checked. If the first
byte is `[`, then it must be an old snapshot, since the file list in the new
snapshot format always starts with a string encoded in msgpack, the first
byte of which can't be `[`.
This option is useful only when erasure coding is enabled. It will
download and re-upload chunks that contain corruption but are
generally recoverable. It can also be used to fix chunks that
are created by 3.0.1 on arm64 machines with wrong hashes.
highwayhash 1.0.1 contains a bug leading to incorrect hashes on arm64 machines.
The 1.0.1 version is retained in github.com/gilbertchen/highwayhash so the hash
can be checked again if a mismatch is detected by 1.0.2.
This commit fixed 2 bugs. The first bug occurs when an incomplete backup
contains a zero-byte file and no chunks. The second bug occurs when the
repository contains only zero-byte files.
The url format is storj://satellite/bucket/path. You can get the
satellite along with the api access key when requesting an Access
Grant of type API Access.
Main changes:
* Change the listing order of files/directories so that the local and remote
snapshots can be compared on-the-fly.
* Introduce a new struct called EntryList that maintains a list of
files/directories, which are kept in memory when the number is lower, and
serialized into a file when there are too many.
* EntryList can also be turned into an on-disk incomplete snapshot quickly,
to support fast-resume on next run.
* ChunkOperator can now download and upload chunks, thus replacing original
ChunkDownloader and ChunkUploader. The new ChunkDownloader is only used
to prefetch chunks during the restore operation.
This is to avoid the read-after-rename consistency issue where the effect
of renaming may not be observed by the subsequent attempt to download the
just renamed chunk.
When connecting to Google Drive with a service account key, only files in the
service account's own hidden drive space are listable. This change finds
the given storage path among shared folders first so that folders from the user
space can be made accessible via service account.
