Also, don't attempt xattr ops on special files on the BSD likes.
TODO: Should have a way to allow restore without special files,
otherwise very cumbersome for a regular user.
Fix Linux, Darwin, and other BSD (untested) to allow proper
handling of xattrs with symlinks. On Linux we cannot use the f*
syscalls for symlinks because symlinks cannot be opened.
File flags must be handled differently on darwin and other BSD due
to the lack of the LCHFLAGS syscall on darwin, and the fact that it
is emulated in libc. However, we do have O_SYMLINK on darwin.
Special files are device nodes and named pipes. The necessity of the
former is clear, the latter is debatable.
In order to preserve backward compatibility, the device number is
encoded in the StartChunk/StartOffset fields of the entry.
This tracks inode/device from the stat info and creates backward
compatible snapshots that allow preserving hardlinks. Backwards
compatibility is preserved by saving a virtual inode number index in the
Link field of the file entry. Since this field was previously only used
for symlinks, this won't break old versions. Additionally, the entry
data is cloned so restoration with an old version works.
Current limitations are primarility with restore. They include:
- no command line option to prevent hard link restore
- if a file has the immutable or append only flag it will be set before
hardlinks are restored, so hardlinking will fail.
- if a partial restore includes a hardlink but not the parent
directories the hardlink will fail.
These will be solved by grouping restore of hardlinks together
with file, prior to applying final metadata.
- if a file is changed and is being rewritten by a restore hardlinks are
not preserved.
Basic support for BSD and Darwin style chflags (stat flags). Applies
these flags at the end of file restore.
Supports linux style ioctl_iflags(2) in a 2 step process. Flags that
need to be applied prior to writes such as compress and especially no-COW
are applied immediately upon file open.
The flags format is backwards compatible. An attribute starting with a
null byte is used to store flags in the entry attributes table. With
an old version of duplicacy the restore of this attribute should silently
fail (effectively be ignored).
Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink.
TODO: Tests, possible option to switch off mutable/append prior to
restore of existing file similar to rsync. Does not apply attributes
or flags to the top most directory.
The exclude by attribute function is broken on non-Darwin POSIX: linux and freebsd.
This is because those xattrs must be prefixed by a legal namespace. The old xattr
library implicitly appended the user namespace to the xattr, but the current
official go pkg does not (which is just as well).
Also fix the test to remove the discordant old xattr dependency and provide
test cases for both darwin and non-darwin POSIX.
ListObjects has been deprecated since 2016 and ListObjectsV2 with use of
explicit pagination tokens is more performant for large listings as well.
This also mitigates an issue with iDrive E2 where the StartAfter/Marker
is included in the output, leading to duplicate entries. Right now this
causes an exhaustive prune to delete chunks erroneously flagged as
duplicate, destroying the storage.
The prune command can remove redundant chunks (chunks with the same chunk id
but at different subdirectory level). However, if the same chunk appears
mutliple times in the listing returned by the storage, it will be treated as
a redundant chunk and thus removed.
This is useful for upgrading an existing storage to zstd compression or others.
Chunks need to be decompressed and re-compressed during copy anyway. Only
the bit-identical option requires the same compression level
Also fix a typo: compatiable -> compatible
This is mainly to handle the case when a fossil needs to be downloaded.
This happens when a metadata chunk has been marked as a fossil while the
correpsonding snapshot must be reconstructed for determine referenced
chunks.
The storage url is smb://user@server[:port]/share/path. The password can be
set in the environment variable DUPLICACY_SMB_PASSWORD for default storage or
DUPLICACY_<STORAGE_NAME>_SMB_PASSWORD.
This backend is based on https://github.com/hirochachacha/go-smb2. The
previous samba:// backend is just an alias for the disk-based backend with
caching enabled.
This is needed when a metadata chunk has been turned into a fossil. In B2,
the fossil file is the last version of the file with an 'upload' action, so
to download the file, the 'b2_list_file_versions' api is called to find the
file id and then download it using 'b2_download_file_by_id'.
This is needed when a metadata chunk has been turned into a fossil
Zstd compression can be enabled by providing `-zstd` or `-zstd-level <level>`
to `init`, `add`, or `backup`. With `-zstd` the compression level will be
`default`, and with `-zstd-level` the level can be any of `fastest`, `default`,
`better`, or `best`.
This bug leaks a chunk every time files in a revision are listed. Not a big
deal for backup and restore, but it becomes problematic when listing files in
many revisions for commands such check and history.
The version bit should not be set to 1 when encoding a snapshot. Instead,
it must be set to 1 on snapshot creation.
To correctly process old snapshots encoded incorrectly with version bit set
to 1, the first byte of the encoded file list is also checked. If the first
byte is `[`, then it must be an old snapshot, since the file list in the new
snapshot format always starts with a string encoded in msgpack, the first
byte of which can't be `[`.
This option is useful only when erasure coding is enabled. It will
download and re-upload chunks that contain corruption but are
generally recoverable. It can also be used to fix chunks that
are created by 3.0.1 on arm64 machines with wrong hashes.
highwayhash 1.0.1 contains a bug leading to incorrect hashes on arm64 machines.
The 1.0.1 version is retained in github.com/gilbertchen/highwayhash so the hash
can be checked again if a mismatch is detected by 1.0.2.
This commit fixed 2 bugs. The first bug occurs when an incomplete backup
contains a zero-byte file and no chunks. The second bug occurs when the
repository contains only zero-byte files.
