Add ignoring EINTR to syscall, set O_NOATIME on open, ignore
ENOTTY error (unsupported ioctl) to avoid massive spam for
non-exceptional failure on nonsupporting file systems
Don't bother using checked type assertion for stat since we are
guarded by specific build configuration, we should know the
correct type, and if not, panicing is fine.
Despite syscall being deprecated a decade ago, we still need it
for FileInfo and sys/windows still calls it as well.
- Improve error handling of file flags and xattrs. You now get warnings
- file flags on *BSD/Darwin use a mask and combine with the user/superuser
mask to prevent spurious warnings. The mask is not yet implemented as
a preference
- error handling/report for reading special files
- bump to go 1.20 for errors.Join
Move system specific code out of entry so the build continues to work on
Windows. Hard links are not currently supported on Windows, the behavior
is a no-op.
- the xattr handling of namespaces is different on BSD with the
xattr package, the user namespace is implicit
- Don't backup/exclude files with flag nodump on BSD/Darwin
Also, don't attempt xattr ops on special files on the BSD likes.
TODO: Should have a way to allow restore without special files,
otherwise very cumbersome for a regular user.
Fix Linux, Darwin, and other BSD (untested) to allow proper
handling of xattrs with symlinks. On Linux we cannot use the f*
syscalls for symlinks because symlinks cannot be opened.
File flags must be handled differently on darwin and other BSD due
to the lack of the LCHFLAGS syscall on darwin, and the fact that it
is emulated in libc. However, we do have O_SYMLINK on darwin.
Special files are device nodes and named pipes. The necessity of the
former is clear, the latter is debatable.
In order to preserve backward compatibility, the device number is
encoded in the StartChunk/StartOffset fields of the entry.
This tracks inode/device from the stat info and creates backward
compatible snapshots that allow preserving hardlinks. Backwards
compatibility is preserved by saving a virtual inode number index in the
Link field of the file entry. Since this field was previously only used
for symlinks, this won't break old versions. Additionally, the entry
data is cloned so restoration with an old version works.
Current limitations are primarility with restore. They include:
- no command line option to prevent hard link restore
- if a file has the immutable or append only flag it will be set before
hardlinks are restored, so hardlinking will fail.
- if a partial restore includes a hardlink but not the parent
directories the hardlink will fail.
These will be solved by grouping restore of hardlinks together
with file, prior to applying final metadata.
- if a file is changed and is being rewritten by a restore hardlinks are
not preserved.
Basic support for BSD and Darwin style chflags (stat flags). Applies
these flags at the end of file restore.
Supports linux style ioctl_iflags(2) in a 2 step process. Flags that
need to be applied prior to writes such as compress and especially no-COW
are applied immediately upon file open.
The flags format is backwards compatible. An attribute starting with a
null byte is used to store flags in the entry attributes table. With
an old version of duplicacy the restore of this attribute should silently
fail (effectively be ignored).
Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink.
TODO: Tests, possible option to switch off mutable/append prior to
restore of existing file similar to rsync. Does not apply attributes
or flags to the top most directory.
The exclude by attribute function is broken on non-Darwin POSIX: linux and freebsd.
This is because those xattrs must be prefixed by a legal namespace. The old xattr
library implicitly appended the user namespace to the xattr, but the current
official go pkg does not (which is just as well).
Also fix the test to remove the discordant old xattr dependency and provide
test cases for both darwin and non-darwin POSIX.
ListObjects has been deprecated since 2016 and ListObjectsV2 with use of
explicit pagination tokens is more performant for large listings as well.
This also mitigates an issue with iDrive E2 where the StartAfter/Marker
is included in the output, leading to duplicate entries. Right now this
causes an exhaustive prune to delete chunks erroneously flagged as
duplicate, destroying the storage.
The prune command can remove redundant chunks (chunks with the same chunk id
but at different subdirectory level). However, if the same chunk appears
mutliple times in the listing returned by the storage, it will be treated as
a redundant chunk and thus removed.
This is useful for upgrading an existing storage to zstd compression or others.
Chunks need to be decompressed and re-compressed during copy anyway. Only
the bit-identical option requires the same compression level
Also fix a typo: compatiable -> compatible
This is mainly to handle the case when a fossil needs to be downloaded.
This happens when a metadata chunk has been marked as a fossil while the
correpsonding snapshot must be reconstructed for determine referenced
chunks.
The storage url is smb://user@server[:port]/share/path. The password can be
set in the environment variable DUPLICACY_SMB_PASSWORD for default storage or
DUPLICACY_<STORAGE_NAME>_SMB_PASSWORD.
This backend is based on https://github.com/hirochachacha/go-smb2. The
previous samba:// backend is just an alias for the disk-based backend with
caching enabled.
This is needed when a metadata chunk has been turned into a fossil. In B2,
the fossil file is the last version of the file with an 'upload' action, so
to download the file, the 'b2_list_file_versions' api is called to find the
file id and then download it using 'b2_download_file_by_id'.
This is needed when a metadata chunk has been turned into a fossil
Zstd compression can be enabled by providing `-zstd` or `-zstd-level <level>`
to `init`, `add`, or `backup`. With `-zstd` the compression level will be
`default`, and with `-zstd-level` the level can be any of `fastest`, `default`,
`better`, or `best`.
