- Move more function arguments to structs with some convenience
functions.
- Add a flag to enable backup/restore of file flags. It works a little
bit differently depending on system. Since we don't need to do anything
to get file flags on darwin/BSD, it's in the stat buffer so just save it.
For linux we have to open the file and do a
- Add the normalize flag to ReadAttributes. Implementation TBD.
- stub out common xattr file for doc comments
- Move the pref based (repository associated) options back
to init of the backup manager. This is still a bit of an unholy mess but
the existing setup is a little insane.
There are mostly runtime associated settings, like revision, tag, thread
count, quick mode, show stats. Then there are the more fuzzy ones,
like allow failures, patterns. However something like set-owner, mostly
likely that's going to be static for a "repository", same with
exclude by attribute. In any event things that make sense to be
associated with the repository preferences will stay in a backup mananger
initialization time options structure, and no more growing of the
arguments list - go doesn't look good there.
RestoreOptions for the more ephemeral run associated settings.
The Backup routine with it's large argument list stays for now, but
will get revamped if anything new gets added.
Add options to control if xattrs and specials get restored. Add
file flag mask preference. It's a rough interface, but this is a niche
use case.
Refactoring of options passing since golang ergonomics are poor for long
parameter lists.
Add ignoring EINTR to syscall, set O_NOATIME on open, ignore
ENOTTY error (unsupported ioctl) to avoid massive spam for
non-exceptional failure on nonsupporting file systems
Don't bother using checked type assertion for stat since we are
guarded by specific build configuration, we should know the
correct type, and if not, panicing is fine.
Despite syscall being deprecated a decade ago, we still need it
for FileInfo and sys/windows still calls it as well.
- Improve error handling of file flags and xattrs. You now get warnings
- file flags on *BSD/Darwin use a mask and combine with the user/superuser
mask to prevent spurious warnings. The mask is not yet implemented as
a preference
- error handling/report for reading special files
- bump to go 1.20 for errors.Join
Move system specific code out of entry so the build continues to work on
Windows. Hard links are not currently supported on Windows, the behavior
is a no-op.
- the xattr handling of namespaces is different on BSD with the
xattr package, the user namespace is implicit
- Don't backup/exclude files with flag nodump on BSD/Darwin
Also, don't attempt xattr ops on special files on the BSD likes.
TODO: Should have a way to allow restore without special files,
otherwise very cumbersome for a regular user.
Fix Linux, Darwin, and other BSD (untested) to allow proper
handling of xattrs with symlinks. On Linux we cannot use the f*
syscalls for symlinks because symlinks cannot be opened.
File flags must be handled differently on darwin and other BSD due
to the lack of the LCHFLAGS syscall on darwin, and the fact that it
is emulated in libc. However, we do have O_SYMLINK on darwin.
Special files are device nodes and named pipes. The necessity of the
former is clear, the latter is debatable.
In order to preserve backward compatibility, the device number is
encoded in the StartChunk/StartOffset fields of the entry.
This tracks inode/device from the stat info and creates backward
compatible snapshots that allow preserving hardlinks. Backwards
compatibility is preserved by saving a virtual inode number index in the
Link field of the file entry. Since this field was previously only used
for symlinks, this won't break old versions. Additionally, the entry
data is cloned so restoration with an old version works.
Current limitations are primarility with restore. They include:
- no command line option to prevent hard link restore
- if a file has the immutable or append only flag it will be set before
hardlinks are restored, so hardlinking will fail.
- if a partial restore includes a hardlink but not the parent
directories the hardlink will fail.
These will be solved by grouping restore of hardlinks together
with file, prior to applying final metadata.
- if a file is changed and is being rewritten by a restore hardlinks are
not preserved.
Basic support for BSD and Darwin style chflags (stat flags). Applies
these flags at the end of file restore.
Supports linux style ioctl_iflags(2) in a 2 step process. Flags that
need to be applied prior to writes such as compress and especially no-COW
are applied immediately upon file open.
The flags format is backwards compatible. An attribute starting with a
null byte is used to store flags in the entry attributes table. With
an old version of duplicacy the restore of this attribute should silently
fail (effectively be ignored).
Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink.
TODO: Tests, possible option to switch off mutable/append prior to
restore of existing file similar to rsync. Does not apply attributes
or flags to the top most directory.
The exclude by attribute function is broken on non-Darwin POSIX: linux and freebsd.
This is because those xattrs must be prefixed by a legal namespace. The old xattr
library implicitly appended the user namespace to the xattr, but the current
official go pkg does not (which is just as well).
Also fix the test to remove the discordant old xattr dependency and provide
test cases for both darwin and non-darwin POSIX.
ListObjects has been deprecated since 2016 and ListObjectsV2 with use of
explicit pagination tokens is more performant for large listings as well.
This also mitigates an issue with iDrive E2 where the StartAfter/Marker
is included in the output, leading to duplicate entries. Right now this
causes an exhaustive prune to delete chunks erroneously flagged as
duplicate, destroying the storage.
The prune command can remove redundant chunks (chunks with the same chunk id
but at different subdirectory level). However, if the same chunk appears
mutliple times in the listing returned by the storage, it will be treated as
a redundant chunk and thus removed.
This is useful for upgrading an existing storage to zstd compression or others.
Chunks need to be decompressed and re-compressed during copy anyway. Only
the bit-identical option requires the same compression level
Also fix a typo: compatiable -> compatible
