Normalize and add options for backup, add more file flags options

- Move more function arguments to structs with some convenience functions. - Add a flag to enable backup/restore of file flags. It works a little bit differently depending on system. Since we don't need to do anything to get file flags on darwin/BSD, it's in the stat buffer so just save it. For linux we have to open the file and do a - Add the normalize flag to ReadAttributes. Implementation TBD. - stub out common xattr file for doc comments
Refactor the BackupManager options interface some more
2026-01-09 15:14:40 -06:00 · 2023-10-06 22:10:38 -05:00 · 2023-10-06 17:14:04 -05:00 · 2023-10-06 16:41:42 -05:00 · 2023-10-06 16:41:39 -05:00 · 2023-10-06 15:47:50 -05:00
4 changed files with 12 additions and 18 deletions
--- a/duplicacy/duplicacy_main.go
+++ b/duplicacy/duplicacy_main.go
@@ -2287,7 +2287,7 @@ func main() {
 	app.Name = "duplicacy"
 	app.HelpName = "duplicacy"
 	app.Usage = "A new generation cloud backup tool based on lock-free deduplication"
-	app.Version = "3.2.3" + " (" + GitCommit + ")"
+	app.Version = "3.2.2" + " (" + GitCommit + ")"

 	// Exit with code 2 if an invalid command is provided
 	app.CommandNotFound = func(context *cli.Context, command string) {
--- a/go.mod
+++ b/go.mod
@@ -29,7 +29,7 @@ require (
 	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
 	golang.org/x/sys v0.11.0
 	google.golang.org/api v0.21.0
-	storj.io/uplink v1.12.1
+	storj.io/uplink v1.12.0
 )

 require (
@@ -56,7 +56,7 @@ require (
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/satori/go.uuid v1.2.0 // indirect
 	github.com/segmentio/go-env v1.1.0 // indirect
-	github.com/spacemonkeygo/monkit/v3 v3.0.22 // indirect
+	github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230227152157-d00b379de191 // indirect
 	github.com/vaughan0/go-ini v0.0.0-20130923145212-a98ad7ee00ec // indirect
 	github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 // indirect
 	github.com/zeebo/blake3 v0.2.3 // indirect
@@ -71,7 +71,7 @@ require (
 	google.golang.org/genproto v0.0.0-20200409111301-baae70f3302d // indirect
 	google.golang.org/grpc v1.28.1 // indirect
 	google.golang.org/protobuf v1.28.1 // indirect
-	storj.io/common v0.0.0-20230920095429-0ce0a575e6f8 // indirect
+	storj.io/common v0.0.0-20230907123639-5fd0608fd947 // indirect
 	storj.io/drpc v0.0.33 // indirect
 	storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c // indirect
 )
--- a/go.sum
+++ b/go.sum
@@ -140,8 +140,8 @@ github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=
 github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
 github.com/segmentio/go-env v1.1.0 h1:AGJ7OnCx9M5NWpkYPGYELS6III/pFSnAs1GvKWStiEo=
 github.com/segmentio/go-env v1.1.0/go.mod h1:pEKO2ieHe8zF098OMaAHw21SajMuONlnI/vJNB3pB7I=
-github.com/spacemonkeygo/monkit/v3 v3.0.22 h1:4/g8IVItBDKLdVnqrdHZrCVPpIrwDBzl1jrV0IHQHDU=
-github.com/spacemonkeygo/monkit/v3 v3.0.22/go.mod h1:XkZYGzknZwkD0AKUnZaSXhRiVTLCkq7CWVa3IsE72gA=
+github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230227152157-d00b379de191 h1:QVUfVxilbPp8fBJ7701LL/WEUjBSiSxbs9LUaCIe5qM=
+github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230227152157-d00b379de191/go.mod h1:kj1ViJhlyADa7DiA4xVnTuPA46lFKbM7mxQTrXCuJP4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
@@ -195,6 +195,7 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
@@ -293,11 +294,11 @@ honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWh
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
-storj.io/common v0.0.0-20230920095429-0ce0a575e6f8 h1:i+bWPhVnNL6z/TLW3vDZytB6/0bsvJM0a1GhLCxrlxQ=
-storj.io/common v0.0.0-20230920095429-0ce0a575e6f8/go.mod h1:ZmeGPzRb2sm705Nwt/WwuH3e6mliShfvvoUNy1bb9v4=
+storj.io/common v0.0.0-20230907123639-5fd0608fd947 h1:X75A5hX1nFjQH8GIvei4T1LNQTLa++bsDKMxXxfPHE8=
+storj.io/common v0.0.0-20230907123639-5fd0608fd947/go.mod h1:FMVOxf2+SgsmfjxwFCM1MZCKwXis4U7l22M/6nIhIas=
 storj.io/drpc v0.0.33 h1:yCGZ26r66ZdMP0IcTYsj7WDAUIIjzXk6DJhbhvt9FHI=
 storj.io/drpc v0.0.33/go.mod h1:vR804UNzhBa49NOJ6HeLjd2H3MakC1j5Gv8bsOQT6N4=
 storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c h1:or/DtG5uaZpzimL61ahlgAA+MTYn/U3txz4fe+XBFUg=
 storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c/go.mod h1:JCuc3C0gzCJHQ4J6SOx/Yjg+QTpX0D+Fvs5H46FETCk=
-storj.io/uplink v1.12.1 h1:bDc2dI6Q7EXcvPJLZuH9jIOTIf2oKxvW3xKEA+Y5EI0=
-storj.io/uplink v1.12.1/go.mod h1:1+czctHG25pMzcUp4Mds6QnoJ7LvbgYA5d1qlpFFexg=
+storj.io/uplink v1.12.0 h1:rTODjbKRo/lzz5Hp0isjoRfqDcH7kJg6aujD2M9v9Ro=
+storj.io/uplink v1.12.0/go.mod h1:nMAuoWi5AHio+8NQa33VRzCiRg0B0UhYKuT0a0CdXOg=
--- a/src/duplicacy_backupmanager.go
+++ b/src/duplicacy_backupmanager.go
@@ -70,14 +70,6 @@ func (manager *BackupManager) SetCompressionLevel(level int) {
 	manager.config.CompressionLevel = level
 }

-func (manager *BackupManager) Config() *Config {
-	return manager.config
-}
-
-func (manager *BackupManager) SnapshotCache() *FileStorage {
-	return manager.snapshotCache
-}
-
 // CreateBackupManager creates a backup manager using the specified 'storage'.  'snapshotID' is a unique id to
 // identify snapshots created for this repository.  'top' is the top directory of the repository.  'password' is the
 // master key which can be nil if encryption is not enabled.
@@ -103,6 +95,7 @@ func CreateBackupManager(snapshotID string, storage Storage, top string, passwor
 		SnapshotManager: snapshotManager,

 		config:  config,
+		options: *options,
 	}
 	if options != nil {
 		backupManager.options = *options
Author	SHA1	Message	Date
John K. Luebs	bc35037e88	Normalize and add options for backup, add more file flags options - Move more function arguments to structs with some convenience functions. - Add a flag to enable backup/restore of file flags. It works a little bit differently depending on system. Since we don't need to do anything to get file flags on darwin/BSD, it's in the stat buffer so just save it. For linux we have to open the file and do a - Add the normalize flag to ReadAttributes. Implementation TBD. - stub out common xattr file for doc comments	2023-10-06 22:10:38 -05:00
John K. Luebs	f74ce50068	Refactor the BackupManager options interface some more - Move the pref based (repository associated) options back to init of the backup manager. This is still a bit of an unholy mess but the existing setup is a little insane. There are mostly runtime associated settings, like revision, tag, thread count, quick mode, show stats. Then there are the more fuzzy ones, like allow failures, patterns. However something like set-owner, mostly likely that's going to be static for a "repository", same with exclude by attribute. In any event things that make sense to be associated with the repository preferences will stay in a backup mananger initialization time options structure, and no more growing of the arguments list - go doesn't look good there. RestoreOptions for the more ephemeral run associated settings. The Backup routine with it's large argument list stays for now, but will get revamped if anything new gets added.	2023-10-06 17:14:04 -05:00
John K. Luebs	52e8a8280e	Remove deprecated rand functions in tests With requirement for golang 1.20 the deprecated rand functions can be dropped. Do not have to call seed.	2023-10-06 16:41:42 -05:00
John K. Luebs	6a8d3d1c0b	Add metadata and special files restore options Add options to control if xattrs and specials get restored. Add file flag mask preference. It's a rough interface, but this is a niche use case. Refactoring of options passing since golang ergonomics are poor for long parameter lists.	2023-10-06 16:41:39 -05:00
John K. Luebs	9046b12f92	ReadFileFlags on linux minor fix Add ignoring EINTR to syscall, set O_NOATIME on open, ignore ENOTTY error (unsupported ioctl) to avoid massive spam for non-exceptional failure on nonsupporting file systems	2023-10-06 15:47:50 -05:00
John K. Luebs	95ba772e2d	Fix hardlink to symlink naming bug When creating cloned entries of hardlinked symlinks we have to preserve the target link data in the entry obviously.	2023-10-06 06:03:58 -05:00
John K. Luebs	e791bc1f9b	Split out xattr and flags routines to new files	2023-10-06 06:03:57 -05:00
John K. Luebs	fd0f544c04	Cleanup some of the unix code and don't pass ptr to iface to GetOwner Don't bother using checked type assertion for stat since we are guarded by specific build configuration, we should know the correct type, and if not, panicing is fine. Despite syscall being deprecated a decade ago, we still need it for FileInfo and sys/windows still calls it as well.	2023-10-06 05:57:08 -05:00
John K. Luebs	84822200ed	Fix some deprecated functions in windows code	2023-10-06 05:57:08 -05:00
John K. Luebs	9f28a6900d	Break up Restore options into struct	2023-10-06 05:57:08 -05:00
John K. Luebs	42ee20a9e1	Improve error handling and setting of xattr and file flags - Improve error handling of file flags and xattrs. You now get warnings - file flags on *BSD/Darwin use a mask and combine with the user/superuser mask to prevent spurious warnings. The mask is not yet implemented as a preference - error handling/report for reading special files - bump to go 1.20 for errors.Join	2023-10-06 05:57:02 -05:00
John K. Luebs	7df1ba5d82	RestoreMetadata improved error messages	2023-10-05 21:39:41 -05:00
John K. Luebs	60870cdabd	Remove unnecessary fileInfo indirection (it's already interface)	2023-10-05 21:28:36 -05:00
John K. Luebs	1a2896ca8e	Remove obsolete fileEncoder	2023-10-05 16:41:16 -05:00
John K. Luebs	52e511d5f3	Set file attributes before setting mode and owner.	2023-10-05 16:30:55 -05:00
John K. Luebs	1e045c2320	update readme	2023-10-05 13:20:59 -05:00
John K. Luebs	381d4a7159	Factor out hard link code for cross platform build Move system specific code out of entry so the build continues to work on Windows. Hard links are not currently supported on Windows, the behavior is a no-op.	2023-10-05 12:00:30 -05:00
John K. Luebs	c782def49b	Fix and reorganize for build for FreeBSD - the xattr handling of namespaces is different on BSD with the xattr package, the user namespace is implicit - Don't backup/exclude files with flag nodump on BSD/Darwin	2023-10-04 23:55:30 -05:00
John K. Luebs	f931d3e9f1	Don't rewrite existing special file	2023-10-04 22:58:36 -05:00
John K. Luebs	7843df9584	update readme	2023-10-04 19:49:24 -05:00
John K. Luebs	3b50b7ec05	Rename hardlink to hard link (two words)	2023-10-04 13:37:26 -05:00
John K. Luebs	e5a72ddf4f	minor: missing variable assignment in CheckSnapshot, struct-tag	2023-10-04 13:37:26 -05:00
John K. Luebs	de2a5c5fd9	fix broken tests	2023-10-04 13:37:26 -05:00
John K. Luebs	bdbd75f88c	Fix formatting of diagnostic some diagnostic messages	2023-10-04 13:37:26 -05:00
John K. Luebs	ab74d19d2b	Cleanup hardlinking code This moves special file link data to the chunk fields like symlink hardlinks.	2023-10-04 13:37:22 -05:00
John K. Luebs	9755b666b3	tweak the symlink hardlink chunk ids	2023-10-04 04:09:38 -05:00
John K. Luebs	28efe91c3f	Support backup of sockets This only works on Linux. Darwin does not allow mknod of a socket. Have not tested BSD.	2023-10-04 02:53:48 -05:00
John K. Luebs	73ca9794ab	gofmt some files	2023-10-04 02:53:48 -05:00
John K. Luebs	70ea4d3acf	Fix handling of hardlinks and special files Also, don't attempt xattr ops on special files on the BSD likes. TODO: Should have a way to allow restore without special files, otherwise very cumbersome for a regular user.	2023-10-04 02:53:48 -05:00
John K. Luebs	a1a3f3d4cb	Support for hardlinks to symlinks This is a specialized use-case, but it is indeed possible to do this and can be used if xattrs are attached to the symlink.	2023-10-04 02:53:48 -05:00
John K. Luebs	5b40bf3d93	Fix handling of xattrs with symlinks Fix Linux, Darwin, and other BSD (untested) to allow proper handling of xattrs with symlinks. On Linux we cannot use the f* syscalls for symlinks because symlinks cannot be opened. File flags must be handled differently on darwin and other BSD due to the lack of the LCHFLAGS syscall on darwin, and the fact that it is emulated in libc. However, we do have O_SYMLINK on darwin.	2023-10-04 02:53:48 -05:00
John K. Luebs	5087ac738d	Support backup and restore of special files on POSIX style systems Special files are device nodes and named pipes. The necessity of the former is clear, the latter is debatable. In order to preserve backward compatibility, the device number is encoded in the StartChunk/StartOffset fields of the entry.	2023-10-04 02:53:48 -05:00
John K. Luebs	99e4dcae00	Don't overwrite symlinks if file already exists	2023-10-04 02:53:48 -05:00
John K. Luebs	997ff5bdf6	Support backup and restore of hardlinks This tracks inode/device from the stat info and creates backward compatible snapshots that allow preserving hardlinks. Backwards compatibility is preserved by saving a virtual inode number index in the Link field of the file entry. Since this field was previously only used for symlinks, this won't break old versions. Additionally, the entry data is cloned so restoration with an old version works. Current limitations are primarility with restore. They include: - no command line option to prevent hard link restore - if a file has the immutable or append only flag it will be set before hardlinks are restored, so hardlinking will fail. - if a partial restore includes a hardlink but not the parent directories the hardlink will fail. These will be solved by grouping restore of hardlinks together with file, prior to applying final metadata. - if a file is changed and is being rewritten by a restore hardlinks are not preserved.	2023-10-04 02:53:48 -05:00
John K. Luebs	34b1e19278	Initial implementation of file/inode flags (Linux, BSD, darwin) Basic support for BSD and Darwin style chflags (stat flags). Applies these flags at the end of file restore. Supports linux style ioctl_iflags(2) in a 2 step process. Flags that need to be applied prior to writes such as compress and especially no-COW are applied immediately upon file open. The flags format is backwards compatible. An attribute starting with a null byte is used to store flags in the entry attributes table. With an old version of duplicacy the restore of this attribute should silently fail (effectively be ignored). Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink. TODO: Tests, possible option to switch off mutable/append prior to restore of existing file similar to rsync. Does not apply attributes or flags to the top most directory.	2023-10-04 02:53:48 -05:00
John K. Luebs	c1c8af1de9	Increase b2 client max file listing count to 10000 Considerable speed improvement with listing large storage.	2023-10-04 02:53:48 -05:00
John K. Luebs	8b788572c8	Fix exclude_by_attribute feature on POSIX The exclude by attribute function is broken on non-Darwin POSIX: linux and freebsd. This is because those xattrs must be prefixed by a legal namespace. The old xattr library implicitly appended the user namespace to the xattr, but the current official go pkg does not (which is just as well). Also fix the test to remove the discordant old xattr dependency and provide test cases for both darwin and non-darwin POSIX.	2023-10-04 02:53:48 -05:00
John K. Luebs	b592484b54	Use S3 ListObjectsV2 for listing files ListObjects has been deprecated since 2016 and ListObjectsV2 with use of explicit pagination tokens is more performant for large listings as well. This also mitigates an issue with iDrive E2 where the StartAfter/Marker is included in the output, leading to duplicate entries. Right now this causes an exhaustive prune to delete chunks erroneously flagged as duplicate, destroying the storage.	2023-10-04 02:53:48 -05:00