gofmt some files

Fix handling of hardlinks and special files
Also, don't attempt xattr ops on special files on the BSD likes. TODO: Should have a way to allow restore without special files, otherwise very cumbersome for a regular user.
2026-01-04 20:54:44 -06:00 · 2023-10-04 01:23:03 -05:00 · 2023-10-04 01:12:13 -05:00 · 2023-10-04 01:12:11 -05:00 · 2023-10-04 01:11:20 -05:00 · 2023-10-03 16:26:19 -05:00
28 changed files with 877 additions and 3840 deletions
--- a/README.md
+++ b/README.md
@@ -1,45 +1,3 @@
-# Dupluxy
-
-An experimental Duplicacy derivative with improved support for preserving state on UNIX like systems. Produces snapshots compatible with Duplicacy.
-
-NOTE: This project/repository is not affiliated with nor endorsed by Duplicacy, Acrosync or their associated rights holders. This project is open source but is not free/libre software. It is developed and distributed in accordance with the associated LICENSE. Commercial use may require purchase of a license from Acrosync, please contact them if you have any doubts.
-
-## Added Features
-* Support for hard links. Hard links are tracked during local file listing. All linked entries will reuse the same chunk data, so this can give a time and space saving benefit as hard-linked files only need to be packed once. Hard links are supported to everything (regular files, symlinks, special files) except directories.
-* Optional File flags, that is chflags(1) on BSD/Darwin, and ioctl_iflags(2) on Linux. The primary use case is to preserve iflags used by btrfs for no-COW and compression.
-* Optional Special files (character/block devices, FIFOs, and sockets) are preserved along with associated metadata.
-
-## Assorted Changes
-* The S3 backend uses the newer ListObjectsV2 interface originally because of a bug with some providers with the old, obsolete interface, but now because this API is considerably faster on a number of providers tested.
-* B2 client max listing per request increased to 10,000
-* A fix for the exclude_by_attribute feature on BSD/Unix which has been broken upstream for ages.
-
-## Snapshot Format
-The generated preserves snapshots are backward compatible with vanilla versions of duplicacy and also do not increase the encoding size of metadata significantly. Unfortunately duplicacy does not have a formal forward-compatible snapshot versioning system, but that's not too surprising. This does mean that the data encoding is somewhat abusive of the existing format.
-
-### Hard links
-The storage differs for regular files vs. every other target. Entry records contain a `Link` string field for the symlink target. When a likely hard linked file is encountered (`st_nlink > 1`) that entry is marked as a hard link root with the string `"/"` in the `Link` field and it is placed in an array, the index of this array will serve as a link address. Plain duplicacy only uses the `Link` field for symlinks. Files that hard link to this initial file have the index of the array of root files encoded as a base-16 integer into their `Link` field. These entries are placed in the snapshot with valid start/end chunk and offset values and all metadata is cloned so official Duplicacy will recover them as regular files with all metadata, it just will never make hard links.
-For hard links to symlinks and special files, the `Link` isn't used. Instead, since these files never have content the `EndChunk/EndOffset` fields are used. A magic number (-9) is encoded in `EndChunk` for root entries and (-10) for clone/child entries. The `EndOffset` contains the index into the root entry array.
-
-### Special Files
-Duplicacy simply skips special files. Dupluxy does not skip them. The `st_rdev` (device number) for character and block devices is stored with the lower 32-bits in `StartChunk` and the upper 32-bits in `StartOffset`, though no actual supported system uses anything bigger than 32-bits. The packing of this quantity is OS specific, but major, minor numbers are also OS specific.
-
-### File flags
-Files flags are stored in the extended attributes table with a short (2 character) OS specific key prefixed with a null-byte. Duplicacy will try to set these xattrs, however they will be ignored as the name appears to be empty with the initial null-byte.
-
-
-## Motivation
-Arguably system root directories are better preserved in a filesystem image format, however the line becomes blurred for home and data directories the former which tends to become a magnet for all kinds of data layout. This gives the option of a convenient random addressable cloud backup with easy partial restore while also being able to backup a nearly exact replica for use in disaster recovery. Nearly exact, the only metadata not preserved are times other than mtimes and ACLs on BSD-like systems.
-Hard links are a pain and might be better to not exist but in actual use things like git repos and SDKs have a tendency to use them. Often one has no choice but to deal with them, and forgoing preserving them is painful.
-File flags are primarily for the use case of btrfs snapshot backups, specifically with regards to compression and no-COW. The implementation applies certain flags immediately on open so that these flags apply to written blocks.
-Special files serve a couple purposes. Backup of FIFOs and sockets are primarily for preserving metadata since these files have no useful content and can always be created on the fly. The other is support for backup of overlay2 file systems. overlay2 uses character mode dev-nodes for whiteouts in addition to trusted namespace xattrs. Dupluxy should be able to faithfully reproduce overlay2 fs layers.
-
-## Caveats/TODO
-* Improve handling of preferences. There are preferences to enable and disable most features (not hardlinks though) with reasonable defaults but nothing is much documented. Take a look at the generated `.duplicacy/preferences` file.
-* File flags for immutability aren't handled smartly. Specifically immutable and append only files will break badly with hardlinks, since hardlink creation is deferred to after flags application.
-* Some corner cases of replacing existing files with hard links might end up breaking links if not doing a full restore. Again not a pressing use case. For the primary use of disaster recovery of large portions or an entire volume it works fine.
-* Possibly encode ACLs on Mac OS/FreeBSD. On Linux the crappy POSIX 1e ACLs that no one likes to use are picked up in the xattrs for free.
-
 # Duplicacy: A lock-free deduplication cloud backup tool

 Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).
--- a/duplicacy/duplicacy_main.go
+++ b/duplicacy/duplicacy_main.go
@@ -22,7 +22,9 @@ import (

 	"github.com/gilbertchen/cli"

-	duplicacy "github.com/gilbertchen/duplicacy/src"
+	"io/ioutil"
+
+	"github.com/gilbertchen/duplicacy/src"
 )

 const (
@@ -314,7 +316,7 @@ func configRepository(context *cli.Context, init bool) {
 			// write real path into .duplicacy file inside repository
 			duplicacyFileName := path.Join(repository, duplicacy.DUPLICACY_FILE)
 			d1 := []byte(preferencePath)
-			err = os.WriteFile(duplicacyFileName, d1, 0644)
+			err = ioutil.WriteFile(duplicacyFileName, d1, 0644)
 			if err != nil {
 				duplicacy.LOG_ERROR("REPOSITORY_PATH", "Failed to write %s file inside repository  %v", duplicacyFileName, err)
 				return
@@ -703,7 +705,7 @@ func changePassword(context *cli.Context) {
 	}

 	configPath := path.Join(duplicacy.GetDuplicacyPreferencePath(), "config")
-	err = os.WriteFile(configPath, description, 0600)
+	err = ioutil.WriteFile(configPath, description, 0600)
 	if err != nil {
 		duplicacy.LOG_ERROR("CONFIG_SAVE", "Failed to save the old config to %s: %v", configPath, err)
 		return
@@ -787,17 +789,7 @@ func backupRepository(context *cli.Context) {
 	uploadRateLimit := context.Int("limit-rate")
 	enumOnly := context.Bool("enum-only")
 	storage.SetRateLimits(0, uploadRateLimit)
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password,
-		&duplicacy.BackupManagerOptions{
-			NobackupFile:       preference.NobackupFile,
-			FiltersFile:        preference.FiltersFile,
-			ExcludeByAttribute: preference.ExcludeByAttribute,
-			ExcludeXattrs:      preference.ExcludeXattrs,
-			NormalizeXattrs:    preference.NormalizeXattrs,
-			IncludeFileFlags:   preference.IncludeFileFlags,
-			IncludeSpecials:    preference.IncludeSpecials,
-			FileFlagsMask:      uint32(preference.FileFlagsMask),
-		})
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile, preference.FiltersFile, preference.ExcludeByAttribute)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
@@ -858,6 +850,14 @@ func restoreRepository(context *cli.Context) {
 		password = duplicacy.GetPassword(*preference, "password", "Enter storage password:", false, false)
 	}

+	quickMode := !context.Bool("hash")
+	overwrite := context.Bool("overwrite")
+	deleteMode := context.Bool("delete")
+	setOwner := !context.Bool("ignore-owner")
+
+	showStatistics := context.Bool("stats")
+	persist := context.Bool("persist")
+
 	var patterns []string
 	for _, pattern := range context.Args() {

@@ -881,38 +881,13 @@ func restoreRepository(context *cli.Context) {
 	duplicacy.LOG_INFO("SNAPSHOT_FILTER", "Loaded %d include/exclude pattern(s)", len(patterns))

 	storage.SetRateLimits(context.Int("limit-rate"), 0)
-
-	excludeOwner := preference.ExcludeOwner
-	// TODO: for backward compat, eventually make them all overridable?
-	if context.IsSet("ignore-owner") {
-		excludeOwner = context.Bool("ignore-owner")
-	}
-
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password,
-		&duplicacy.BackupManagerOptions{
-			NobackupFile:       preference.NobackupFile,
-			FiltersFile:        preference.FiltersFile,
-			ExcludeByAttribute: preference.ExcludeByAttribute,
-			SetOwner:           excludeOwner,
-			ExcludeXattrs:      preference.ExcludeXattrs,
-			NormalizeXattrs:    preference.NormalizeXattrs,
-			IncludeSpecials:    preference.IncludeSpecials,
-			FileFlagsMask:      uint32(preference.FileFlagsMask),
-		})
-
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile, preference.FiltersFile, preference.ExcludeByAttribute)
 	duplicacy.SavePassword(*preference, "password", password)

 	loadRSAPrivateKey(context.String("key"), context.String("key-passphrase"), preference, backupManager, false)

 	backupManager.SetupSnapshotCache(preference.Name)
-	failed := backupManager.Restore(repository, revision, &duplicacy.RestoreOptions{
-		InPlace:        true,
-		QuickMode:      !context.Bool("hash"),
-		Overwrite:      context.Bool("overwrite"),
-		DeleteMode:     context.Bool("delete"),
-		ShowStatistics: context.Bool("stats"),
-		AllowFailures:  context.Bool("persist"),
-	})
+	failed := backupManager.Restore(repository, revision, true, quickMode, threads, overwrite, deleteMode, setOwner, showStatistics, patterns, persist)
 	if failed > 0 {
 		duplicacy.LOG_ERROR("RESTORE_FAIL", "%d file(s) were not restored correctly", failed)
 		return
@@ -952,8 +927,7 @@ func listSnapshots(context *cli.Context) {
 	tag := context.String("t")
 	revisions := getRevisions(context)

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password,
-		&duplicacy.BackupManagerOptions{ExcludeByAttribute: preference.ExcludeByAttribute})
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, "", "", preference.ExcludeByAttribute)
 	duplicacy.SavePassword(*preference, "password", password)

 	id := preference.SnapshotID
@@ -1009,7 +983,7 @@ func checkSnapshots(context *cli.Context) {
 	tag := context.String("t")
 	revisions := getRevisions(context)

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, nil)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, "", "", false)
 	duplicacy.SavePassword(*preference, "password", password)

 	loadRSAPrivateKey(context.String("key"), context.String("key-passphrase"), preference, backupManager, false)
@@ -1069,7 +1043,8 @@ func printFile(context *cli.Context) {
 		snapshotID = context.String("id")
 	}

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, nil)
+
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, "", "", false)
 	duplicacy.SavePassword(*preference, "password", password)

 	loadRSAPrivateKey(context.String("key"), context.String("key-passphrase"), preference, backupManager, false)
@@ -1127,14 +1102,13 @@ func diff(context *cli.Context) {
 	}

 	compareByHash := context.Bool("hash")
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, nil)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, "", "", false)
 	duplicacy.SavePassword(*preference, "password", password)

 	loadRSAPrivateKey(context.String("key"), context.String("key-passphrase"), preference, backupManager, false)

 	backupManager.SetupSnapshotCache(preference.Name)
-	backupManager.SnapshotManager.Diff(repository, snapshotID, revisions, path, compareByHash,
-		duplicacy.NewListFilesOptions(preference))
+	backupManager.SnapshotManager.Diff(repository, snapshotID, revisions, path, compareByHash, preference.NobackupFile, preference.FiltersFile, preference.ExcludeByAttribute)

 	runScript(context, preference.Name, "post")
 }
@@ -1173,7 +1147,7 @@ func showHistory(context *cli.Context) {

 	revisions := getRevisions(context)
 	showLocalHash := context.Bool("hash")
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, nil)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, "", "", false)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
@@ -1236,7 +1210,7 @@ func pruneSnapshots(context *cli.Context) {
 		os.Exit(ArgumentExitCode)
 	}

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, nil)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, "", "", false)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
@@ -1281,7 +1255,7 @@ func copySnapshots(context *cli.Context) {
 		sourcePassword = duplicacy.GetPassword(*source, "password", "Enter source storage password:", false, false)
 	}

-	sourceManager := duplicacy.CreateBackupManager(source.SnapshotID, sourceStorage, repository, sourcePassword, nil)
+	sourceManager := duplicacy.CreateBackupManager(source.SnapshotID, sourceStorage, repository, sourcePassword, "", "", false)
 	sourceManager.SetupSnapshotCache(source.Name)
 	duplicacy.SavePassword(*source, "password", sourcePassword)

@@ -1316,7 +1290,7 @@ func copySnapshots(context *cli.Context) {
 	destinationStorage.SetRateLimits(0, context.Int("upload-limit-rate"))

 	destinationManager := duplicacy.CreateBackupManager(destination.SnapshotID, destinationStorage, repository,
-		destinationPassword, nil)
+		                                                  destinationPassword, "", "", false)
 	duplicacy.SavePassword(*destination, "password", destinationPassword)
 	destinationManager.SetupSnapshotCache(destination.Name)

@@ -1590,6 +1564,7 @@ func main() {
 					Usage:    "the maximum number of entries kept in memory (defaults to 1M)",
 					Argument: "<number>",
 				},
+
 			},
 			Usage:     "Save a snapshot of the repository to the storage",
 			ArgsUsage: " ",
@@ -2287,7 +2262,7 @@ func main() {
 	app.Name = "duplicacy"
 	app.HelpName = "duplicacy"
 	app.Usage = "A new generation cloud backup tool based on lock-free deduplication"
-	app.Version = "3.2.3" + " (" + GitCommit + ")"
+	app.Version = "3.2.1" + " (" + GitCommit + ")"

 	// Exit with code 2 if an invalid command is provided
 	app.CommandNotFound = func(context *cli.Context, command string) {
--- a/dupluxy/dupluxy.go
+++ b/dupluxy/dupluxy.go
--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,6 @@
-module github.com/dupluxy/dupluxy
+module github.com/gilbertchen/duplicacy

-go 1.20
+go 1.19

 require (
 	cloud.google.com/go v0.38.0
@@ -22,14 +22,13 @@ require (
 	github.com/minio/highwayhash v1.0.2
 	github.com/ncw/swift/v2 v2.0.1
 	github.com/pkg/sftp v1.11.0
-	github.com/pkg/xattr v0.4.9
+	github.com/pkg/xattr v0.4.1
 	github.com/vmihailenco/msgpack v4.0.4+incompatible
 	golang.org/x/crypto v0.12.0
 	golang.org/x/net v0.10.0
 	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
-	golang.org/x/sys v0.11.0
 	google.golang.org/api v0.21.0
-	storj.io/uplink v1.12.1
+	storj.io/uplink v1.12.0
 )

 require (
@@ -56,7 +55,7 @@ require (
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/satori/go.uuid v1.2.0 // indirect
 	github.com/segmentio/go-env v1.1.0 // indirect
-	github.com/spacemonkeygo/monkit/v3 v3.0.22 // indirect
+	github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230227152157-d00b379de191 // indirect
 	github.com/vaughan0/go-ini v0.0.0-20130923145212-a98ad7ee00ec // indirect
 	github.com/vivint/infectious v0.0.0-20200605153912-25a574ae18a3 // indirect
 	github.com/zeebo/blake3 v0.2.3 // indirect
@@ -64,6 +63,7 @@ require (
 	go.opencensus.io v0.22.3 // indirect
 	golang.org/x/mod v0.10.0 // indirect
 	golang.org/x/sync v0.3.0 // indirect
+	golang.org/x/sys v0.11.0 // indirect
 	golang.org/x/term v0.11.0 // indirect
 	golang.org/x/text v0.12.0 // indirect
 	golang.org/x/tools v0.9.1 // indirect
@@ -71,7 +71,7 @@ require (
 	google.golang.org/genproto v0.0.0-20200409111301-baae70f3302d // indirect
 	google.golang.org/grpc v1.28.1 // indirect
 	google.golang.org/protobuf v1.28.1 // indirect
-	storj.io/common v0.0.0-20230920095429-0ce0a575e6f8 // indirect
+	storj.io/common v0.0.0-20230907123639-5fd0608fd947 // indirect
 	storj.io/drpc v0.0.33 // indirect
 	storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c // indirect
 )
--- a/go.sum
+++ b/go.sum
@@ -128,8 +128,6 @@ github.com/pkg/sftp v1.11.0 h1:4Zv0OGbpkg4yNuUtH0s8rvoYxRCNyT29NVUo6pgPmxI=
 github.com/pkg/sftp v1.11.0/go.mod h1:lYOWFsE0bwd1+KfKJaKeuokY15vzFx25BLbzYYoAxZI=
 github.com/pkg/xattr v0.4.1 h1:dhclzL6EqOXNaPDWqoeb9tIxATfBSmjqL0b4DpSjwRw=
 github.com/pkg/xattr v0.4.1/go.mod h1:W2cGD0TBEus7MkUgv0tNZ9JutLtVO3cXu+IBRuHqnFs=
-github.com/pkg/xattr v0.4.9 h1:5883YPCtkSd8LFbs13nXplj9g9tlrwoJRjgpgMu1/fE=
-github.com/pkg/xattr v0.4.9/go.mod h1:di8WF84zAKk8jzR1UBTEWh9AUlIZZ7M/JNt8e9B6ktU=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
@@ -140,8 +138,8 @@ github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=
 github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
 github.com/segmentio/go-env v1.1.0 h1:AGJ7OnCx9M5NWpkYPGYELS6III/pFSnAs1GvKWStiEo=
 github.com/segmentio/go-env v1.1.0/go.mod h1:pEKO2ieHe8zF098OMaAHw21SajMuONlnI/vJNB3pB7I=
-github.com/spacemonkeygo/monkit/v3 v3.0.22 h1:4/g8IVItBDKLdVnqrdHZrCVPpIrwDBzl1jrV0IHQHDU=
-github.com/spacemonkeygo/monkit/v3 v3.0.22/go.mod h1:XkZYGzknZwkD0AKUnZaSXhRiVTLCkq7CWVa3IsE72gA=
+github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230227152157-d00b379de191 h1:QVUfVxilbPp8fBJ7701LL/WEUjBSiSxbs9LUaCIe5qM=
+github.com/spacemonkeygo/monkit/v3 v3.0.20-0.20230227152157-d00b379de191/go.mod h1:kj1ViJhlyADa7DiA4xVnTuPA46lFKbM7mxQTrXCuJP4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
@@ -195,6 +193,7 @@ golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn
 golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
@@ -227,7 +226,6 @@ golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220408201424-a24fb2fb8a0f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.11.0 h1:eG7RXZHdqOJ1i+0lgLgCpSXAp6M3LYlAo6osgSi0xOM=
 golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -293,11 +291,11 @@ honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWh
 honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
 rsc.io/pdf v0.1.1/go.mod h1:n8OzWcQ6Sp37PL01nO98y4iUCRdTGarVfzxY20ICaU4=
-storj.io/common v0.0.0-20230920095429-0ce0a575e6f8 h1:i+bWPhVnNL6z/TLW3vDZytB6/0bsvJM0a1GhLCxrlxQ=
-storj.io/common v0.0.0-20230920095429-0ce0a575e6f8/go.mod h1:ZmeGPzRb2sm705Nwt/WwuH3e6mliShfvvoUNy1bb9v4=
+storj.io/common v0.0.0-20230907123639-5fd0608fd947 h1:X75A5hX1nFjQH8GIvei4T1LNQTLa++bsDKMxXxfPHE8=
+storj.io/common v0.0.0-20230907123639-5fd0608fd947/go.mod h1:FMVOxf2+SgsmfjxwFCM1MZCKwXis4U7l22M/6nIhIas=
 storj.io/drpc v0.0.33 h1:yCGZ26r66ZdMP0IcTYsj7WDAUIIjzXk6DJhbhvt9FHI=
 storj.io/drpc v0.0.33/go.mod h1:vR804UNzhBa49NOJ6HeLjd2H3MakC1j5Gv8bsOQT6N4=
 storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c h1:or/DtG5uaZpzimL61ahlgAA+MTYn/U3txz4fe+XBFUg=
 storj.io/picobuf v0.0.2-0.20230906122608-c4ba17033c6c/go.mod h1:JCuc3C0gzCJHQ4J6SOx/Yjg+QTpX0D+Fvs5H46FETCk=
-storj.io/uplink v1.12.1 h1:bDc2dI6Q7EXcvPJLZuH9jIOTIf2oKxvW3xKEA+Y5EI0=
-storj.io/uplink v1.12.1/go.mod h1:1+czctHG25pMzcUp4Mds6QnoJ7LvbgYA5d1qlpFFexg=
+storj.io/uplink v1.12.0 h1:rTODjbKRo/lzz5Hp0isjoRfqDcH7kJg6aujD2M9v9Ro=
+storj.io/uplink v1.12.0/go.mod h1:nMAuoWi5AHio+8NQa33VRzCiRg0B0UhYKuT0a0CdXOg=
--- a/src/duplicacy_backupmanager.go
+++ b/src/duplicacy_backupmanager.go
@@ -7,9 +7,9 @@ package duplicacy
 import (
 	"bytes"
 	"encoding/hex"
+	"encoding/json"
 	"fmt"
 	"io"
-	"math"
 	"os"
 	"path"
 	"path/filepath"
@@ -26,6 +26,7 @@ import (

 // BackupManager performs the two major operations, backup and restore, and passes other operations, mostly related to
 // snapshot management, to the snapshot manager.
+
 type BackupManager struct {
 	snapshotID string  // Unique id for each repository
 	storage    Storage // the storage for storing backups
@@ -34,34 +35,14 @@ type BackupManager struct {
 	snapshotCache   *FileStorage     // for copies of chunks needed by snapshots

 	config *Config // contains a number of options
-	options BackupManagerOptions
+
+	nobackupFile       string // don't backup directory when this file name is found
+	filtersFile        string // the path to the filters file
+	excludeByAttribute bool   // don't backup file based on file attribute

 	cachePath string
 }

-type BackupManagerOptions struct {
-	NobackupFile       string // don't backup directory when this file name is found
-	FiltersFile        string // the path to the filters file
-	ExcludeByAttribute bool   // don't backup file based on file attribute
-	SetOwner           bool
-	ExcludeXattrs      bool
-	NormalizeXattrs    bool
-	IncludeFileFlags   bool
-	IncludeSpecials    bool
-	FileFlagsMask      uint32
-}
-
-type RestoreOptions struct {
-	Threads        int
-	Patterns       []string
-	InPlace        bool
-	QuickMode      bool
-	Overwrite      bool
-	DeleteMode     bool
-	ShowStatistics bool
-	AllowFailures  bool
-}
-
 func (manager *BackupManager) SetDryRun(dryRun bool) {
 	manager.config.dryRun = dryRun
 }
@@ -70,19 +51,10 @@ func (manager *BackupManager) SetCompressionLevel(level int) {
 	manager.config.CompressionLevel = level
 }

-func (manager *BackupManager) Config() *Config {
-	return manager.config
-}
-
-func (manager *BackupManager) SnapshotCache() *FileStorage {
-	return manager.snapshotCache
-}
-
 // CreateBackupManager creates a backup manager using the specified 'storage'.  'snapshotID' is a unique id to
 // identify snapshots created for this repository.  'top' is the top directory of the repository.  'password' is the
 // master key which can be nil if encryption is not enabled.
-func CreateBackupManager(snapshotID string, storage Storage, top string, password string,
-	options *BackupManagerOptions) *BackupManager {
+func CreateBackupManager(snapshotID string, storage Storage, top string, password string, nobackupFile string, filtersFile string, excludeByAttribute bool) *BackupManager {

 	config, _, err := DownloadConfig(storage, password)
 	if err != nil {
@@ -103,9 +75,12 @@ func CreateBackupManager(snapshotID string, storage Storage, top string, passwor
 		SnapshotManager: snapshotManager,

 		config: config,
-	}
-	if options != nil {
-		backupManager.options = *options
+
+		nobackupFile: nobackupFile,
+
+		filtersFile: filtersFile,
+
+		excludeByAttribute: excludeByAttribute,
 	}

 	if IsDebugging() {
@@ -156,7 +131,8 @@ func (manager *BackupManager) SetupSnapshotCache(storageName string) bool {
 func (manager *BackupManager) Backup(top string, quickMode bool, threads int, tag string,
 	showStatistics bool, shadowCopy bool, shadowCopyTimeout int, enumOnly bool, metadataChunkSize int, maximumInMemoryEntries int) bool {

-	top, err := filepath.Abs(top)
+	var err error
+	top, err = filepath.Abs(top)
 	if err != nil {
 		LOG_ERROR("REPOSITORY_ERR", "Failed to obtain the absolute path of the repository: %v", err)
 		return false
@@ -177,7 +153,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 		LOG_INFO("BACKUP_KEY", "RSA encryption is enabled")
 	}

-	if manager.options.ExcludeByAttribute {
+	if manager.excludeByAttribute {
 		LOG_INFO("BACKUP_EXCLUDE", "Exclude files with no-backup attributes")
 	}

@@ -262,16 +238,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 	go func() {
 		// List local files
 		defer CatchLogException()
-		localSnapshot.ListLocalFiles(shadowTop, localListingChannel, &skippedDirectories, &skippedFiles,
-			&ListFilesOptions{
-				NoBackupFile:       manager.options.NobackupFile,
-				FiltersFile:        manager.options.FiltersFile,
-				ExcludeByAttribute: manager.options.ExcludeByAttribute,
-				ExcludeXattrs:      manager.options.ExcludeXattrs,
-				NormalizeXattr:     manager.options.NormalizeXattrs,
-				IncludeFileFlags:   manager.options.IncludeFileFlags,
-				IncludeSpecials:    manager.options.IncludeSpecials,
-			})
+		localSnapshot.ListLocalFiles(shadowTop, manager.nobackupFile, manager.filtersFile, manager.excludeByAttribute, localListingChannel, &skippedDirectories, &skippedFiles)
 	}()

 	go func() {
@@ -656,34 +623,21 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 }

 // Restore downloads the specified snapshot, compares it with what's on the repository, and then downloads
-// files that are different.'QuickMode' will bypass files with unchanged sizes and timestamps. 'DeleteMode' will
-// remove local files that don't exist in the snapshot. 'Patterns' is used to include/exclude certain files.
-func (manager *BackupManager) Restore(top string, revision int, options *RestoreOptions) int {
-	if options.Threads < 1 {
-		options.Threads = 1
-	}
-
-	patterns := options.Patterns
-
-	overwrite := options.Overwrite
-	allowFailures := options.AllowFailures
-
-	metadataOptions := RestoreMetadataOptions{
-		SetOwner:         manager.options.SetOwner,
-		ExcludeXattrs:    manager.options.ExcludeXattrs,
-		NormalizeXattrs:  manager.options.NormalizeXattrs,
-		IncludeFileFlags: manager.options.IncludeFileFlags,
-		FileFlagsMask:    manager.options.FileFlagsMask,
-	}
+// files that are different. 'base' is a directory that contains files at a different revision which can
+// serve as a local cache to avoid download chunks available locally.  It is perfectly ok for 'base' to be
+// the same as 'top'.  'quickMode' will bypass files with unchanged sizes and timestamps.  'deleteMode' will
+// remove local files that don't exist in the snapshot. 'patterns' is used to include/exclude certain files.
+func (manager *BackupManager) Restore(top string, revision int, inPlace bool, quickMode bool, threads int, overwrite bool,
+	deleteMode bool, setOwner bool, showStatistics bool, patterns []string, allowFailures bool) int {

 	startTime := time.Now().Unix()

 	LOG_DEBUG("RESTORE_PARAMETERS", "top: %s, revision: %d, in-place: %t, quick: %t, delete: %t",
-		top, revision, options.InPlace, options.QuickMode, options.DeleteMode)
+		top, revision, inPlace, quickMode, deleteMode)

 	if !strings.HasPrefix(GetDuplicacyPreferencePath(), top) {
 		LOG_INFO("RESTORE_INPLACE", "Forcing in-place mode with a non-default preference path")
-		options.InPlace = true
+		inPlace = true
 	}

 	if len(patterns) > 0 {
@@ -725,23 +679,13 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore

 	localListingChannel := make(chan *Entry)
 	remoteListingChannel := make(chan *Entry)
-	chunkOperator := CreateChunkOperator(manager.config, manager.storage, manager.snapshotCache, options.ShowStatistics,
-		false, options.Threads, allowFailures)
+	chunkOperator := CreateChunkOperator(manager.config, manager.storage, manager.snapshotCache, showStatistics, false, threads, allowFailures)

 	LOG_INFO("RESTORE_INDEXING", "Indexing %s", top)
 	go func() {
 		// List local files
 		defer CatchLogException()
-		localSnapshot.ListLocalFiles(top, localListingChannel, nil, nil,
-			&ListFilesOptions{
-				NoBackupFile:       manager.options.NobackupFile,
-				FiltersFile:        manager.options.FiltersFile,
-				ExcludeByAttribute: manager.options.ExcludeByAttribute,
-				ExcludeXattrs:      manager.options.ExcludeXattrs,
-				NormalizeXattr:     manager.options.NormalizeXattrs,
-				IncludeFileFlags:   manager.options.IncludeFileFlags,
-				IncludeSpecials:    manager.options.IncludeSpecials,
-			})
+		localSnapshot.ListLocalFiles(top, manager.nobackupFile, manager.filtersFile, manager.excludeByAttribute, localListingChannel, nil, nil)
 	}()

 	remoteSnapshot := manager.SnapshotManager.DownloadSnapshot(manager.snapshotID, revision)
@@ -766,23 +710,23 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 	var hardLinkTable []hardLinkEntry
 	var hardLinks []*Entry

-	restoreHardLink := func(entry *Entry, fullPath string) bool {
-		if entry.IsHardLinkRoot() {
+	restoreHardlink := func(entry *Entry, fullPath string) bool {
+		if entry.IsHardlinkRoot() {
 			hardLinkTable[len(hardLinkTable)-1].willExist = true
-		} else if entry.IsHardLinkChild() {
-			i, err := entry.GetHardLinkId()
+		} else if entry.IsHardlinkedFrom() {
+			i, err := entry.GetHardlinkId()
 			if err != nil {
-				LOG_ERROR("RESTORE_HARDLINK", "Decode error for hard link entry %s: %v", entry.Path, err)
+				LOG_ERROR("RESTORE_HARDLINK", "Decode error for hardlinked entry %s, %v", entry.Path, err)
 				return false
 			}
 			if !hardLinkTable[i].willExist {
 				hardLinkTable[i] = hardLinkEntry{entry, true}
 			} else {
 				sourcePath := joinPath(top, hardLinkTable[i].entry.Path)
+				LOG_INFO("RESTORE_HARDLINK", "Hard linking %s to %s", fullPath, sourcePath)
 				if err := MakeHardlink(sourcePath, fullPath); err != nil {
-					LOG_ERROR("RESTORE_HARDLINK", "Failed to create hard link %s to %s: %v", fullPath, sourcePath, err)
+					LOG_ERROR("RESTORE_HARDLINK", "Failed to create hard link %s to %s %v", fullPath, sourcePath, err)
 				}
-				LOG_TRACE("DOWNLOAD_DONE", "Hard linked %s to %s", entry.Path, hardLinkTable[i].entry.Path)
 				return true
 			}
 		}
@@ -791,7 +735,7 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore

 	for remoteEntry := range remoteListingChannel {

-		if remoteEntry.IsHardLinkRoot() {
+		if remoteEntry.IsHardlinkRoot() {
 			hardLinkTable = append(hardLinkTable, hardLinkEntry{remoteEntry, false})
 		}

@@ -820,7 +764,7 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 		}

 		if compareResult == 0 {
-			if options.QuickMode && localEntry.IsFile() && localEntry.IsSameAs(remoteEntry) {
+			if quickMode && localEntry.IsFile() && localEntry.IsSameAs(remoteEntry) {
 				LOG_TRACE("RESTORE_SKIP", "File %s unchanged (by size and timestamp)", localEntry.Path)
 				skippedFileSize += localEntry.Size
 				skippedFileCount++
@@ -837,8 +781,8 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 				if stat.Mode()&os.ModeSymlink != 0 {
 					isRegular, link, err := Readlink(fullPath)
 					if err == nil && link == remoteEntry.Link && !isRegular {
-						remoteEntry.RestoreMetadata(fullPath, stat, metadataOptions)
-						if remoteEntry.IsHardLinkRoot() {
+						remoteEntry.RestoreMetadata(fullPath, nil, setOwner)
+						if remoteEntry.IsHardlinkRoot() {
 							hardLinkTable[len(hardLinkTable)-1].willExist = true
 						}
 						continue
@@ -854,7 +798,7 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 				os.Remove(fullPath)
 			}

-			if restoreHardLink(remoteEntry, fullPath) {
+			if restoreHardlink(remoteEntry, fullPath) {
 				continue
 			}

@@ -862,9 +806,9 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 				LOG_ERROR("RESTORE_SYMLINK", "Can't create symlink %s: %v", remoteEntry.Path, err)
 				return 0
 			}
-			remoteEntry.RestoreMetadata(fullPath, nil, metadataOptions)
-			LOG_TRACE("DOWNLOAD_DONE", "Symlink %s updated", remoteEntry.Path)
+			remoteEntry.RestoreMetadata(fullPath, nil, setOwner)

+			LOG_TRACE("DOWNLOAD_DONE", "Symlink %s updated", remoteEntry.Path)
 		} else if remoteEntry.IsDir() {

 			stat, err := os.Stat(fullPath)
@@ -883,21 +827,15 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 					return 0
 				}
 			}
-			if metadataOptions.IncludeFileFlags {
-				err = remoteEntry.RestoreEarlyDirFlags(fullPath, manager.options.FileFlagsMask)
-				if err != nil {
-					LOG_WARN("DOWNLOAD_FLAGS", "Failed to set early file flags on %s: %v", fullPath, err)
-				}
-			}
+			remoteEntry.RestoreEarlyDirFlags(fullPath)
 			directoryEntries = append(directoryEntries, remoteEntry)
-		} else if remoteEntry.IsSpecial() && manager.options.IncludeSpecials {
+		} else if remoteEntry.IsSpecial() {
 			if stat, _ := os.Lstat(fullPath); stat != nil {
 				if remoteEntry.IsSameSpecial(stat) {
-					remoteEntry.RestoreMetadata(fullPath, nil, metadataOptions)
-					if remoteEntry.IsHardLinkRoot() {
+					remoteEntry.RestoreMetadata(fullPath, nil, setOwner)
+					if remoteEntry.IsHardlinkRoot() {
 						hardLinkTable[len(hardLinkTable)-1].willExist = true
 					}
-					continue
 				}
 				if !overwrite {
 					LOG_WERROR(allowFailures, "DOWNLOAD_OVERWRITE",
@@ -907,24 +845,22 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 				os.Remove(fullPath)
 			}

-			if restoreHardLink(remoteEntry, fullPath) {
+			if restoreHardlink(remoteEntry, fullPath) {
 				continue
 			}

 			if err := remoteEntry.RestoreSpecial(fullPath); err != nil {
-				LOG_ERROR("RESTORE_SPECIAL", "Failed to restore special file %s: %v", fullPath, err)
+				LOG_ERROR("RESTORE_SPECIAL", "Unable to restore special file %s: %v", remoteEntry.Path, err)
 				return 0
 			}
-			remoteEntry.RestoreMetadata(fullPath, nil, metadataOptions)
-			LOG_TRACE("DOWNLOAD_DONE", "Special %s %s restored", remoteEntry.Path, remoteEntry.FmtSpecial())
-
+			remoteEntry.RestoreMetadata(fullPath, nil, setOwner)
 		} else {
-			if remoteEntry.IsHardLinkRoot() {
+			if remoteEntry.IsHardlinkRoot() {
 				hardLinkTable[len(hardLinkTable)-1].willExist = true
-			} else if remoteEntry.IsHardLinkChild() {
-				i, err := remoteEntry.GetHardLinkId()
+			} else if remoteEntry.IsHardlinkedFrom() {
+				i, err := remoteEntry.GetHardlinkId()
 				if err != nil {
-					LOG_ERROR("RESTORE_HARDLINK", "Decode error for hard link entry %s: %v", remoteEntry.Path, err)
+					LOG_ERROR("RESTORE_HARDLINK", "Decode error for hardlinked entry %s, %v", remoteEntry.Path, err)
 					return 0
 				}
 				if !hardLinkTable[i].willExist {
@@ -989,7 +925,7 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 		fullPath := joinPath(top, file.Path)
 		stat, _ := os.Stat(fullPath)
 		if stat != nil {
-			if options.QuickMode {
+			if quickMode {
 				if file.IsSameAsFileInfo(stat) {
 					LOG_TRACE("RESTORE_SKIP", "File %s unchanged (by size and timestamp)", file.Path)
 					skippedFileSize += file.Size
@@ -1021,8 +957,8 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 			}
 			newFile.Close()

-			file.RestoreMetadata(fullPath, nil, metadataOptions)
-			if !options.ShowStatistics {
+			file.RestoreMetadata(fullPath, nil, setOwner)
+			if !showStatistics {
 				LOG_INFO("DOWNLOAD_DONE", "Downloaded %s (0)", file.Path)
 				downloadedFileSize += file.Size
 				downloadedFiles = append(downloadedFiles, file)
@@ -1031,13 +967,8 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 			continue
 		}

-		fileFlagsMask := metadataOptions.FileFlagsMask
-		if !metadataOptions.IncludeFileFlags {
-			fileFlagsMask = math.MaxUint32
-		}
-		downloaded, err := manager.RestoreFile(chunkDownloader, chunkMaker, file, top, options.InPlace, overwrite,
-			options.ShowStatistics, totalFileSize, downloadedFileSize, startDownloadingTime, allowFailures,
-			fileFlagsMask)
+		downloaded, err := manager.RestoreFile(chunkDownloader, chunkMaker, file, top, inPlace, overwrite, showStatistics,
+			totalFileSize, downloadedFileSize, startDownloadingTime, allowFailures)
 		if err != nil {
 			// RestoreFile returned an error; if allowFailures is false RestoerFile would error out and not return so here
 			// we just need to show a warning
@@ -1056,12 +987,12 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 			skippedFileSize += file.Size
 			skippedFileCount++
 		}
-		file.RestoreMetadata(fullPath, nil, metadataOptions)
+		file.RestoreMetadata(fullPath, nil, setOwner)
 	}

 	for _, linkEntry := range hardLinks {

-		i, _ := linkEntry.GetHardLinkId()
+		i, _ := linkEntry.GetHardlinkId()
 		sourcePath := joinPath(top, hardLinkTable[i].entry.Path)
 		fullPath := joinPath(top, linkEntry.Path)

@@ -1084,14 +1015,14 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 			os.Remove(fullPath)
 		}

+		LOG_DEBUG("RESTORE_HARDLINK", "Hard linking %s to %s", fullPath, sourcePath)
 		if err := MakeHardlink(sourcePath, fullPath); err != nil {
-			LOG_ERROR("RESTORE_HARDLINK", "Failed to create hard link %s to %s: %v", fullPath, sourcePath, err)
+			LOG_ERROR("RESTORE_HARDLINK", "Failed to create hard link %s to %s", fullPath, sourcePath)
 			return 0
 		}
-		LOG_TRACE("RESTORE_HARDLINK", "Hard linked %s to %s", linkEntry.Path, hardLinkTable[i].entry.Path)
 	}

-	if options.DeleteMode && len(patterns) == 0 {
+	if deleteMode && len(patterns) == 0 {
 		// Reverse the order to make sure directories are empty before being deleted
 		for i := range extraFiles {
 			file := extraFiles[len(extraFiles)-1-i]
@@ -1103,10 +1034,10 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore

 	for _, entry := range directoryEntries {
 		dir := joinPath(top, entry.Path)
-		entry.RestoreMetadata(dir, nil, metadataOptions)
+		entry.RestoreMetadata(dir, nil, setOwner)
 	}

-	if options.ShowStatistics {
+	if showStatistics {
 		for _, file := range downloadedFiles {
 			LOG_INFO("DOWNLOAD_DONE", "Downloaded %s (%d)", file.Path, file.Size)
 		}
@@ -1117,7 +1048,7 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 	}

 	LOG_INFO("RESTORE_END", "Restored %s to revision %d", top, revision)
-	if options.ShowStatistics {
+	if showStatistics {
 		LOG_INFO("RESTORE_STATS", "Files: %d total, %s bytes", len(fileEntries), PrettySize(totalFileSize))
 		LOG_INFO("RESTORE_STATS", "Downloaded %d file, %s bytes, %d chunks",
 			len(downloadedFiles), PrettySize(downloadedFileSize), chunkDownloader.numberOfDownloadedChunks)
@@ -1136,6 +1067,55 @@ func (manager *BackupManager) Restore(top string, revision int, options *Restore
 	return 0
 }

+// fileEncoder encodes one file at a time to avoid loading the full json description of the entire file tree
+// in the memory
+type fileEncoder struct {
+	top            string
+	readAttributes bool
+	files          []*Entry
+	currentIndex   int
+	buffer         *bytes.Buffer
+}
+
+// Read reads data from the embedded buffer
+func (encoder fileEncoder) Read(data []byte) (n int, err error) {
+	return encoder.buffer.Read(data)
+}
+
+// NextFile switches to the next file and generates its json description in the buffer.  It also takes care of
+// the ending ']' and the commas between files.
+func (encoder *fileEncoder) NextFile() (io.Reader, bool) {
+	if encoder.currentIndex == len(encoder.files) {
+		return nil, false
+	}
+	if encoder.currentIndex == len(encoder.files)-1 {
+		encoder.buffer.Write([]byte("]"))
+		encoder.currentIndex++
+		return encoder, true
+	}
+
+	encoder.currentIndex++
+	entry := encoder.files[encoder.currentIndex]
+	if encoder.readAttributes {
+		entry.ReadAttributes(encoder.top)
+	}
+	description, err := json.Marshal(entry)
+	if err != nil {
+		LOG_FATAL("SNAPSHOT_ENCODE", "Failed to encode file %s: %v", encoder.files[encoder.currentIndex].Path, err)
+		return nil, false
+	}
+
+	if encoder.readAttributes {
+		entry.Attributes = nil
+	}
+
+	if encoder.currentIndex != 0 {
+		encoder.buffer.Write([]byte(","))
+	}
+	encoder.buffer.Write(description)
+	return encoder, true
+}
+
 // UploadSnapshot uploads the specified snapshot to the storage. It turns Files, ChunkHashes, and ChunkLengths into
 // sequences of chunks, and uploads these chunks, and finally the snapshot file.
 func (manager *BackupManager) UploadSnapshot(chunkOperator *ChunkOperator, top string, snapshot *Snapshot,
@@ -1217,7 +1197,8 @@ func (manager *BackupManager) UploadSnapshot(chunkOperator *ChunkOperator, top s
 			entry.StartChunk -= delta
 			entry.EndChunk -= delta

-			if entry.IsHardLinkRoot() {
+			if entry.IsHardlinkRoot() {
+				LOG_DEBUG("SNAPSHOT_UPLOAD", "Hard link root %s %v %v", entry.Path, entry.StartChunk, entry.EndChunk)
 				hardLinkTable = append(hardLinkTable, hardLinkEntry{entry, entry.StartChunk})
 			}

@@ -1225,24 +1206,28 @@ func (manager *BackupManager) UploadSnapshot(chunkOperator *ChunkOperator, top s
 			entry.StartChunk -= lastEndChunk
 			lastEndChunk = entry.EndChunk
 			entry.EndChunk = delta
-		} else if entry.IsHardLinkChild() {
-			i, err := entry.GetHardLinkId()
+		} else if entry.IsHardlinkedFrom() && !entry.IsLink() {
+			i, err := entry.GetHardlinkId()
 			if err != nil {
-				LOG_ERROR("SNAPSHOT_UPLOAD", "Decode error for hard link entry %s: %v", entry.Link, err)
+				LOG_ERROR("SNAPSHOT_UPLOAD", "Decode error for hardlinked entry %s, %v", entry.Link, err)
 				return err
 			}

 			targetEntry := hardLinkTable[i].entry
 			var startChunk, endChunk int

-			if targetEntry.IsFile() && targetEntry.Size > 0 {
+			if targetEntry.Size > 0 {
 				startChunk = hardLinkTable[i].startChunk - lastEndChunk
 				endChunk = targetEntry.EndChunk
-				lastEndChunk = hardLinkTable[i].startChunk + endChunk
 			}
 			entry = entry.HardLinkTo(targetEntry, startChunk, endChunk)

-		} else if entry.IsHardLinkRoot() {
+			if targetEntry.Size > 0 {
+				lastEndChunk = hardLinkTable[i].startChunk + endChunk
+			}
+
+			LOG_DEBUG("SNAPSHOT_UPLOAD", "Uploading cloned hardlink for %s to %s (%v %v)", entry.Path, targetEntry.Path, startChunk, endChunk)
+		} else if entry.IsHardlinkRoot() {
 			hardLinkTable = append(hardLinkTable, hardLinkEntry{entry, 0})
 		}

@@ -1317,8 +1302,7 @@ func (manager *BackupManager) UploadSnapshot(chunkOperator *ChunkOperator, top s
 //	false, nil:   Skipped file;
 //	false, error: Failure to restore file (only if allowFailures == true)
 func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chunkMaker *ChunkMaker, entry *Entry, top string, inPlace bool, overwrite bool,
-	showStatistics bool, totalFileSize int64, downloadedFileSize int64, startTime int64, allowFailures bool,
-	fileFlagsMask uint32) (bool, error) {
+	showStatistics bool, totalFileSize int64, downloadedFileSize int64, startTime int64, allowFailures bool) (bool, error) {

 	LOG_TRACE("DOWNLOAD_START", "Downloading %s", entry.Path)

@@ -1365,10 +1349,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 					LOG_ERROR("DOWNLOAD_CREATE", "Failed to create the file %s for in-place writing: %v", fullPath, err)
 					return false, nil
 				}
-				err = entry.RestoreEarlyFileFlags(existingFile, fileFlagsMask)
-				if err != nil {
-					LOG_WARN("DOWNLOAD_FLAGS", "Failed to set early file flags on %s: %v", fullPath, err)
-				}
+				entry.RestoreEarlyFileFlags(existingFile)

 				n := int64(1)
 				// There is a go bug on Windows (https://github.com/golang/go/issues/21681) that causes Seek to fail
@@ -1552,10 +1533,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 				return false, nil
 			}
 		}
-		err = entry.RestoreEarlyFileFlags(existingFile, fileFlagsMask)
-		if err != nil {
-			LOG_WARN("DOWNLOAD_FLAGS", "Failed to set early file flags on %s: %v", fullPath, err)
-		}
+		entry.RestoreEarlyFileFlags(existingFile)

 		existingFile.Seek(0, 0)

@@ -1638,10 +1616,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 			LOG_ERROR("DOWNLOAD_OPEN", "Failed to open file for writing: %v", err)
 			return false, nil
 		}
-		err = entry.RestoreEarlyFileFlags(newFile, fileFlagsMask)
-		if err != nil {
-			LOG_WARN("DOWNLOAD_FLAGS", "Failed to set early file flags on %s: %v", fullPath, err)
-		}
+		entry.RestoreEarlyFileFlags(newFile)

 		hasher := manager.config.NewFileHasher()

--- a/src/duplicacy_backupmanager_test.go
+++ b/src/duplicacy_backupmanager_test.go
@@ -176,6 +176,8 @@ func assertRestoreFailures(t *testing.T, failedFiles int, expectedFailedFiles in
 }

 func TestBackupManager(t *testing.T) {
+
+	rand.Seed(time.Now().UnixNano())
 	setTestingT(t)
 	SetLoggingLevel(INFO)

@@ -251,23 +253,15 @@ func TestBackupManager(t *testing.T) {
 	time.Sleep(time.Duration(delay) * time.Second)

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	backupManager := CreateBackupManager("host1", storage, testDir, password, nil)
+	backupManager := CreateBackupManager("host1", storage, testDir, password, "", "", false)
 	backupManager.SetupSnapshotCache("default")

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
 	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false, 0, false, 1024, 1024)
 	time.Sleep(time.Duration(delay) * time.Second)
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-	failedFiles := backupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-		Threads:        threads,
-		Patterns:       nil,
-		InPlace:        false,
-		QuickMode:      false,
-		Overwrite:      true,
-		DeleteMode:     false,
-		ShowStatistics: false,
-		AllowFailures:  false,
-	})
+	failedFiles := backupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, false /*quickMode=*/, false, threads /*overwrite=*/, true,
+		/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, false)
 	assertRestoreFailures(t, failedFiles, 0)

 	for _, f := range []string{"file1", "file2", "dir1/file3"} {
@@ -291,16 +285,8 @@ func TestBackupManager(t *testing.T) {
 	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "second", false, false, 0, false, 1024, 1024)
 	time.Sleep(time.Duration(delay) * time.Second)
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-	failedFiles = backupManager.Restore(testDir+"/repository2", 2, &RestoreOptions{
-		Threads:        threads,
-		Patterns:       nil,
-		InPlace:        true,
-		QuickMode:      true,
-		Overwrite:      true,
-		DeleteMode:     false,
-		ShowStatistics: false,
-		AllowFailures:  false,
-	})
+	failedFiles = backupManager.Restore(testDir+"/repository2", 2 /*inPlace=*/, true /*quickMode=*/, true, threads /*overwrite=*/, true,
+		/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, false)
 	assertRestoreFailures(t, failedFiles, 0)

 	for _, f := range []string{"file1", "file2", "dir1/file3"} {
@@ -328,16 +314,8 @@ func TestBackupManager(t *testing.T) {
 	createRandomFile(testDir+"/repository2/dir5/file5", 100)

 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-	failedFiles = backupManager.Restore(testDir+"/repository2", 3, &RestoreOptions{
-		Threads:        threads,
-		Patterns:       nil,
-		InPlace:        true,
-		QuickMode:      false,
-		Overwrite:      true,
-		DeleteMode:     true,
-		ShowStatistics: false,
-		AllowFailures:  false,
-	})
+	failedFiles = backupManager.Restore(testDir+"/repository2", 3 /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+		/*deleteMode=*/ true /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, false)
 	assertRestoreFailures(t, failedFiles, 0)

 	for _, f := range []string{"file1", "file2", "dir1/file3"} {
@@ -364,16 +342,8 @@ func TestBackupManager(t *testing.T) {
 	os.Remove(testDir + "/repository1/file2")
 	os.Remove(testDir + "/repository1/dir1/file3")
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	failedFiles = backupManager.Restore(testDir+"/repository1", 3, &RestoreOptions{
-		Threads:        threads,
-		Patterns:       []string{"+file2", "+dir1/file3", "-*"},
-		InPlace:        true,
-		QuickMode:      false,
-		Overwrite:      true,
-		DeleteMode:     false,
-		ShowStatistics: false,
-		AllowFailures:  false,
-	})
+	failedFiles = backupManager.Restore(testDir+"/repository1", 3 /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+		/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, []string{"+file2", "+dir1/file3", "-*"} /*allowFailures=*/, false)
 	assertRestoreFailures(t, failedFiles, 0)

 	for _, f := range []string{"file1", "file2", "dir1/file3"} {
@@ -389,16 +359,16 @@ func TestBackupManager(t *testing.T) {
 		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
 	}
 			
-	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1, 2, 3} /*tag*/, "" /*showStatistics*/, false,
-		/*showTabular*/ false /*checkFiles*/, false /*checkChunks*/, false /*searchFossils*/, false /*resurrect*/, false /*rewiret*/, false, 1 /*allowFailures*/, false)
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{1, 2, 3}, /*tag*/ "", /*showStatistics*/ false,
+	    /*showTabular*/ false, /*checkFiles*/ false, /*checkChunks*/ false, /*searchFossils*/ false, /*resurrect*/ false, /*rewiret*/ false, 1,  /*allowFailures*/false)
 	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, []int{1} /*tags*/, nil /*retentions*/, nil,
 		/*exhaustive*/ false /*exclusive=*/, false /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false, 1)
 	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
 	if numberOfSnapshots != 2 {
 		t.Errorf("Expected 2 snapshots but got %d", numberOfSnapshots)
 	}
-	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3} /*tag*/, "" /*showStatistics*/, false,
-		/*showTabular*/ false /*checkFiles*/, false /*checkChunks*/, false /*searchFossils*/, false /*resurrect*/, false /*rewiret*/, false, 1 /*allowFailures*/, false)
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{2, 3}, /*tag*/ "", /*showStatistics*/ false,
+	     /*showTabular*/ false, /*checkFiles*/ false, /*checkChunks*/ false, /*searchFossils*/ false, /*resurrect*/ false, /*rewiret*/ false, 1, /*allowFailures*/ false)
 	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "fourth", false, false, 0, false, 1024, 1024)
 	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, nil /*tags*/, nil /*retentions*/, nil,
 		/*exhaustive*/ false /*exclusive=*/, true /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false, 1)
@@ -406,8 +376,8 @@ func TestBackupManager(t *testing.T) {
 	if numberOfSnapshots != 3 {
 		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
 	}
-	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3, 4} /*tag*/, "" /*showStatistics*/, false,
-		/*showTabular*/ false /*checkFiles*/, false /*checkChunks*/, false /*searchFossils*/, false /*resurrect*/, false /*rewiret*/, false, 1 /*allowFailures*/, false)
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{2, 3, 4}, /*tag*/ "", /*showStatistics*/ false,
+	    /*showTabular*/ false, /*checkFiles*/ false, /*checkChunks*/ false, /*searchFossils*/ false, /*resurrect*/ false, /*rewiret*/ false, 1, /*allowFailures*/ false)

 	/*buf := make([]byte, 1<<16)
 	  runtime.Stack(buf, true)
@@ -416,7 +386,7 @@ func TestBackupManager(t *testing.T) {

 // Create file with random file with certain seed
 func createRandomFileSeeded(path string, maxSize int, seed int64) {
-	r := rand.New(rand.NewSource(seed))
+	rand.Seed(seed)
 	file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
 	if err != nil {
 		LOG_ERROR("RANDOM_FILE", "Can't open %s for writing: %v", path, err)
@@ -433,7 +403,7 @@ func createRandomFileSeeded(path string, maxSize int, seed int64) {
 		if bytes > cap(buffer) {
 			bytes = cap(buffer)
 		}
-		r.Read(buffer[:bytes])
+		rand.Read(buffer[:bytes])
 		bytes, err = file.Write(buffer[:bytes])
 		if err != nil {
 			LOG_ERROR("RANDOM_FILE", "Failed to write to %s: %v", path, err)
@@ -444,7 +414,7 @@ func createRandomFileSeeded(path string, maxSize int, seed int64) {
 }

 func corruptFile(path string, start int, length int, seed int64) {
-	r := rand.New(rand.NewSource(seed))
+	rand.Seed(seed)

 	file, err := os.OpenFile(path, os.O_WRONLY, 0644)
 	if err != nil {
@@ -465,7 +435,7 @@ func corruptFile(path string, start int, length int, seed int64) {
 	}

 	buffer := make([]byte, length)
-	r.Read(buffer)
+	rand.Read(buffer)

 	_, err = file.Write(buffer)
 	if err != nil {
@@ -560,30 +530,32 @@ func TestPersistRestore(t *testing.T) {

 	// do unencrypted backup
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	unencBackupManager := CreateBackupManager("host1", unencStorage, testDir, "", nil)
+	unencBackupManager := CreateBackupManager("host1", unencStorage, testDir, "", "", "", false)
 	unencBackupManager.SetupSnapshotCache("default")

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
 	unencBackupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false, 0, false, 1024, 1024)
 	time.Sleep(time.Duration(delay) * time.Second)

+
 	// do encrypted backup
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	encBackupManager := CreateBackupManager("host1", storage, testDir, password, nil)
+	encBackupManager := CreateBackupManager("host1", storage, testDir, password, "", "", false)
 	encBackupManager.SetupSnapshotCache("default")

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
 	encBackupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false, 0, false, 1024, 1024)
 	time.Sleep(time.Duration(delay) * time.Second)

-	// check snapshots
-	unencBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1} /*tag*/, "",
-		/*showStatistics*/ true /*showTabular*/, false /*checkFiles*/, true /*checkChunks*/, false,
-		/*searchFossils*/ false /*resurrect*/, false /*rewiret*/, false, 1 /*allowFailures*/, false)

-	encBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1} /*tag*/, "",
-		/*showStatistics*/ true /*showTabular*/, false /*checkFiles*/, true /*checkChunks*/, false,
-		/*searchFossils*/ false /*resurrect*/, false /*rewiret*/, false, 1 /*allowFailures*/, false)
+	// check snapshots
+	unencBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{1}, /*tag*/ "",
+		/*showStatistics*/ true, /*showTabular*/ false, /*checkFiles*/ true, /*checkChunks*/ false,
+		/*searchFossils*/ false, /*resurrect*/ false, /*rewiret*/ false, 1, /*allowFailures*/ false)
+
+	encBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{1}, /*tag*/ "",
+		/*showStatistics*/ true, /*showTabular*/ false, /*checkFiles*/ true, /*checkChunks*/ false,
+		 /*searchFossils*/ false, /*resurrect*/ false, /*rewiret*/ false, 1, /*allowFailures*/ false)
 		
 	// check functions
 	checkAllUncorrupted := func(cmpRepository string) {
@@ -632,7 +604,7 @@ func TestPersistRestore(t *testing.T) {
 	
 			hash1 := getFileHash(testDir + "/repository1/" + f)
 			hash2 := getFileHash(testDir + cmpRepository + "/" + f)
-			if f == expectCorrupted {
+			if (f==expectCorrupted) {
 				if hash1 == hash2 {
 					t.Errorf("File %s has same hashes, expected to be corrupted: %s vs %s", f, hash1, hash2)
 				}
@@ -647,16 +619,8 @@ func TestPersistRestore(t *testing.T) {

 	// test restore all uncorrupted to repository3
 	SetDuplicacyPreferencePath(testDir + "/repository3/.duplicacy")
-	failedFiles := unencBackupManager.Restore(testDir+"/repository3", 1, &RestoreOptions{
-		Threads:        threads,
-		Patterns:       nil,
-		InPlace:        true,
-		QuickMode:      false,
-		Overwrite:      false,
-		DeleteMode:     false,
-		ShowStatistics: false,
-		AllowFailures:  false,
-	})
+	failedFiles := unencBackupManager.Restore(testDir+"/repository3", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, false,
+		 /*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, false)
 	assertRestoreFailures(t, failedFiles, 0)
 	checkAllUncorrupted("/repository3")

@@ -690,44 +654,30 @@ func TestPersistRestore(t *testing.T) {

 		// check snapshots with --persist (allowFailures == true)
 		// this would cause a panic and os.Exit from duplicacy_log if allowFailures == false
-		unencBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1} /*tag*/, "",
-			/*showStatistics*/ true /*showTabular*/, false /*checkFiles*/, true /*checkChunks*/, false,
-			/*searchFossils*/ false /*resurrect*/, false /*rewrite*/, false, 1 /*allowFailures*/, true)
+		unencBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{1}, /*tag*/ "",
+			/*showStatistics*/ true, /*showTabular*/ false, /*checkFiles*/ true, /*checkChunks*/ false,
+			/*searchFossils*/ false, /*resurrect*/ false, /*rewrite*/ false, 1, /*allowFailures*/ true)
+
+		encBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1", /*revisions*/ []int{1}, /*tag*/ "",
+			/*showStatistics*/ true, /*showTabular*/ false, /*checkFiles*/ true, /*checkChunks*/ false,
+			/*searchFossils*/ false, /*resurrect*/ false, /*rewrite*/ false, 1, /*allowFailures*/ true)

-		encBackupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1} /*tag*/, "",
-			/*showStatistics*/ true /*showTabular*/, false /*checkFiles*/, true /*checkChunks*/, false,
-			/*searchFossils*/ false /*resurrect*/, false /*rewrite*/, false, 1 /*allowFailures*/, true)
 	
 		// test restore corrupted, inPlace = true, corrupted files will have hash failures
 		os.RemoveAll(testDir+"/repository2")
 		SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-		failedFiles = unencBackupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        true,
-			QuickMode:      false,
-			Overwrite:      false,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = unencBackupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, false,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 1)

 		// check restore, expect file1 to be corrupted
 		checkCorruptedFile("/repository2", "file1")

+		
 		os.RemoveAll(testDir+"/repository2")
 		SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-		failedFiles = encBackupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        true,
-			QuickMode:      false,
-			Overwrite:      false,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = encBackupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, false,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 1)

 		// check restore, expect file3 to be corrupted
@@ -737,33 +687,18 @@ func TestPersistRestore(t *testing.T) {
 		// test restore corrupted, inPlace = false, corrupted files will be missing
 		os.RemoveAll(testDir+"/repository2")
 		SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-		failedFiles = unencBackupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        false,
-			QuickMode:      false,
-			Overwrite:      false,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = unencBackupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, false /*quickMode=*/, false, threads /*overwrite=*/, false,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 1)

 		// check restore, expect file1 to be corrupted
 		checkMissingFile("/repository2", "file1")

+		
 		os.RemoveAll(testDir+"/repository2")
 		SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-		failedFiles = encBackupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        false,
-			QuickMode:      false,
-			Overwrite:      false,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = encBackupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, false /*quickMode=*/, false, threads /*overwrite=*/, false,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 1)

 		// check restore, expect file3 to be corrupted
@@ -773,57 +708,25 @@ func TestPersistRestore(t *testing.T) {
 		// with overwrite=true, corrupted file1 from unenc will be restored correctly from enc
 		// the latter will not touch the existing file3 with correct hash
 		os.RemoveAll(testDir+"/repository2")
-		failedFiles = unencBackupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        true,
-			QuickMode:      false,
-			Overwrite:      false,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = unencBackupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, false,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 1)

-		failedFiles = encBackupManager.Restore(testDir+"/repository2", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        true,
-			QuickMode:      false,
-			Overwrite:      true,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = encBackupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 0)
 		checkAllUncorrupted("/repository2")

 		// restore to repository3, with overwrite and allowFailures (true/false), quickMode = false (use hashes)
 		// should always succeed as uncorrupted files already exist with correct hash, so these will be ignored
 		SetDuplicacyPreferencePath(testDir + "/repository3/.duplicacy")
-		failedFiles = unencBackupManager.Restore(testDir+"/repository3", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        true,
-			QuickMode:      false,
-			Overwrite:      true,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  false,
-		})
+		failedFiles = unencBackupManager.Restore(testDir+"/repository3", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, false)
 		assertRestoreFailures(t, failedFiles, 0)
 		checkAllUncorrupted("/repository3")

-		failedFiles = unencBackupManager.Restore(testDir+"/repository3", 1, &RestoreOptions{
-			Threads:        threads,
-			Patterns:       nil,
-			InPlace:        true,
-			QuickMode:      false,
-			Overwrite:      true,
-			DeleteMode:     false,
-			ShowStatistics: false,
-			AllowFailures:  true,
-		})
+		failedFiles = unencBackupManager.Restore(testDir+"/repository3", threads /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+			/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil /*allowFailures=*/, true)
 		assertRestoreFailures(t, failedFiles, 0)
 		checkAllUncorrupted("/repository3")
 	}
--- a/src/duplicacy_config.go
+++ b/src/duplicacy_config.go
@@ -85,8 +85,8 @@ type Config struct {
 	FileKey []byte `json:"-"`

 	// for erasure coding
-	DataShards int `json:"data-shards"`
-	ParityShards int `json:"parity-shards"`
+	DataShards int `json:'data-shards'`
+	ParityShards int `json:'parity-shards'`

 	// for RSA encryption
 	rsaPrivateKey *rsa.PrivateKey
--- a/src/duplicacy_entry.go
+++ b/src/duplicacy_entry.go
@@ -18,14 +18,15 @@ import (
 	"sort"
 	"strconv"
 	"strings"
+	"syscall"
 	"time"

 	"github.com/vmihailenco/msgpack"
 )

 const (
-	entryHardLinkRootChunkMarker   = -9
-	entryHardLinkTargetChunkMarker = -10
+	entrySymHardLinkRootChunkMarker   = -72
+	entrySymHardLinkTargetChunkMarker = -73
 )

 // This is the hidden directory in the repository for storing various files.
@@ -98,7 +99,7 @@ func CreateEntryFromFileInfo(fileInfo os.FileInfo, directory string) *Entry {
 		Mode: uint32(mode),
 	}

-	GetOwner(entry, fileInfo)
+	GetOwner(entry, &fileInfo)

 	return entry
 }
@@ -125,21 +126,12 @@ func (entry *Entry) Copy() *Entry {
 }

 func (entry *Entry) HardLinkTo(target *Entry, startChunk int, endChunk int) *Entry {
-	endOffset := target.EndOffset
-	link := entry.Link
-
-	if !target.IsFile() {
-		startChunk = target.StartChunk
-		endChunk = entry.EndChunk
-		endOffset = entry.EndOffset
-		link = target.Link
-	}
 	return &Entry{
 		Path: entry.Path,
 		Size: target.Size,
 		Time: target.Time,
 		Mode: target.Mode,
-		Link: link,
+		Link: entry.Link,
 		Hash: target.Hash,

 		UID: target.UID,
@@ -148,7 +140,7 @@ func (entry *Entry) HardLinkTo(target *Entry, startChunk int, endChunk int) *Ent
 		StartChunk:  startChunk,
 		StartOffset: target.StartOffset,
 		EndChunk:    endChunk,
-		EndOffset:   endOffset,
+		EndOffset:   target.EndOffset,

 		Attributes: target.Attributes,
 	}
@@ -524,30 +516,34 @@ func (entry *Entry) IsLink() bool {
 }

 func (entry *Entry) IsSpecial() bool {
-	return entry.Mode&uint32(os.ModeNamedPipe|os.ModeDevice|os.ModeCharDevice|os.ModeSocket) != 0
+	return entry.Mode&uint32(os.ModeNamedPipe|os.ModeDevice|os.ModeCharDevice) != 0
+}
+
+func (entry *Entry) IsFileOrSpecial() bool {
+	return entry.Mode&uint32(os.ModeDir|os.ModeSymlink|os.ModeIrregular) == 0
 }

 func (entry *Entry) IsComplete() bool {
 	return entry.Size >= 0
 }

-func (entry *Entry) IsHardLinkChild() bool {
-	return (entry.IsFile() && len(entry.Link) > 0 && entry.Link != "/") || (!entry.IsDir() && entry.EndChunk == entryHardLinkTargetChunkMarker)
+func (entry *Entry) IsHardlinkedFrom() bool {
+	return (entry.IsFileOrSpecial() && len(entry.Link) > 0 && entry.Link != "/") || (entry.IsLink() && entry.StartChunk == entrySymHardLinkTargetChunkMarker)
 }

-func (entry *Entry) IsHardLinkRoot() bool {
-	return (entry.IsFile() && entry.Link == "/") || (!entry.IsDir() && entry.EndChunk == entryHardLinkRootChunkMarker)
+func (entry *Entry) IsHardlinkRoot() bool {
+	return (entry.IsFileOrSpecial() && entry.Link == "/") || (entry.IsLink() && entry.StartChunk == entrySymHardLinkRootChunkMarker)
 }

-func (entry *Entry) GetHardLinkId() (int, error) {
-	if entry.IsFile() {
+func (entry *Entry) GetHardlinkId() (int, error) {
+	if entry.IsLink() {
+		if entry.StartChunk != entrySymHardLinkTargetChunkMarker {
+			return 0, errors.New("Symlink entry not marked as hardlinked")
+		}
+		return entry.StartOffset, nil
+	} else {
 		i, err := strconv.ParseUint(entry.Link, 16, 64)
 		return int(i), err
-	} else {
-		if entry.EndChunk != entryHardLinkTargetChunkMarker {
-			return 0, errors.New("Entry not marked as hard link child")
-		}
-		return entry.EndOffset, nil
 	}
 }

@@ -582,66 +578,47 @@ func (entry *Entry) String(maxSizeDigits int) string {
 	return fmt.Sprintf("%*d %s %64s %s", maxSizeDigits, entry.Size, modifiedTime, entry.Hash, entry.Path)
 }

-type RestoreMetadataOptions struct {
-	SetOwner         bool
-	ExcludeXattrs    bool
-	NormalizeXattrs  bool
-	IncludeFileFlags bool
-	FileFlagsMask    uint32
-}
-
-func (entry *Entry) RestoreMetadata(fullPath string, fileInfo os.FileInfo,
-	options RestoreMetadataOptions) bool {
+func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo, setOwner bool) bool {

 	if fileInfo == nil {
-		var err error
-		fileInfo, err = os.Lstat(fullPath)
+		stat, err := os.Lstat(fullPath)
+		fileInfo = &stat
 		if err != nil {
-			LOG_ERROR("RESTORE_STAT", "Failed to retrieve the file info on %s: %v", entry.Path, err)
+			LOG_ERROR("RESTORE_STAT", "Failed to retrieve the file info: %v", err)
 			return false
 		}
 	}

-	if !options.ExcludeXattrs {
-		err := entry.SetAttributesToFile(fullPath, options.NormalizeXattrs)
-		if err != nil {
-			LOG_WARN("RESTORE_ATTR", "Failed to set extended attributes on %s: %v", entry.Path, err)
-		}
-	}
-
 	// Note that chown can remove setuid/setgid bits so should be called before chmod
-	if options.SetOwner {
+	if setOwner {
 		if !SetOwner(fullPath, entry, fileInfo) {
 			return false
 		}
 	}

 	// Only set the permission if the file is not a symlink
-	if !entry.IsLink() && fileInfo.Mode()&fileModeMask != entry.GetPermissions() {
+	if !entry.IsLink() && (*fileInfo).Mode()&fileModeMask != entry.GetPermissions() {
 		err := os.Chmod(fullPath, entry.GetPermissions())
 		if err != nil {
-			LOG_ERROR("RESTORE_CHMOD", "Failed to set the file permissions on %s: %v", entry.Path, err)
+			LOG_ERROR("RESTORE_CHMOD", "Failed to set the file permissions: %v", err)
 			return false
 		}
 	}

+	if entry.Attributes != nil && len(*entry.Attributes) > 0 {
+		entry.SetAttributesToFile(fullPath)
+	}
+
 	// Only set the time if the file is not a symlink
-	if !entry.IsLink() && fileInfo.ModTime().Unix() != entry.Time {
+	if !entry.IsLink() && (*fileInfo).ModTime().Unix() != entry.Time {
 		modifiedTime := time.Unix(entry.Time, 0)
 		err := os.Chtimes(fullPath, modifiedTime, modifiedTime)
 		if err != nil {
-			LOG_ERROR("RESTORE_CHTIME", "Failed to set the modification time on %s: %v", entry.Path, err)
+			LOG_ERROR("RESTORE_CHTIME", "Failed to set the modification time: %v", err)
 			return false
 		}
 	}

-	if options.IncludeFileFlags {
-		err := entry.RestoreLateFileFlags(fullPath, fileInfo, options.FileFlagsMask)
-		if err != nil {
-			LOG_WARN("RESTORE_FLAGS", "Failed to set file flags on %s: %v", entry.Path, err)
-		}
-	}
-
 	return true
 }

@@ -772,39 +749,27 @@ func (files FileInfoCompare) Less(i, j int) bool {
 	}
 }

-type EntryListerOptions struct {
-	Patterns           []string
-	NoBackupFile       string
-	ExcludeByAttribute bool
-	ExcludeXattrs      bool
-	NormalizeXattr     bool
-	IncludeFileFlags   bool
-	IncludeSpecials    bool
+type listEntryLinkKey struct {
+	dev uint64
+	ino uint64
 }

-type EntryLister interface {
-	ListDir(top string, path string, listingChannel chan *Entry, options *EntryListerOptions) (directoryList []*Entry, skippedFiles []string, err error)
-}
-
-type LocalDirectoryLister struct {
+type ListingState struct {
 	linkIndex int
 	linkTable map[listEntryLinkKey]int // map unique inode details to initially found path
 }

-func NewLocalDirectoryLister() *LocalDirectoryLister {
-	return &LocalDirectoryLister{
+func NewListingState() *ListingState {
+	return &ListingState{
 		linkTable: make(map[listEntryLinkKey]int),
 	}
 }

-// ListDir returns a list of entries representing file and subdirectories under the directory 'path'.
-// Entry paths are normalized as relative to 'top'.
-func (dl *LocalDirectoryLister) ListDir(top string, path string, listingChannel chan *Entry,
-	options *EntryListerOptions) (directoryList []*Entry, skippedFiles []string, err error) {
-
-	if options == nil {
-		options = &EntryListerOptions{}
-	}
+// ListEntries returns a list of entries representing file and subdirectories under the directory 'path'.  Entry paths
+// are normalized as relative to 'top'.  'patterns' are used to exclude or include certain files.
+func ListEntries(top string, path string, patterns []string, nobackupFile string, excludeByAttribute bool,
+	listingState *ListingState,
+	listingChannel chan *Entry) (directoryList []*Entry, skippedFiles []string, err error) {

 	LOG_DEBUG("LIST_ENTRIES", "Listing %s", path)

@@ -817,12 +782,10 @@ func (dl *LocalDirectoryLister) ListDir(top string, path string, listingChannel
 		return directoryList, nil, err
 	}

-	patterns := options.Patterns
-
 	// This binary search works because ioutil.ReadDir returns files sorted by Name() by default
-	if options.NoBackupFile != "" {
-		ii := sort.Search(len(files), func(ii int) bool { return strings.Compare(files[ii].Name(), options.NoBackupFile) >= 0 })
-		if ii < len(files) && files[ii].Name() == options.NoBackupFile {
+	if nobackupFile != "" {
+		ii := sort.Search(len(files), func(ii int) bool { return strings.Compare(files[ii].Name(), nobackupFile) >= 0 })
+		if ii < len(files) && files[ii].Name() == nobackupFile {
 			LOG_DEBUG("LIST_NOBACKUP", "%s is excluded due to nobackup file", path)
 			return directoryList, skippedFiles, nil
 		}
@@ -844,44 +807,47 @@ func (dl *LocalDirectoryLister) ListDir(top string, path string, listingChannel
 		if f.Name() == DUPLICACY_DIRECTORY {
 			continue
 		}
+		if f.Mode()&os.ModeSocket != 0 {
+			continue
+		}

 		entry := CreateEntryFromFileInfo(f, normalizedPath)
 		if len(patterns) > 0 && !MatchPath(entry.Path, patterns) {
 			continue
 		}

-		linkKey, isHardLinked := entry.getHardLinkKey(f)
-		if isHardLinked {
-			if linkIndex, seen := dl.linkTable[linkKey]; seen {
+		var linkKey *listEntryLinkKey
+
+		if runtime.GOOS != "windows" && !entry.IsDir() {
+			if stat := f.Sys().(*syscall.Stat_t); stat != nil && stat.Nlink > 1 {
+				k := listEntryLinkKey{dev: uint64(stat.Dev), ino: uint64(stat.Ino)}
+				if linkIndex, seen := listingState.linkTable[k]; seen {
 					if linkIndex == -1 {
-					LOG_DEBUG("LIST_EXCLUDE", "%s was excluded or skipped (hard link)", entry.Path)
+						LOG_DEBUG("LIST_EXCLUDE", "%s is excluded by attribute (hardlink)", entry.Path)
 						continue
 					}
-
 					entry.Size = 0
-				if entry.IsFile() {
+					if entry.IsLink() {
+						entry.StartChunk = entrySymHardLinkTargetChunkMarker
+						entry.StartOffset = linkIndex
+					} else {
 						entry.Link = strconv.FormatInt(int64(linkIndex), 16)
-				} else {
-					entry.EndChunk = entryHardLinkTargetChunkMarker
-					entry.EndOffset = linkIndex
 					}
-				listingChannel <- entry
-				continue
 				} else {
-				if entry.IsFile() {
+					if entry.IsLink() {
+						entry.StartChunk = entrySymHardLinkRootChunkMarker
+					} else {
 						entry.Link = "/"
-				} else {
-					entry.EndChunk = entryHardLinkRootChunkMarker
 					}
-				dl.linkTable[linkKey] = -1
+					listingState.linkTable[k] = -1
+					linkKey = &k
+				}
 			}
 		}
-
-		fullPath := joinPath(top, entry.Path)

 		if entry.IsLink() {
 			isRegular := false
-			isRegular, entry.Link, err = Readlink(fullPath)
+			isRegular, entry.Link, err = Readlink(joinPath(top, entry.Path))
 			if err != nil {
 				LOG_WARN("LIST_LINK", "Failed to read the symlink %s: %v", entry.Path, err)
 				skippedFiles = append(skippedFiles, entry.Path)
@@ -891,7 +857,7 @@ func (dl *LocalDirectoryLister) ListDir(top string, path string, listingChannel
 			if isRegular {
 				entry.Mode ^= uint32(os.ModeSymlink)
 			} else if path == "" && (filepath.IsAbs(entry.Link) || filepath.HasPrefix(entry.Link, `\\`)) && !strings.HasPrefix(entry.Link, normalizedTop) {
-				stat, err := os.Stat(fullPath)
+				stat, err := os.Stat(joinPath(top, entry.Path))
 				if err != nil {
 					LOG_WARN("LIST_LINK", "Failed to read the symlink: %v", err)
 					skippedFiles = append(skippedFiles, entry.Path)
@@ -909,35 +875,24 @@ func (dl *LocalDirectoryLister) ListDir(top string, path string, listingChannel
 				}
 				entry = newEntry
 			}
-		} else if options.IncludeSpecials && entry.IsSpecial() {
-			if err := entry.ReadSpecial(fullPath, f); err != nil {
-				LOG_WARN("LIST_DEV", "Failed to save device node %s: %v", entry.Path, err)
+		} else if entry.IsSpecial() {
+			if !entry.ReadSpecial(f) {
+				LOG_WARN("LIST_DEV", "Failed to save device node %s", entry.Path)
 				skippedFiles = append(skippedFiles, entry.Path)
 				continue
 			}
 		}

-		if !options.ExcludeXattrs {
-			if err := entry.ReadAttributes(f, fullPath, false); err != nil {
-				LOG_WARN("LIST_ATTR", "Failed to read xattrs on %s: %v", entry.Path, err)
-			}
-		}
+		entry.ReadAttributes(top)

-		// if the flags are already in the FileInfo we can keep them
-		if !entry.GetFileFlags(f) && options.IncludeFileFlags {
-			if err := entry.ReadFileFlags(f, fullPath); err != nil {
-				LOG_WARN("LIST_ATTR", "Failed to read file flags on %s: %v", entry.Path, err)
-			}
-		}
-
-		if options.ExcludeByAttribute && entry.Attributes != nil && excludedByAttribute(*entry.Attributes) {
+		if excludeByAttribute && entry.Attributes != nil && excludedByAttribute(*entry.Attributes) {
 			LOG_DEBUG("LIST_EXCLUDE", "%s is excluded by attribute", entry.Path)
 			continue
 		}

-		if isHardLinked {
-			dl.linkTable[linkKey] = dl.linkIndex
-			dl.linkIndex++
+		if linkKey != nil {
+			listingState.linkTable[*linkKey] = listingState.linkIndex
+			listingState.linkIndex++
 		}

 		if entry.IsDir() {
--- a/src/duplicacy_entry_test.go
+++ b/src/duplicacy_entry_test.go
@@ -7,6 +7,7 @@ package duplicacy
 import (
 	"bytes"
 	"encoding/json"
+	"io/ioutil"
 	"math/rand"
 	"os"
 	"path/filepath"
@@ -165,14 +166,12 @@ func TestEntryOrder(t *testing.T) {
 			continue
 		}

-		err := os.WriteFile(fullPath, []byte(file), 0700)
+		err := ioutil.WriteFile(fullPath, []byte(file), 0700)
 		if err != nil {
 			t.Errorf("WriteFile(%s) returned an error: %s", fullPath, err)
 		}
 	}

-	lister := NewLocalDirectoryLister()
-
 	directories := make([]*Entry, 0, 4)
 	directories = append(directories, CreateEntry("", 0, 0, 0))

@@ -183,7 +182,7 @@ func TestEntryOrder(t *testing.T) {
 	for len(directories) > 0 {
 		directory := directories[len(directories)-1]
 		directories = directories[:len(directories)-1]
-		subdirectories, _, err := lister.ListDir(testDir, directory.Path, entryChannel, nil)
+		subdirectories, _, err := ListEntries(testDir, directory.Path, nil, "", false, entryChannel)
 		if err != nil {
 			t.Errorf("ListEntries(%s, %s) returned an error: %s", testDir, directory.Path, err)
 		}
@@ -241,24 +240,16 @@ func TestEntryExcludeByAttribute(t *testing.T) {
 	if runtime.GOOS == "darwin" {
 		excludeAttrName = "com.apple.metadata:com_apple_backup_excludeItem"
 		excludeAttrValue = []byte("com.apple.backupd")
-	} else if runtime.GOOS == "linux" {
+	} else if runtime.GOOS == "linux" || runtime.GOOS == "freebsd" || runtime.GOOS == "netbsd" || runtime.GOOS == "solaris" {
 		excludeAttrName = "user.duplicacy_exclude"
-	} else if runtime.GOOS == "freebsd" || runtime.GOOS == "netbsd" {
-		excludeAttrName = "duplicacy_exclude"
 	} else {
-		t.Skip("skipping test, not darwin, linux, freebsd, or netbsd")
+		t.Skip("skipping test, not darwin, linux, freebsd, netbsd, or solaris")
 	}

-	tmpDir := ""
-	// on linux TempDir is usually a tmpfs which does not support xattrs
-	if runtime.GOOS == "linux" {
-		tmpDir = "."
-	}
-	testDir, err := os.MkdirTemp(tmpDir, "duplicacy_test")
-	if err != nil {
-		t.Errorf("Mkdirtmp() failed: %v", err)
-		return
-	}
+	testDir := filepath.Join(os.TempDir(), "duplicacy_test")
+
+	os.RemoveAll(testDir)
+	os.MkdirAll(testDir, 0700)

 	// Files or folders named with "exclude" below will have the exclusion attribute set on them
 	// When ListEntries is called with excludeByAttribute true, they should be excluded.
@@ -282,7 +273,7 @@ func TestEntryExcludeByAttribute(t *testing.T) {
 			continue
 		}

-		err := os.WriteFile(fullPath, []byte(file), 0700)
+		err := ioutil.WriteFile(fullPath, []byte(file), 0700)
 		if err != nil {
 			t.Errorf("WriteFile(%s) returned an error: %s", fullPath, err)
 		}
@@ -297,8 +288,6 @@ func TestEntryExcludeByAttribute(t *testing.T) {

 	for _, excludeByAttribute := range [2]bool{true, false} {
 		t.Logf("testing excludeByAttribute: %t", excludeByAttribute)
-
-		lister := NewLocalDirectoryLister()
 		directories := make([]*Entry, 0, 4)
 		directories = append(directories, CreateEntry("", 0, 0, 0))

@@ -309,11 +298,7 @@ func TestEntryExcludeByAttribute(t *testing.T) {
 		for len(directories) > 0 {
 			directory := directories[len(directories)-1]
 			directories = directories[:len(directories)-1]
-			subdirectories, _, err := lister.ListDir(testDir, directory.Path, entryChannel,
-				&EntryListerOptions{
-					ExcludeByAttribute: excludeByAttribute,
-				})
-
+			subdirectories, _, err := ListEntries(testDir, directory.Path, nil, "", excludeByAttribute, entryChannel)
 			if err != nil {
 				t.Errorf("ListEntries(%s, %s) returned an error: %s", testDir, directory.Path, err)
 			}
@@ -362,9 +347,10 @@ func TestEntryExcludeByAttribute(t *testing.T) {

 	}

-	if tmpDir != "" || !t.Failed() {
+	if !t.Failed() {
 		os.RemoveAll(testDir)
 	}
+
 }

 func TestEntryEncoding(t *testing.T) {
--- a/src/duplicacy_keyring_windows.go
+++ b/src/duplicacy_keyring_windows.go
@@ -6,7 +6,7 @@ package duplicacy

 import (
 	"encoding/json"
-	"os"
+	"io/ioutil"
 	"syscall"
 	"unsafe"
 )
@@ -86,7 +86,7 @@ func keyringGet(key string) (value string) {
 		return ""
 	}

-	description, err := os.ReadFile(keyringFile)
+	description, err := ioutil.ReadFile(keyringFile)
 	if err != nil {
 		LOG_DEBUG("KEYRING_READ", "Keyring file not read: %v", err)
 		return ""
@@ -125,7 +125,7 @@ func keyringSet(key string, value string) bool {

 	keyring := make(map[string][]byte)

-	description, err := os.ReadFile(keyringFile)
+	description, err := ioutil.ReadFile(keyringFile)
 	if err == nil {
 		err = json.Unmarshal(description, &keyring)
 		if err != nil {
@@ -160,7 +160,7 @@ func keyringSet(key string, value string) bool {
 		return false
 	}

-	err = os.WriteFile(keyringFile, description, 0600)
+	err = ioutil.WriteFile(keyringFile, description, 0600)
 	if err != nil {
 		LOG_DEBUG("KEYRING_WRITE", "Failed to save the keyring storage to file %s: %v", keyringFile, err)
 		return false
--- a/src/duplicacy_preference.go
+++ b/src/duplicacy_preference.go
@@ -6,37 +6,13 @@ package duplicacy

 import (
 	"encoding/json"
-	"fmt"
+	"io/ioutil"
 	"os"
 	"path"
 	"reflect"
-	"strconv"
 	"strings"
 )

-type flagsMask uint32
-
-func (f flagsMask) MarshalJSON() ([]byte, error) {
-	return json.Marshal(fmt.Sprintf("0x%.8x", f))
-}
-
-func (f *flagsMask) UnmarshalJSON(data []byte) error {
-	var str string
-	if err := json.Unmarshal(data, &str); err != nil {
-		return err
-	}
-	if str[0] == '0' && (str[1] == 'x' || str[1] == 'X') {
-		str = str[2:]
-	}
-
-	v, err := strconv.ParseUint(string(str), 16, 32)
-	if err != nil {
-		return err
-	}
-	*f = flagsMask(v)
-	return nil
-}
-
 // Preference stores options for each storage.
 type Preference struct {
 	Name              string            `json:"name"`
@@ -50,13 +26,7 @@ type Preference struct {
 	NobackupFile      string            `json:"nobackup_file"`
 	Keys              map[string]string `json:"keys"`
 	FiltersFile       string            `json:"filters"`
-	ExcludeOwner       bool              `json:"exclude_owner"`
 	ExcludeByAttribute bool             `json:"exclude_by_attribute"`
-	ExcludeXattrs      bool              `json:"exclude_xattrs"`
-	NormalizeXattrs    bool              `json:"normalize_xattrs"`
-	IncludeFileFlags   bool              `json:"include_file_flags"`
-	IncludeSpecials    bool              `json:"include_specials"`
-	FileFlagsMask      flagsMask         `json:"file_flags_mask"`
 }

 var preferencePath string
@@ -73,7 +43,7 @@ func LoadPreferences(repository string) bool {
 	}

 	if !stat.IsDir() {
-		content, err := os.ReadFile(preferencePath)
+		content, err := ioutil.ReadFile(preferencePath)
 		if err != nil {
 			LOG_ERROR("DOT_DUPLICACY_PATH", "Failed to locate the preference path: %v", err)
 			return false
@@ -91,7 +61,7 @@ func LoadPreferences(repository string) bool {
 		preferencePath = realPreferencePath
 	}

-	description, err := os.ReadFile(path.Join(preferencePath, "preferences"))
+	description, err := ioutil.ReadFile(path.Join(preferencePath, "preferences"))
 	if err != nil {
 		LOG_ERROR("PREFERENCE_OPEN", "Failed to read the preference file from repository %s: %v", repository, err)
 		return false
@@ -140,7 +110,7 @@ func SavePreferences() bool {
 	}
 	preferenceFile := path.Join(GetDuplicacyPreferencePath(), "preferences")

-	err = os.WriteFile(preferenceFile, description, 0600)
+	err = ioutil.WriteFile(preferenceFile, description, 0600)
 	if err != nil {
 		LOG_ERROR("PREFERENCE_WRITE", "Failed to save the preference file %s: %v", preferenceFile, err)
 		return false
--- a/src/duplicacy_shadowcopy_darwin.go
+++ b/src/duplicacy_shadowcopy_darwin.go
@@ -10,6 +10,7 @@ package duplicacy
 import (
 	"context"
 	"errors"
+	"io/ioutil"
 	"os"
 	"os/exec"
 	"regexp"
@@ -135,7 +136,7 @@ func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadow
 	}

 	// Create mount point
-	snapshotPath, err = os.MkdirTemp("/tmp/", "snp_")
+	snapshotPath, err = ioutil.TempDir("/tmp/", "snp_")
 	if err != nil {
 		LOG_ERROR("VSS_CREATE", "Failed to create temporary mount directory")
 		return top
--- a/src/duplicacy_snapshot.go
+++ b/src/duplicacy_snapshot.go
@@ -9,13 +9,15 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
+	"io/ioutil"
 	"os"
 	"path/filepath"
-	"sort"
 	"strings"
 	"time"
+	"sort"

    "github.com/vmihailenco/msgpack"
+
 )

 // Snapshot represents a backup of the repository.
@@ -61,38 +63,17 @@ type DirectoryListing struct {
 	files *[]Entry
 }

-type ListFilesOptions struct {
-	NoBackupFile       string
-	FiltersFile        string
-	ExcludeByAttribute bool
-	ExcludeXattrs      bool
-	NormalizeXattr     bool
-	IncludeFileFlags   bool
-	IncludeSpecials    bool
-}
+func (snapshot *Snapshot) ListLocalFiles(top string, nobackupFile string,
+								         filtersFile string, excludeByAttribute bool, listingChannel chan *Entry,
+								         skippedDirectories *[]string, skippedFiles *[]string) {

-func NewListFilesOptions(p *Preference) *ListFilesOptions {
-	return &ListFilesOptions{
-		NoBackupFile:       p.NobackupFile,
-		FiltersFile:        p.FiltersFile,
-		ExcludeByAttribute: p.ExcludeByAttribute,
-		ExcludeXattrs:      p.ExcludeXattrs,
-		NormalizeXattr:     p.NormalizeXattrs,
-		IncludeFileFlags:   p.IncludeFileFlags,
-		IncludeSpecials:    p.IncludeSpecials,
-	}
-}
+	var patterns []string
+	listingState := NewListingState()

-func (snapshot *Snapshot) ListLocalFiles(top string,
-	listingChannel chan *Entry, skippedDirectories *[]string, skippedFiles *[]string,
-	options *ListFilesOptions) {
-
-	if options.FiltersFile == "" {
-		options.FiltersFile = joinPath(GetDuplicacyPreferencePath(), "filters")
+	if filtersFile == "" {
+		filtersFile = joinPath(GetDuplicacyPreferencePath(), "filters")
 	}
-
-	patterns := ProcessFilters(options.FiltersFile)
-	lister := NewLocalDirectoryLister()
+	patterns = ProcessFilters(filtersFile)

 	directories := make([]*Entry, 0, 256)
 	directories = append(directories, CreateEntry("", 0, 0, 0))
@@ -101,16 +82,7 @@ func (snapshot *Snapshot) ListLocalFiles(top string,

 		directory := directories[len(directories)-1]
 		directories = directories[:len(directories)-1]
-		subdirectories, skipped, err := lister.ListDir(top, directory.Path, listingChannel,
-			&EntryListerOptions{
-				Patterns:           patterns,
-				NoBackupFile:       options.NoBackupFile,
-				ExcludeByAttribute: options.ExcludeByAttribute,
-				ExcludeXattrs:      options.ExcludeXattrs,
-				NormalizeXattr:     options.NormalizeXattr,
-				IncludeFileFlags:   options.IncludeFileFlags,
-				IncludeSpecials:    options.IncludeSpecials,
-			})
+		subdirectories, skipped, err := ListEntries(top, directory.Path, patterns, nobackupFile, excludeByAttribute, listingState, listingChannel)
 		if err != nil {
 			if directory.Path == "" {
 				LOG_ERROR("LIST_FAILURE", "Failed to list the repository root: %v", err)
@@ -272,7 +244,7 @@ func ProcessFilterFile(patternFile string, includedFiles []string) (patterns []s
 	}
 	includedFiles = append(includedFiles, patternFile)
 	LOG_INFO("SNAPSHOT_FILTER", "Parsing filter file %s", patternFile)
-	patternFileContent, err := os.ReadFile(patternFile)
+	patternFileContent, err := ioutil.ReadFile(patternFile)
 	if err == nil {
 		patternFileLines := strings.Split(string(patternFileContent), "\n")
 		patterns = ProcessFilterLines(patternFileLines, includedFiles)
@@ -519,3 +491,4 @@ func encodeSequence(sequence []string) []string {

 	return sequenceInHex
 }
+
--- a/src/duplicacy_snapshotmanager.go
+++ b/src/duplicacy_snapshotmanager.go
@@ -18,10 +18,10 @@ import (
 	"sort"
 	"strconv"
 	"strings"
-	"sync"
-	"sync/atomic"
 	"text/tabwriter"
 	"time"
+	"sync"
+	"sync/atomic"

 	"github.com/aryann/difflib"
 )
@@ -1491,8 +1491,7 @@ func (manager *SnapshotManager) PrintFile(snapshotID string, revision int, path

 // Diff compares two snapshots, or two revision of a file if the file argument is given.
 func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []int,
-	filePath string, compareByHash bool,
-	options *ListFilesOptions) bool {
+	filePath string, compareByHash bool, nobackupFile string, filtersFile string, excludeByAttribute bool) bool {

 	LOG_DEBUG("DIFF_PARAMETERS", "top: %s, id: %s, revision: %v, path: %s, compareByHash: %t",
 		top, snapshotID, revisions, filePath, compareByHash)
@@ -1517,7 +1516,7 @@ func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []
 			localListingChannel := make(chan *Entry)
 			go func() {
 				defer CatchLogException()
-				rightSnapshot.ListLocalFiles(top, localListingChannel, nil, nil, options)
+				rightSnapshot.ListLocalFiles(top, nobackupFile, filtersFile, excludeByAttribute, localListingChannel, nil, nil)
 			} ()

 			for entry := range localListingChannel {
@@ -1822,7 +1821,6 @@ func (manager *SnapshotManager) resurrectChunk(fossilPath string, chunkID string

 // PruneSnapshots deletes snapshots by revisions, tags, or a retention policy.  The main idea is two-step
 // fossil collection.
-//
 // 1. Delete snapshots specified by revision, retention policy, with a tag.  Find any resulting unreferenced
 //    chunks, and mark them as fossils (by renaming).  After that, create a fossil collection file containing
 //    fossils collected during current run, and temporary files encountered.  Also in the file is the latest
@@ -2570,7 +2568,7 @@ func (manager *SnapshotManager) CheckSnapshot(snapshot *Snapshot) (err error) {
 		}

 		if entry.EndChunk < entry.StartChunk {
-			err = fmt.Errorf("The file %s starts at chunk %d and ends at chunk %d",
+			fmt.Errorf("The file %s starts at chunk %d and ends at chunk %d",
 				entry.Path, entry.StartChunk, entry.EndChunk)
 			return false
 		}
--- a/src/duplicacy_storage.go
+++ b/src/duplicacy_storage.go
@@ -756,8 +756,6 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 			LOG_ERROR("STORAGE_CREATE", "Failed to load the Storj storage at %s: %v", storageURL, err)
 			return nil
 		}
-		SavePassword(preference, "storj_key", apiKey)
-		SavePassword(preference, "storj_passphrase", passphrase)
 		return storjStorage
 	} else if matched[1] == "smb" {
 		server := matched[3]
--- a/src/duplicacy_utils_bsd_common.go
+++ b/src/duplicacy_utils_bsd_common.go
@@ -0,0 +1,94 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+//go:build freebsd || netbsd || darwin
+// +build freebsd netbsd darwin
+
+package duplicacy
+
+import (
+	"bytes"
+	"encoding/binary"
+	"os"
+	"path/filepath"
+	"syscall"
+
+	"github.com/pkg/xattr"
+)
+
+const bsdFileFlagsKey = "\x00bf"
+
+func (entry *Entry) ReadAttributes(top string) {
+	fullPath := filepath.Join(top, entry.Path)
+	fileInfo, err := os.Lstat(fullPath)
+	if err != nil {
+		return
+	}
+
+	if !entry.IsSpecial() {
+		attributes, _ := xattr.LList(fullPath)
+		if len(attributes) > 0 {
+			entry.Attributes = &map[string][]byte{}
+			for _, name := range attributes {
+				attribute, err := xattr.LGet(fullPath, name)
+				if err == nil {
+					(*entry.Attributes)[name] = attribute
+				}
+			}
+		}
+	}
+	if err := entry.readFileFlags(fileInfo); err != nil {
+		LOG_INFO("ATTR_BACKUP", "Could not backup flags for file %s: %v", fullPath, err)
+	}
+}
+
+func (entry *Entry) SetAttributesToFile(fullPath string) {
+	if !entry.IsSpecial() {
+		names, _ := xattr.LList(fullPath)
+		for _, name := range names {
+			newAttribute, found := (*entry.Attributes)[name]
+			if found {
+				oldAttribute, _ := xattr.LGet(fullPath, name)
+				if !bytes.Equal(oldAttribute, newAttribute) {
+					xattr.LSet(fullPath, name, newAttribute)
+				}
+				delete(*entry.Attributes, name)
+			} else {
+				xattr.LRemove(fullPath, name)
+			}
+		}
+
+		for name, attribute := range *entry.Attributes {
+			if len(name) > 0 && name[0] == '\x00' {
+				continue
+			}
+			xattr.LSet(fullPath, name, attribute)
+		}
+	}
+	if err := entry.restoreLateFileFlags(fullPath); err != nil {
+		LOG_DEBUG("ATTR_RESTORE", "Could not restore flags for file %s: %v", fullPath, err)
+	}
+}
+
+func (entry *Entry) readFileFlags(fileInfo os.FileInfo) error {
+	stat, ok := fileInfo.Sys().(*syscall.Stat_t)
+	if ok && stat.Flags != 0 {
+		if entry.Attributes == nil {
+			entry.Attributes = &map[string][]byte{}
+		}
+		v := make([]byte, 4)
+		binary.LittleEndian.PutUint32(v, stat.Flags)
+		(*entry.Attributes)[bsdFileFlagsKey] = v
+		LOG_DEBUG("ATTR_READ", "Read flags 0x%x for %s", stat.Flags, entry.Path)
+	}
+	return nil
+}
+
+func (entry *Entry) RestoreEarlyDirFlags(path string) error {
+	return nil
+}
+
+func (entry *Entry) RestoreEarlyFileFlags(f *os.File) error {
+	return nil
+}
--- a/src/duplicacy_utils_darwin.go
+++ b/src/duplicacy_utils_darwin.go
@@ -8,31 +8,26 @@ import (
 	"encoding/binary"
 	"os"
 	"strings"
-
-	"golang.org/x/sys/unix"
+	"syscall"
 )

 func excludedByAttribute(attributes map[string][]byte) bool {
 	value, ok := attributes["com.apple.metadata:com_apple_backup_excludeItem"]
-	excluded := ok && strings.Contains(string(value), "com.apple.backupd")
-	if !excluded {
-		flags, ok := attributes[darwinFileFlagsKey]
-		excluded = ok && (binary.LittleEndian.Uint32(flags)&unix.UF_NODUMP) != 0
-	}
-	return excluded
+	return ok && strings.Contains(string(value), "com.apple.backupd")
 }

-func (entry *Entry) RestoreSpecial(fullPath string) error {
-	mode := entry.Mode & uint32(fileModeMask)
-
-	if entry.Mode&uint32(os.ModeNamedPipe) != 0 {
-		mode |= unix.S_IFIFO
-	} else if entry.Mode&uint32(os.ModeCharDevice) != 0 {
-		mode |= unix.S_IFCHR
-	} else if entry.Mode&uint32(os.ModeDevice) != 0 {
-		mode |= unix.S_IFBLK
-	} else {
+func (entry *Entry) restoreLateFileFlags(path string) error {
+	if entry.Attributes == nil {
 		return nil
 	}
-	return unix.Mknod(fullPath, mode, int(entry.GetRdev()))
+	if v, have := (*entry.Attributes)[bsdFileFlagsKey]; have {
+		f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_SYMLINK, 0)
+		if err != nil {
+			return err
+		}
+		err = syscall.Fchflags(int(f.Fd()), int(binary.LittleEndian.Uint32(v)))
+		f.Close()
+		return err
+	}
+	return nil
 }
--- a/src/duplicacy_utils_linux.go
+++ b/src/duplicacy_utils_linux.go
@@ -5,34 +5,221 @@
 package duplicacy

 import (
+	"bytes"
 	"encoding/binary"
 	"os"
+	"path/filepath"
+	"syscall"
+	"unsafe"

-	"golang.org/x/sys/unix"
+	"github.com/pkg/xattr"
 )

-func excludedByAttribute(attributes map[string][]byte) bool {
-	_, excluded := attributes["user.duplicacy_exclude"]
-	if !excluded {
-		flags, ok := attributes[linuxFileFlagsKey]
-		excluded = ok && (binary.LittleEndian.Uint32(flags)&linux_FS_NODUMP_FL) != 0
-	}
-	return excluded
-}
+const (
+	linux_FS_SECRM_FL        = 0x00000001 /* Secure deletion */
+	linux_FS_UNRM_FL         = 0x00000002 /* Undelete */
+	linux_FS_COMPR_FL        = 0x00000004 /* Compress file */
+	linux_FS_SYNC_FL         = 0x00000008 /* Synchronous updates */
+	linux_FS_IMMUTABLE_FL    = 0x00000010 /* Immutable file */
+	linux_FS_APPEND_FL       = 0x00000020 /* writes to file may only append */
+	linux_FS_NODUMP_FL       = 0x00000040 /* do not dump file */
+	linux_FS_NOATIME_FL      = 0x00000080 /* do not update atime */
+	linux_FS_NOCOMP_FL       = 0x00000400 /* Don't compress */
+	linux_FS_JOURNAL_DATA_FL = 0x00004000 /* Reserved for ext3 */
+	linux_FS_NOTAIL_FL       = 0x00008000 /* file tail should not be merged */
+	linux_FS_DIRSYNC_FL      = 0x00010000 /* dirsync behaviour (directories only) */
+	linux_FS_TOPDIR_FL       = 0x00020000 /* Top of directory hierarchies*/
+	linux_FS_NOCOW_FL        = 0x00800000 /* Do not cow file */
+	linux_FS_PROJINHERIT_FL  = 0x20000000 /* Create with parents projid */

-func (entry *Entry) RestoreSpecial(fullPath string) error {
-	mode := entry.Mode & uint32(fileModeMask)
+	linux_FS_IOC_GETFLAGS uintptr = 0x80086601
+	linux_FS_IOC_SETFLAGS uintptr = 0x40086602

-	if entry.Mode&uint32(os.ModeNamedPipe) != 0 {
-		mode |= unix.S_IFIFO
-	} else if entry.Mode&uint32(os.ModeCharDevice) != 0 {
-		mode |= unix.S_IFCHR
-	} else if entry.Mode&uint32(os.ModeDevice) != 0 {
-		mode |= unix.S_IFBLK
-	} else if entry.Mode&uint32(os.ModeSocket) != 0 {
-		mode |= unix.S_IFSOCK
-	} else {
+	linuxIocFlagsFileEarly = linux_FS_SECRM_FL | linux_FS_UNRM_FL | linux_FS_COMPR_FL | linux_FS_NODUMP_FL | linux_FS_NOATIME_FL | linux_FS_NOCOMP_FL | linux_FS_JOURNAL_DATA_FL | linux_FS_NOTAIL_FL | linux_FS_NOCOW_FL
+	linuxIocFlagsDirEarly  = linux_FS_TOPDIR_FL | linux_FS_PROJINHERIT_FL
+	linuxIocFlagsLate      = linux_FS_SYNC_FL | linux_FS_IMMUTABLE_FL | linux_FS_APPEND_FL | linux_FS_DIRSYNC_FL
+
+	linuxFileFlagsKey = "\x00lf"
+)
+
+func ioctl(f *os.File, request uintptr, attrp *uint32) error {
+	argp := uintptr(unsafe.Pointer(attrp))
+
+	if _, _, errno := syscall.Syscall(syscall.SYS_IOCTL, f.Fd(), request, argp); errno != 0 {
+		return os.NewSyscallError("ioctl", errno)
+	}
 	return nil
 }
-	return unix.Mknod(fullPath, mode, int(entry.GetRdev()))
+
+type xattrHandle struct {
+	f        *os.File
+	fullPath string
+}
+
+func (x xattrHandle) list() ([]string, error) {
+	if x.f != nil {
+		return xattr.FList(x.f)
+	} else {
+		return xattr.LList(x.fullPath)
+	}
+}
+
+func (x xattrHandle) get(name string) ([]byte, error) {
+	if x.f != nil {
+		return xattr.FGet(x.f, name)
+	} else {
+		return xattr.LGet(x.fullPath, name)
+	}
+}
+
+func (x xattrHandle) set(name string, value []byte) error {
+	if x.f != nil {
+		return xattr.FSet(x.f, name, value)
+	} else {
+		return xattr.LSet(x.fullPath, name, value)
+	}
+}
+
+func (x xattrHandle) remove(name string) error {
+	if x.f != nil {
+		return xattr.FRemove(x.f, name)
+	} else {
+		return xattr.LRemove(x.fullPath, name)
+	}
+}
+
+func (entry *Entry) ReadAttributes(top string) {
+	fullPath := filepath.Join(top, entry.Path)
+	x := xattrHandle{nil, fullPath}
+
+	if !entry.IsLink() {
+		var err error
+		x.f, err = os.OpenFile(fullPath, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
+		if err != nil {
+			// FIXME: We really should return errors for failure to read
+			return
+		}
+	}
+
+	attributes, _ := x.list()
+
+	if len(attributes) > 0 {
+		entry.Attributes = &map[string][]byte{}
+	}
+	for _, name := range attributes {
+		attribute, err := x.get(name)
+		if err == nil {
+			(*entry.Attributes)[name] = attribute
+		}
+	}
+
+	if entry.IsFile() || entry.IsDir() {
+		if err := entry.readFileFlags(x.f); err != nil {
+			LOG_INFO("ATTR_BACKUP", "Could not backup flags for file %s: %v", fullPath, err)
+		}
+	}
+	x.f.Close()
+}
+
+func (entry *Entry) SetAttributesToFile(fullPath string) {
+	x := xattrHandle{nil, fullPath}
+	if !entry.IsLink() {
+		var err error
+		x.f, err = os.OpenFile(fullPath, os.O_RDONLY|syscall.O_NOFOLLOW, 0)
+		if err != nil {
+			return
+		}
+	}
+
+	names, _ := x.list()
+
+	for _, name := range names {
+		newAttribute, found := (*entry.Attributes)[name]
+		if found {
+			oldAttribute, _ := x.get(name)
+			if !bytes.Equal(oldAttribute, newAttribute) {
+				x.set(name, newAttribute)
+			}
+			delete(*entry.Attributes, name)
+		} else {
+			x.remove(name)
+		}
+	}
+
+	for name, attribute := range *entry.Attributes {
+		if len(name) > 0 && name[0] == '\x00' {
+			continue
+		}
+		x.set(name, attribute)
+	}
+	if entry.IsFile() || entry.IsDir() {
+		if err := entry.restoreLateFileFlags(x.f); err != nil {
+			LOG_DEBUG("ATTR_RESTORE", "Could not restore flags for file %s: %v", fullPath, err)
+		}
+	}
+	x.f.Close()
+}
+
+func (entry *Entry) readFileFlags(f *os.File) error {
+	var flags uint32
+	if err := ioctl(f, linux_FS_IOC_GETFLAGS, &flags); err != nil {
+		return err
+	}
+	if flags != 0 {
+		if entry.Attributes == nil {
+			entry.Attributes = &map[string][]byte{}
+		}
+		v := make([]byte, 4)
+		binary.LittleEndian.PutUint32(v, flags)
+		(*entry.Attributes)[linuxFileFlagsKey] = v
+		LOG_DEBUG("ATTR_READ", "Read flags 0x%x for %s", flags, entry.Path)
+	}
+	return nil
+}
+
+func (entry *Entry) RestoreEarlyDirFlags(path string) error {
+	if entry.Attributes == nil {
+		return nil
+	}
+	if v, have := (*entry.Attributes)[linuxFileFlagsKey]; have {
+		flags := binary.LittleEndian.Uint32(v) & linuxIocFlagsDirEarly
+		f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_DIRECTORY, 0)
+		if err != nil {
+			return err
+		}
+		LOG_DEBUG("ATTR_RESTORE", "Restore dir flags (early) 0x%x for %s", flags, entry.Path)
+		err = ioctl(f, linux_FS_IOC_SETFLAGS, &flags)
+		f.Close()
+		return err
+	}
+	return nil
+}
+
+func (entry *Entry) RestoreEarlyFileFlags(f *os.File) error {
+	if entry.Attributes == nil {
+		return nil
+	}
+	if v, have := (*entry.Attributes)[linuxFileFlagsKey]; have {
+		flags := binary.LittleEndian.Uint32(v) & linuxIocFlagsFileEarly
+		LOG_DEBUG("ATTR_RESTORE", "Restore flags (early) 0x%x for %s", flags, entry.Path)
+		return ioctl(f, linux_FS_IOC_SETFLAGS, &flags)
+	}
+	return nil
+}
+
+func (entry *Entry) restoreLateFileFlags(f *os.File) error {
+	if entry.Attributes == nil {
+		return nil
+	}
+	if v, have := (*entry.Attributes)[linuxFileFlagsKey]; have {
+		flags := binary.LittleEndian.Uint32(v) & (linuxIocFlagsFileEarly | linuxIocFlagsDirEarly | linuxIocFlagsLate)
+		LOG_DEBUG("ATTR_RESTORE", "Restore flags (late) 0x%x for %s", flags, entry.Path)
+		return ioctl(f, linux_FS_IOC_SETFLAGS, &flags)
+	}
+	return nil
+}
+
+func excludedByAttribute(attributes map[string][]byte) bool {
+	_, ok := attributes["user.duplicacy_exclude"]
+	return ok
 }
--- a/src/duplicacy_utils_others.go
+++ b/src/duplicacy_utils_others.go
@@ -8,7 +8,6 @@
 package duplicacy

 import (
-	"fmt"
 	"os"
 	"path"
 	"syscall"
@@ -21,19 +20,24 @@ func Readlink(path string) (isRegular bool, s string, err error) {
 	return false, s, err
 }

-func GetOwner(entry *Entry, fileInfo os.FileInfo) {
-	stat := fileInfo.Sys().(*syscall.Stat_t)
+func GetOwner(entry *Entry, fileInfo *os.FileInfo) {
+	stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
+	if ok && stat != nil {
 		entry.UID = int(stat.Uid)
 		entry.GID = int(stat.Gid)
+	} else {
+		entry.UID = -1
+		entry.GID = -1
+	}
 }

-func SetOwner(fullPath string, entry *Entry, fileInfo os.FileInfo) bool {
-	stat := fileInfo.Sys().(*syscall.Stat_t)
-	if (int(stat.Uid) != entry.UID || int(stat.Gid) != entry.GID) {
+func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool {
+	stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
+	if ok && stat != nil && (int(stat.Uid) != entry.UID || int(stat.Gid) != entry.GID) {
 		if entry.UID != -1 && entry.GID != -1 {
 			err := os.Lchown(fullPath, entry.UID, entry.GID)
 			if err != nil {
-				LOG_ERROR("RESTORE_CHOWN", "Failed to change uid or gid on %s: %v", entry.Path, err)
+				LOG_ERROR("RESTORE_CHOWN", "Failed to change uid or gid: %v", err)
 				return false
 			}
 		}
@@ -42,65 +46,48 @@ func SetOwner(fullPath string, entry *Entry, fileInfo os.FileInfo) bool {
 	return true
 }

-type listEntryLinkKey struct {
-	dev uint64
-	ino uint64
-}
-
-func (entry *Entry) getHardLinkKey(f os.FileInfo) (key listEntryLinkKey, linked bool) {
-	if entry.IsDir() {
-		return
-	}
-	stat := f.Sys().(*syscall.Stat_t)
-	if stat.Nlink <= 1 {
-		return
-	}
-	key.dev = uint64(stat.Dev)
-	key.ino = uint64(stat.Ino)
-	linked = true
-	return
-}
-
-func (entry *Entry) ReadSpecial(fullPath string, fileInfo os.FileInfo) error {
+func (entry *Entry) ReadSpecial(fileInfo os.FileInfo) bool {
 	if fileInfo.Mode()&(os.ModeDevice|os.ModeCharDevice) == 0 {
-		return nil
+		return true
+	}
+	stat := fileInfo.Sys().(*syscall.Stat_t)
+	if stat == nil {
+		return false
 	}
-	rdev := uint64(fileInfo.Sys().(*syscall.Stat_t).Rdev)
 	entry.Size = 0
+	rdev := uint64(stat.Rdev)
 	entry.StartChunk = int(rdev & 0xFFFFFFFF)
 	entry.StartOffset = int(rdev >> 32)
-	return nil
+	return true
 }

 func (entry *Entry) GetRdev() uint64 {
 	return uint64(entry.StartChunk) | uint64(entry.StartOffset)<<32
 }

+func (entry *Entry) RestoreSpecial(fullPath string) error {
+	mode := entry.Mode & uint32(fileModeMask)
+
+	if entry.Mode&uint32(os.ModeNamedPipe) != 0 {
+		mode |= syscall.S_IFIFO
+	} else if entry.Mode&uint32(os.ModeCharDevice) != 0 {
+		mode |= syscall.S_IFCHR
+	} else if entry.Mode&uint32(os.ModeDevice) != 0 {
+		mode |= syscall.S_IFBLK
+	} else {
+		return nil
+	}
+	return syscall.Mknod(fullPath, mode, int(entry.GetRdev()))
+}
+
 func (entry *Entry) IsSameSpecial(fileInfo os.FileInfo) bool {
 	stat := fileInfo.Sys().(*syscall.Stat_t)
+	if stat == nil {
+		return false
+	}
 	return (uint32(fileInfo.Mode()) == entry.Mode) && (uint64(stat.Rdev) == entry.GetRdev())
 }

-func (entry *Entry) FmtSpecial() string {
-	var c string
-	mode := entry.Mode & uint32(os.ModeType)
-
-	if mode&uint32(os.ModeNamedPipe) != 0 {
-		c = "p"
-	} else if mode&uint32(os.ModeCharDevice) != 0 {
-		c = "c"
-	} else if mode&uint32(os.ModeDevice) != 0 {
-		c = "b"
-	} else if mode&uint32(os.ModeSocket) != 0 {
-		c = "s"
-	} else {
-		return ""
-	}
-
-	rdev := entry.GetRdev()
-	return fmt.Sprintf("%s (%d, %d)", c, unix.Major(rdev), unix.Minor(rdev))
-}
-
 func MakeHardlink(source string, target string) error {
 	return unix.Linkat(unix.AT_FDCWD, source, unix.AT_FDCWD, target, 0)
 }
--- a/src/duplicacy_utils_posix.go
+++ b/src/duplicacy_utils_posix.go
@@ -0,0 +1,13 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+//go:build freebsd || netbsd || solaris
+// +build freebsd netbsd solaris
+
+package duplicacy
+
+func excludedByAttribute(attributes map[string][]byte) bool {
+	_, ok := attributes["user.duplicacy_exclude"]
+	return ok
+}
--- a/src/duplicacy_utils_windows.go
+++ b/src/duplicacy_utils_windows.go
@@ -56,11 +56,7 @@ const (

 // Readlink returns the destination of the named symbolic link.
 func Readlink(path string) (isRegular bool, s string, err error) {
-	pPath, err := syscall.UTF16PtrFromString(path)
-	if err != nil {
-		return false, "", err
-	}
-	fd, err := syscall.CreateFile(pPath, FILE_READ_ATTRIBUTES,
+	fd, err := syscall.CreateFile(syscall.StringToUTF16Ptr(path), FILE_READ_ATTRIBUTES,
 		syscall.FILE_SHARE_READ, nil, syscall.OPEN_EXISTING,
 		syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS, 0)
 	if err != nil {
@@ -105,15 +101,30 @@ func Readlink(path string) (isRegular bool, s string, err error) {
 	return false, s, nil
 }

-func GetOwner(entry *Entry, fileInfo os.FileInfo) {
+func GetOwner(entry *Entry, fileInfo *os.FileInfo) {
 	entry.UID = -1
 	entry.GID = -1
 }

-func SetOwner(fullPath string, entry *Entry, fileInfo os.FileInfo) bool {
+func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool {
 	return true
 }

+func (entry *Entry) ReadAttributes(top string) {
+}
+
+func (entry *Entry) SetAttributesToFile(fullPath string) {
+
+}
+
+func (entry *Entry) ReadDeviceNode(fileInfo os.FileInfo) bool {
+	return nil
+}
+
+func (entry *Entry) RestoreSpecial(fullPath string) error {
+	return nil
+}
+
 func MakeHardlink(source string, target string) error {
 	return os.Link(source, target)
 }
@@ -133,28 +144,18 @@ func SplitDir(fullPath string) (dir string, file string) {
 	return fullPath[:i+1], fullPath[i+1:]
 }

-func excludedByAttribute(attributes map[string][]byte) bool {
-	return false
-}
-
-type listEntryLinkKey struct{}
-
-func (entry *Entry) getHardLinkKey(f os.FileInfo) (key listEntryLinkKey, linked bool) {
-	return
-}
-
-func (entry *Entry) ReadSpecial(fullPath string, fileInfo os.FileInfo) error {
+func (entry *Entry) ReadFileFlags(f *os.File) error {
 	return nil
 }

-func (entry *Entry) IsSameSpecial(fileInfo os.FileInfo) bool {
-	return false
-}
-
-func (entry *Entry) RestoreSpecial(fullPath string) error {
+func (entry *Entry) RestoreEarlyDirFlags(path string) error {
 	return nil
 }

-func (entry *Entry) FmtSpecial() string {
-	return ""
+func (entry *Entry) RestoreEarlyFileFlags(f *os.File) error {
+	return nil
+}
+
+func (entry *Entry) RestoreLateFileFlags(f *os.File) error {
+	return nil
 }
--- a/src/duplicacy_utils_xbsd.go
+++ b/src/duplicacy_utils_xbsd.go
@@ -2,37 +2,29 @@
 // Free for personal use and commercial trial
 // Commercial use requires per-user licenses available from https://duplicacy.com

-//go:build freebsd
-// +build freebsd
+//go:build freebsd || netbsd
+// +build freebsd netbsd

 package duplicacy

 import (
+	"bytes"
 	"encoding/binary"
 	"os"
+	"path/filepath"
 	"syscall"
+
+	"github.com/pkg/xattr"
 )

-func excludedByAttribute(attributes map[string][]byte) bool {
-	_, excluded := attributes["duplicacy_exclude"]
-	if !excluded {
-		flags, ok := attributes[bsdFileFlagsKey]
-		excluded = ok && (binary.LittleEndian.Uint32(flags)&bsd_UF_NODUMP) != 0
-	}
-	return excluded
-}
-
-func (entry *Entry) RestoreSpecial(fullPath string) error {
-	mode := entry.Mode & uint32(fileModeMask)
-
-	if entry.Mode&uint32(os.ModeNamedPipe) != 0 {
-		mode |= syscall.S_IFIFO
-	} else if entry.Mode&uint32(os.ModeCharDevice) != 0 {
-		mode |= syscall.S_IFCHR
-	} else if entry.Mode&uint32(os.ModeDevice) != 0 {
-		mode |= syscall.S_IFBLK
-	} else {
+func (entry *Entry) restoreLateFileFlags(path string) error {
+	if entry.Attributes == nil {
 		return nil
 	}
-	return syscall.Mknod(fullPath, mode, entry.GetRdev())
+	if v, have := (*entry.Attributes)[bsdFileFlagsKey]; have {
+		if _, _, errno := syscall.Syscall(syscall.SYS_LCHFLAGS, uintptr(unsafe.Pointer(syscall.StringBytePtr(path))), uintptr(v), 0); errno != 0 {
+			return os.NewSyscallError("lchflags", errno)
+		}
+	}
+	return nil
 }
--- a/src/duplicacy_xattr.go
+++ b/src/duplicacy_xattr.go
@@ -1,35 +0,0 @@
-// Copyright (c) Acrosync LLC. All rights reserved.
-// Free for personal use and commercial trial
-// Commercial use requires per-user licenses available from https://duplicacy.com
-
-package duplicacy
-
-import "os"
-
-func (entry *Entry) ReadAttributes(fi os.FileInfo, fullPath string, normalize bool) error {
-	return entry.readAttributes(fi, fullPath, normalize)
-}
-
-func (entry *Entry) GetFileFlags(fileInfo os.FileInfo) bool {
-	return entry.getFileFlags(fileInfo)
-}
-
-func (entry *Entry) ReadFileFlags(fileInfo os.FileInfo, fullPath string) error {
-	return entry.readFileFlags(fileInfo, fullPath)
-}
-
-func (entry *Entry) RestoreEarlyDirFlags(fullPath string, mask uint32) error {
-	return entry.restoreEarlyDirFlags(fullPath, mask)
-}
-
-func (entry *Entry) RestoreEarlyFileFlags(f *os.File, mask uint32) error {
-	return entry.restoreEarlyFileFlags(f, mask)
-}
-
-func (entry *Entry) RestoreLateFileFlags(fullPath string, fileInfo os.FileInfo, mask uint32) error {
-	return entry.restoreLateFileFlags(fullPath, fileInfo, mask)
-}
-
-func (entry *Entry) SetAttributesToFile(fullPath string, normalize bool) error {
-	return entry.setAttributesToFile(fullPath, normalize)
-}
--- a/src/duplicacy_xattr_darwin.go
+++ b/src/duplicacy_xattr_darwin.go
@@ -1,149 +0,0 @@
-// Copyright (c) Acrosync LLC. All rights reserved.
-// Free for personal use and commercial trial
-// Commercial use requires per-user licenses available from https://duplicacy.com
-
-package duplicacy
-
-import (
-	"bytes"
-	"encoding/binary"
-	"errors"
-	"math"
-	"os"
-	"syscall"
-
-	"github.com/pkg/xattr"
-	"golang.org/x/sys/unix"
-)
-
-const (
-	darwinFileFlagsKey = "\x00bf"
-)
-
-var darwinIsSuperUser bool
-
-func init() {
-	darwinIsSuperUser = unix.Geteuid() == 0
-}
-
-func (entry *Entry) readAttributes(fi os.FileInfo, fullPath string, normalize bool) error {
-	if entry.IsSpecial() {
-		return nil
-	}
-
-	attributes, err := xattr.LList(fullPath)
-	if err != nil {
-		return err
-	}
-
-	if len(attributes) > 0 {
-		entry.Attributes = &map[string][]byte{}
-	}
-	var allErrors error
-	for _, name := range attributes {
-		value, err := xattr.LGet(fullPath, name)
-		if err != nil {
-			allErrors = errors.Join(allErrors, err)
-		} else {
-			(*entry.Attributes)[name] = value
-		}
-	}
-
-	return allErrors
-}
-
-func (entry *Entry) getFileFlags(fileInfo os.FileInfo) bool {
-	stat := fileInfo.Sys().(*syscall.Stat_t)
-	if stat.Flags != 0 {
-		if entry.Attributes == nil {
-			entry.Attributes = &map[string][]byte{}
-		}
-		v := make([]byte, 4)
-		binary.LittleEndian.PutUint32(v, stat.Flags)
-		(*entry.Attributes)[darwinFileFlagsKey] = v
-	}
-	return true
-}
-
-func (entry *Entry) readFileFlags(fileInfo os.FileInfo, fullPath string) error {
-	return nil
-}
-
-func (entry *Entry) setAttributesToFile(fullPath string, normalize bool) error {
-	if entry.Attributes == nil || len(*entry.Attributes) == 0 || entry.IsSpecial() {
-		return nil
-	}
-	attributes := *entry.Attributes
-
-	if _, haveFlags := attributes[darwinFileFlagsKey]; haveFlags && len(attributes) <= 1 {
-		return nil
-	}
-
-	names, err := xattr.LList(fullPath)
-	if err != nil {
-		return err
-	}
-	for _, name := range names {
-		newAttribute, found := attributes[name]
-		if found {
-			oldAttribute, _ := xattr.LGet(fullPath, name)
-			if !bytes.Equal(oldAttribute, newAttribute) {
-				err = errors.Join(err, xattr.LSet(fullPath, name, newAttribute))
-			}
-			delete(attributes, name)
-		} else {
-			err = errors.Join(err, xattr.LRemove(fullPath, name))
-		}
-	}
-
-	for name, attribute := range attributes {
-		if len(name) > 0 && name[0] == '\x00' {
-			continue
-		}
-		err = errors.Join(err, xattr.LSet(fullPath, name, attribute))
-	}
-	return err
-}
-
-func (entry *Entry) restoreEarlyDirFlags(fullPath string, mask uint32) error {
-	return nil
-}
-
-func (entry *Entry) restoreEarlyFileFlags(f *os.File, mask uint32) error {
-	return nil
-}
-
-func (entry *Entry) restoreLateFileFlags(fullPath string, fileInfo os.FileInfo, mask uint32) error {
-	if mask == math.MaxUint32 {
-		return nil
-	}
-
-	if darwinIsSuperUser {
-		mask |= ^uint32(unix.UF_SETTABLE | unix.SF_SETTABLE)
-	} else {
-		mask |= ^uint32(unix.UF_SETTABLE)
-	}
-
-	var flags uint32
-
-	if entry.Attributes != nil {
-		if v, have := (*entry.Attributes)[darwinFileFlagsKey]; have {
-			flags = binary.LittleEndian.Uint32(v)
-		}
-	}
-
-	stat := fileInfo.Sys().(*syscall.Stat_t)
-
-	flags = (flags & ^mask) | (stat.Flags & mask)
-
-	if flags != stat.Flags {
-		f, err := os.OpenFile(fullPath, os.O_RDONLY|unix.O_SYMLINK, 0)
-		if err != nil {
-			return err
-		}
-		err = unix.Fchflags(int(f.Fd()), int(flags))
-		f.Close()
-		return err
-	}
-	return nil
-}
--- a/src/duplicacy_xattr_linux.go
+++ b/src/duplicacy_xattr_linux.go
@@ -1,234 +0,0 @@
-// Copyright (c) Acrosync LLC. All rights reserved.
-// Free for personal use and commercial trial
-// Commercial use requires per-user licenses available from https://duplicacy.com
-
-package duplicacy
-
-import (
-	"bytes"
-	"encoding/binary"
-	"errors"
-	"fmt"
-	"math"
-	"os"
-	"unsafe"
-
-	"github.com/pkg/xattr"
-	"golang.org/x/sys/unix"
-)
-
-const (
-	linux_FS_SECRM_FL        = 0x00000001 /* Secure deletion */
-	linux_FS_UNRM_FL         = 0x00000002 /* Undelete */
-	linux_FS_COMPR_FL        = 0x00000004 /* Compress file */
-	linux_FS_SYNC_FL         = 0x00000008 /* Synchronous updates */
-	linux_FS_IMMUTABLE_FL    = 0x00000010 /* Immutable file */
-	linux_FS_APPEND_FL       = 0x00000020 /* writes to file may only append */
-	linux_FS_NODUMP_FL       = 0x00000040 /* do not dump file */
-	linux_FS_NOATIME_FL      = 0x00000080 /* do not update atime */
-	linux_FS_NOCOMP_FL       = 0x00000400 /* Don't compress */
-	linux_FS_JOURNAL_DATA_FL = 0x00004000 /* Reserved for ext3 */
-	linux_FS_NOTAIL_FL       = 0x00008000 /* file tail should not be merged */
-	linux_FS_DIRSYNC_FL      = 0x00010000 /* dirsync behaviour (directories only) */
-	linux_FS_TOPDIR_FL       = 0x00020000 /* Top of directory hierarchies*/
-	linux_FS_NOCOW_FL        = 0x00800000 /* Do not cow file */
-	linux_FS_PROJINHERIT_FL  = 0x20000000 /* Create with parents projid */
-
-	linuxIocFlagsFileEarly = linux_FS_SECRM_FL | linux_FS_UNRM_FL | linux_FS_COMPR_FL | linux_FS_NODUMP_FL | linux_FS_NOATIME_FL | linux_FS_NOCOMP_FL | linux_FS_JOURNAL_DATA_FL | linux_FS_NOTAIL_FL | linux_FS_NOCOW_FL
-	linuxIocFlagsDirEarly  = linux_FS_TOPDIR_FL | linux_FS_PROJINHERIT_FL
-	linuxIocFlagsLate      = linux_FS_SYNC_FL | linux_FS_IMMUTABLE_FL | linux_FS_APPEND_FL | linux_FS_DIRSYNC_FL
-
-	linuxFileFlagsKey = "\x00lf"
-)
-
-var (
-	errENOTTY error = unix.ENOTTY
-)
-
-func ignoringEINTR(fn func() error) (err error) {
-	for {
-		err = fn()
-		if err != unix.EINTR {
-			break
-		}
-	}
-	return err
-}
-
-func ioctl(f *os.File, request uintptr, attrp *uint32) error {
-	return ignoringEINTR(func() error {
-		argp := uintptr(unsafe.Pointer(attrp))
-
-		_, _, errno := unix.Syscall(unix.SYS_IOCTL, f.Fd(), request, argp)
-		if errno == 0 {
-			return nil
-		} else if errno == unix.ENOTTY {
-			return errENOTTY
-		}
-		return errno
-	})
-}
-
-func (entry *Entry) readAttributes(fi os.FileInfo, fullPath string, normalize bool) error {
-	attributes, err := xattr.LList(fullPath)
-	if err != nil {
-		return err
-	}
-
-	if len(attributes) > 0 {
-		entry.Attributes = &map[string][]byte{}
-	}
-	var allErrors error
-	for _, name := range attributes {
-		value, err := xattr.LGet(fullPath, name)
-		if err != nil {
-			allErrors = errors.Join(allErrors, err)
-		} else {
-			(*entry.Attributes)[name] = value
-		}
-	}
-
-	return allErrors
-}
-
-func (entry *Entry) getFileFlags(fileInfo os.FileInfo) bool {
-	return false
-}
-
-func (entry *Entry) readFileFlags(fileInfo os.FileInfo, fullPath string) error {
-	// the linux file flags interface is quite depressing. The half assed attempt at statx
-	// doesn't even cover the flags we're usually interested in for btrfs
-	if !(entry.IsFile() || entry.IsDir()) {
-		return nil
-	}
-
-	f, err := os.OpenFile(fullPath, os.O_RDONLY|unix.O_NONBLOCK|unix.O_NOFOLLOW|unix.O_NOATIME, 0)
-	if err != nil {
-		return err
-	}
-
-	var flags uint32
-
-	err = ioctl(f, unix.FS_IOC_GETFLAGS, &flags)
-	f.Close()
-	if err != nil {
-		// inappropriate ioctl for device means flags aren't a thing on that FS
-		if err == unix.ENOTTY {
-			return nil
-		}
-		return err
-	}
-
-	if flags != 0 {
-		if entry.Attributes == nil {
-			entry.Attributes = &map[string][]byte{}
-		}
-		v := make([]byte, 4)
-		binary.LittleEndian.PutUint32(v, flags)
-		(*entry.Attributes)[linuxFileFlagsKey] = v
-	}
-	return nil
-}
-
-func (entry *Entry) setAttributesToFile(fullPath string, normalize bool) error {
-	if entry.Attributes == nil || len(*entry.Attributes) == 0 {
-		return nil
-	}
-	attributes := *entry.Attributes
-
-	if _, haveFlags := attributes[linuxFileFlagsKey]; haveFlags && len(attributes) <= 1 {
-		return nil
-	}
-
-	names, err := xattr.LList(fullPath)
-	if err != nil {
-		return err
-	}
-	for _, name := range names {
-		newAttribute, found := (*entry.Attributes)[name]
-		if found {
-			oldAttribute, _ := xattr.LGet(fullPath, name)
-			if !bytes.Equal(oldAttribute, newAttribute) {
-				err = errors.Join(err, xattr.LSet(fullPath, name, newAttribute))
-			}
-			delete(*entry.Attributes, name)
-		} else {
-			err = errors.Join(err, xattr.LRemove(fullPath, name))
-		}
-	}
-
-	for name, attribute := range *entry.Attributes {
-		if len(name) > 0 && name[0] == '\x00' {
-			continue
-		}
-		err = errors.Join(err, xattr.LSet(fullPath, name, attribute))
-	}
-	return err
-}
-
-func (entry *Entry) restoreEarlyDirFlags(fullPath string, mask uint32) error {
-	if entry.Attributes == nil || mask == math.MaxUint32 {
-		return nil
-	}
-	var flags uint32
-
-	if v, have := (*entry.Attributes)[linuxFileFlagsKey]; have {
-		flags = binary.LittleEndian.Uint32(v) & linuxIocFlagsDirEarly & ^mask
-	}
-
-	if flags != 0 {
-		f, err := os.OpenFile(fullPath, os.O_RDONLY|unix.O_DIRECTORY, 0)
-		if err != nil {
-			return err
-		}
-		err = ioctl(f, unix.FS_IOC_SETFLAGS, &flags)
-		f.Close()
-		if err != nil {
-			return fmt.Errorf("Set flags 0x%.8x failed: %w", flags, err)
-		}
-	}
-	return nil
-}
-
-func (entry *Entry) restoreEarlyFileFlags(f *os.File, mask uint32) error {
-	if entry.Attributes == nil || mask == math.MaxUint32 {
-		return nil
-	}
-	var flags uint32
-
-	if v, have := (*entry.Attributes)[linuxFileFlagsKey]; have {
-		flags = binary.LittleEndian.Uint32(v) & linuxIocFlagsFileEarly & ^mask
-	}
-
-	if flags != 0 {
-		err := ioctl(f, unix.FS_IOC_SETFLAGS, &flags)
-		if err != nil {
-			return fmt.Errorf("Set flags 0x%.8x failed: %w", flags, err)
-		}
-	}
-	return nil
-}
-
-func (entry *Entry) restoreLateFileFlags(fullPath string, fileInfo os.FileInfo, mask uint32) error {
-	if entry.IsLink() || entry.Attributes == nil || mask == math.MaxUint32 {
-		return nil
-	}
-	var flags uint32
-
-	if v, have := (*entry.Attributes)[linuxFileFlagsKey]; have {
-		flags = binary.LittleEndian.Uint32(v) & (linuxIocFlagsFileEarly | linuxIocFlagsDirEarly | linuxIocFlagsLate) & ^mask
-	}
-
-	if flags != 0 {
-		f, err := os.OpenFile(fullPath, os.O_RDONLY|unix.O_NOFOLLOW, 0)
-		if err != nil {
-			return err
-		}
-		err = ioctl(f, unix.FS_IOC_SETFLAGS, &flags)
-		f.Close()
-		if err != nil {
-			return fmt.Errorf("Set flags 0x%.8x failed: %w", flags, err)
-		}
-	}
-	return nil
-}
--- a/src/duplicacy_xattr_windows.go
+++ b/src/duplicacy_xattr_windows.go
@@ -1,35 +0,0 @@
-// Copyright (c) Acrosync LLC. All rights reserved.
-// Free for personal use and commercial trial
-// Commercial use requires per-user licenses available from https://duplicacy.com
-
-package duplicacy
-
-import "os"
-
-func (entry *Entry) readAttributes(fi os.FileInfo, fullPath string, normalize bool) error {
-	return nil
-}
-
-func (entry *Entry) getFileFlags(fileInfo os.FileInfo) bool {
-	return true
-}
-
-func (entry *Entry) readFileFlags(fileInfo os.FileInfo, fullPath string) error {
-	return nil
-}
-
-func (entry *Entry) setAttributesToFile(fullPath string, normalize bool) error {
-	return nil
-}
-
-func (entry *Entry) restoreEarlyDirFlags(fullPath string, mask uint32) error {
-	return nil
-}
-
-func (entry *Entry) restoreEarlyFileFlags(f *os.File, mask uint32) error {
-	return nil
-}
-
-func (entry *Entry) restoreLateFileFlags(fullPath string, fileInfo os.FileInfo, mask uint32) error {
-	return nil
-}
--- a/src/duplicacy_xattr_xbsd.go
+++ b/src/duplicacy_xattr_xbsd.go
@@ -1,155 +0,0 @@
-// Copyright (c) Acrosync LLC. All rights reserved.
-// Free for personal use and commercial trial
-// Commercial use requires per-user licenses available from https://duplicacy.com
-
-//go:build freebsd || netbsd
-// +build freebsd netbsd
-
-package duplicacy
-
-import (
-	"bytes"
-	"encoding/binary"
-	"errors"
-	"math"
-	"os"
-	"syscall"
-	"unsafe"
-
-	"github.com/pkg/xattr"
-)
-
-const (
-	bsd_UF_NODUMP   = 0x1
-	bsd_SF_SETTABLE = 0xffff0000
-	bsd_UF_SETTABLE = 0x0000ffff
-
-	bsdFileFlagsKey = "\x00bf"
-)
-
-var bsdIsSuperUser bool
-
-func init() {
-	bsdIsSuperUser = syscall.Geteuid() == 0
-}
-
-func (entry *Entry) readAttributes(fi os.FileInfo, fullPath string, normalize bool) error {
-	if entry.IsSpecial() {
-		return nil
-	}
-
-	attributes, err := xattr.LList(fullPath)
-	if err != nil {
-		return err
-	}
-
-	if len(attributes) > 0 {
-		entry.Attributes = &map[string][]byte{}
-	}
-	var allErrors error
-	for _, name := range attributes {
-		value, err := xattr.LGet(fullPath, name)
-		if err != nil {
-			allErrors = errors.Join(allErrors, err)
-		} else {
-			(*entry.Attributes)[name] = value
-		}
-	}
-
-	return allErrors
-}
-
-func (entry *Entry) getFileFlags(fileInfo os.FileInfo) bool {
-	stat := fileInfo.Sys().(*syscall.Stat_t)
-	if stat.Flags != 0 {
-		if entry.Attributes == nil {
-			entry.Attributes = &map[string][]byte{}
-		}
-		v := make([]byte, 4)
-		binary.LittleEndian.PutUint32(v, stat.Flags)
-		(*entry.Attributes)[bsdFileFlagsKey] = v
-	}
-	return true
-}
-
-func (entry *Entry) readFileFlags(fileInfo os.FileInfo, fullPath string) error {
-	return nil
-}
-
-func (entry *Entry) setAttributesToFile(fullPath string, normalize bool) error {
-	if entry.Attributes == nil || len(*entry.Attributes) == 0 || entry.IsSpecial() {
-		return nil
-	}
-	attributes := *entry.Attributes
-
-	if _, haveFlags := attributes[bsdFileFlagsKey]; haveFlags && len(attributes) <= 1 {
-		return nil
-	}
-
-	names, err := xattr.LList(fullPath)
-	if err != nil {
-		return err
-	}
-	for _, name := range names {
-		newAttribute, found := attributes[name]
-		if found {
-			oldAttribute, _ := xattr.LGet(fullPath, name)
-			if !bytes.Equal(oldAttribute, newAttribute) {
-				err = errors.Join(err, xattr.LSet(fullPath, name, newAttribute))
-			}
-			delete(attributes, name)
-		} else {
-			err = errors.Join(err, xattr.LRemove(fullPath, name))
-		}
-	}
-
-	for name, attribute := range attributes {
-		if len(name) > 0 && name[0] == '\x00' {
-			continue
-		}
-		err = errors.Join(err, xattr.LSet(fullPath, name, attribute))
-	}
-	return err
-}
-
-func (entry *Entry) restoreEarlyDirFlags(fullPath string, mask uint32) error {
-	return nil
-}
-
-func (entry *Entry) restoreEarlyFileFlags(f *os.File, mask uint32) error {
-	return nil
-}
-
-func (entry *Entry) restoreLateFileFlags(fullPath string, fileInfo os.FileInfo, mask uint32) error {
-	if mask == math.MaxUint32 {
-		return nil
-	}
-
-	if bsdIsSuperUser {
-		mask |= ^uint32(bsd_UF_SETTABLE | bsd_SF_SETTABLE)
-	} else {
-		mask |= ^uint32(bsd_UF_SETTABLE)
-	}
-
-	var flags uint32
-
-	if entry.Attributes != nil {
-		if v, have := (*entry.Attributes)[bsdFileFlagsKey]; have {
-			flags = binary.LittleEndian.Uint32(v)
-		}
-	}
-
-	stat := fileInfo.Sys().(*syscall.Stat_t)
-
-	flags = (flags & ^mask) | (stat.Flags & mask)
-
-	if flags != stat.Flags {
-		pPath, _ := syscall.BytePtrFromString(fullPath)
-		if _, _, errno := syscall.Syscall(syscall.SYS_LCHFLAGS,
-			uintptr(unsafe.Pointer(pPath)),
-			uintptr(flags), 0); errno != 0 {
-			return os.NewSyscallError("lchflags", errno)
-		}
-	}
-	return nil
-}
Author	SHA1	Message	Date
John K. Luebs	498a948b34	gofmt some files	2023-10-04 01:23:03 -05:00
John K. Luebs	2679751896	Fix handling of hardlinks and special files Also, don't attempt xattr ops on special files on the BSD likes. TODO: Should have a way to allow restore without special files, otherwise very cumbersome for a regular user.	2023-10-04 01:12:13 -05:00
John K. Luebs	66a938af67	Support for hardlinks to symlinks This is a specialized use-case, but it is indeed possible to do this and can be used if xattrs are attached to the symlink.	2023-10-04 01:12:11 -05:00
John K. Luebs	015d2200da	Fix handling of xattrs with symlinks Fix Linux, Darwin, and other BSD (untested) to allow proper handling of xattrs with symlinks. On Linux we cannot use the f* syscalls for symlinks because symlinks cannot be opened. File flags must be handled differently on darwin and other BSD due to the lack of the LCHFLAGS syscall on darwin, and the fact that it is emulated in libc. However, we do have O_SYMLINK on darwin.	2023-10-04 01:11:20 -05:00
John K. Luebs	96e7c93a2c	Support backup and restore of special files on POSIX style systems Special files are device nodes and named pipes. The necessity of the former is clear, the latter is debatable. In order to preserve backward compatibility, the device number is encoded in the StartChunk/StartOffset fields of the entry.	2023-10-03 16:26:19 -05:00
John K. Luebs	f06779659e	Don't overwrite symlinks if file already exists	2023-10-03 15:08:47 -05:00
John K. Luebs	16885eaa61	Support backup and restore of hardlinks This tracks inode/device from the stat info and creates backward compatible snapshots that allow preserving hardlinks. Backwards compatibility is preserved by saving a virtual inode number index in the Link field of the file entry. Since this field was previously only used for symlinks, this won't break old versions. Additionally, the entry data is cloned so restoration with an old version works. Current limitations are primarility with restore. They include: - no command line option to prevent hard link restore - if a file has the immutable or append only flag it will be set before hardlinks are restored, so hardlinking will fail. - if a partial restore includes a hardlink but not the parent directories the hardlink will fail. These will be solved by grouping restore of hardlinks together with file, prior to applying final metadata. - if a file is changed and is being rewritten by a restore hardlinks are not preserved.	2023-10-03 12:21:46 -05:00
John K. Luebs	bf2565b5c3	Initial implementation of file/inode flags (Linux, BSD, darwin) Basic support for BSD and Darwin style chflags (stat flags). Applies these flags at the end of file restore. Supports linux style ioctl_iflags(2) in a 2 step process. Flags that need to be applied prior to writes such as compress and especially no-COW are applied immediately upon file open. The flags format is backwards compatible. An attribute starting with a null byte is used to store flags in the entry attributes table. With an old version of duplicacy the restore of this attribute should silently fail (effectively be ignored). Fixes xattr restore to use O_NOFOLLOW so attributes are applied to symlink. TODO: Tests, possible option to switch off mutable/append prior to restore of existing file similar to rsync. Does not apply attributes or flags to the top most directory.	2023-10-03 12:15:54 -05:00
John K. Luebs	c07eef5063	Increase b2 client max file listing count to 10000 Considerable speed improvement with listing large storage.	2023-10-02 12:46:02 -05:00
John K. Luebs	2fdedcb9dd	Fix exclude_by_attribute feature on POSIX The exclude by attribute function is broken on non-Darwin POSIX: linux and freebsd. This is because those xattrs must be prefixed by a legal namespace. The old xattr library implicitly appended the user namespace to the xattr, but the current official go pkg does not (which is just as well). Also fix the test to remove the discordant old xattr dependency and provide test cases for both darwin and non-darwin POSIX.	2023-10-02 12:41:50 -05:00
John K. Luebs	7bdd1cabd3	Use S3 ListObjectsV2 for listing files ListObjects has been deprecated since 2016 and ListObjectsV2 with use of explicit pagination tokens is more performant for large listings as well. This also mitigates an issue with iDrive E2 where the StartAfter/Marker is included in the output, leading to duplicate entries. Right now this causes an exhaustive prune to delete chunks erroneously flagged as duplicate, destroying the storage.	2023-10-02 12:41:50 -05:00