Continue to check other snapshots when one snapshot has missing chunks

Record deleted snapshots in the fossil collection and if any deleted snapshot still exist nude the fossil collection
Print git commit number
2026-01-02 11:44:45 -06:00 · 2018-08-06 21:20:04 -04:00 · 2018-08-04 22:59:25 -04:00 · 2018-08-03 23:45:23 -04:00 · 2018-08-03 11:32:24 -04:00 · 2018-07-26 21:24:33 -04:00
30 changed files with 2239 additions and 438 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -0,0 +1,5 @@
+Please submit an issue for bug reports or feature requests.  If you have any questions please post them on https://forum.duplicacy.com.
+
+When you're reporting a bug, please specify the OS, version, command line arguments, or any info that you think is helpful for the diagnosis.  If Duplicacy reports an error, please post the program output here.
+
+Note that this repository hosts the CLI version of Duplicacy only.  If you're reporting anything related to the GUI version, please visit https://forum.duplicacy.com.
--- a/README.md
+++ b/README.md
@@ -1,129 +1,92 @@
 # Duplicacy: A lock-free deduplication cloud backup tool

-Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).  It is the only cloud backup tool that allows multiple computers to back up to the same storage simultaneously without using any locks (thus readily amenable to various cloud storage services).
+Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).  

-The repository hosts source code, design documents, and binary releases of the command line version.  There is also a Duplicacy GUI frontend built for Windows and Mac OS X available from https://duplicacy.com.
+This repository hosts source code, design documents, and binary releases of the command line version of Duplicacy.  There is also a Duplicacy GUI frontend built for Windows and Mac OS X available from https://duplicacy.com.

 There is a special edition of Duplicacy developed for VMware vSphere (ESXi) named [Vertical Backup](https://www.verticalbackup.com) that can back up virtual machine files on ESXi to local drives, network or cloud storages.

 ## Features

-Duplicacy currently supports major cloud storage providers (Amazon S3, Google Cloud Storage, Microsoft Azure, Dropbox, Backblaze B2, Google Drive, Microsoft OneDrive, and Hubic) and offers all essential features of a modern backup tool:
+There are 3 core advantages of Duplicacy over any other open-source or commercial backup tools:

-* Incremental backup: only back up what has been changed
-* Full snapshot: although each backup is incremental, it must behave like a full snapshot for easy restore and deletion
-* Deduplication: identical files must be stored as one copy (file-level deduplication), and identical parts from different files must be stored as one copy (block-level deduplication)
-* Encryption: encrypt not only file contents but also file paths, sizes, times, etc.
-* Deletion: every backup can be deleted independently without affecting others
-* Concurrent access: multiple clients can back up to the same storage at the same time
-* Snapshot migration: all or selected snapshots can be migrated from one storage to another
+* Duplicacy is the *only* cloud backup tool that allows multiple computers to back up to the same cloud storage, taking advantage of cross-computer deduplication whenever possible, without direct communication among them.  This feature literally turns any cloud storage server supporting only a basic set of file operations into a sophisticated deduplication-aware server.  

-The key idea of **[Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication)** can be summarized as follows:
+* Unlike other chunk-based backup tools where chunks are grouped into pack files and a chunk database is used to track which chunks are stored inside each pack file, Duplicacy takes a database-less approach where every chunk is saved independently using its hash as the file name to facilitate quick lookups.  The lack of a centralized chunk database not only makes the implementation less error-prone, but also produces a highly maintainable piece of software with plenty of room for development of new features and usability enhancements.

-* Use variable-size chunking algorithm to split files into chunks
-* Store each chunk in the storage using a file name derived from its hash, and rely on the file system API to manage chunks without using a centralized indexing database
-* Apply a *two-step fossil collection* algorithm to remove chunks that become unreferenced after a backup is deleted
+* Duplicacy is fast.  While the performance wasn't the top-priority design goal, Duplicacy has been shown to outperform other backup tools by a considerable margin, as indicated by the following results obtained from a [benchmarking experiment](https://github.com/gilbertchen/benchmarking) backing up the [Linux code base](https://github.com/torvalds/linux) using Duplicacy and 3 other open-source backup tools.
+
+[![Comparison of Duplicacy, restic, Attic, duplicity](https://github.com/gilbertchen/duplicacy/blob/master/images/duplicacy_benchmark_speed.png "Comparison of Duplicacy, restic, Attic, duplicity")](https://github.com/gilbertchen/benchmarking)

 ## Getting Started

 * [A brief introduction](https://github.com/gilbertchen/duplicacy/wiki/Quick-Start)
 * [Command references](https://github.com/gilbertchen/duplicacy/wiki)
+* [Building from source](https://github.com/gilbertchen/duplicacy/wiki/Installation)

 ## Storages

-With Duplicacy, you can back up files to local or networked drives, SFTP servers, or many cloud storage providers.  The following table compares the costs of all cloud storages supported by Duplicacy.
+Duplicacy currently provides the following storage backends:

-| Type         |   Storage (monthly)    |   Upload           |    Download    |    API Charge   |
-|:------------:|:-------------:|:------------------:|:--------------:|:-----------:|
-| Amazon S3    | $0.023/GB | free | $0.090/GB | [yes](https://aws.amazon.com/s3/pricing/) |
-| Wasabi       | $3.99 first 1TB <br> $0.0039/GB additional | free | $0.04/GB | no |
-| DigitalOcean Spaces| $5 first 250GB <br> $0.020/GB additional | free | first 1TB free <br> $0.01/GB additional| no |
-| Backblaze B2 | 10GB free <br> $0.005/GB | free | 1GB free/day <br> $0.02/GB | [yes](https://www.backblaze.com/b2/b2-transactions-price.html) |
-| Google Cloud Storage| $0.026/GB | free |$ 0.12/GB | [yes](https://cloud.google.com/storage/pricing) |
-| Google Drive | 15GB free <br> $1.99/100GB <br> $9.99/TB | free | free | no |
-| Microsoft Azure | $0.0184/GB | free | free | [yes](https://azure.microsoft.com/en-us/pricing/details/storage/blobs/) |
-| Microsoft OneDrive | 5GB free <br> $1.99/50GB <br> $5.83/TB | free | free | no |
-| Dropbox | 2GB free <br> $8.25/TB | free | free | no |
-| Hubic | 25GB free <br> €1/100GB <br> €5/10TB | free | free | no |
+* Local disk
+* SFTP
+* Dropbox
+* Amazon S3
+* Wasabi
+* DigitalOcean Spaces
+* Google Cloud Storage
+* Microsoft Azure
+* Backblaze B2
+* Google Drive
+* Microsoft OneDrive
+* Hubic
+* OpenStack Swift
+* WebDAV (under beta testing)
+* pcloud (via WebDAV)
+* Box.com (via WebDAV)

 Please consult the [wiki page](https://github.com/gilbertchen/duplicacy/wiki/Storage-Backends) on how to set up Duplicacy to work with each cloud storage.

-It should be noted that their performances vary a lot.  A [performance comparison](https://github.com/gilbertchen/cloud-storage-comparison) of these storages measured the running times (in seconds) of backing up and restoring the [Linux code base](https://github.com/torvalds/linux) as follows:
+For reference, the following chart shows the running times (in seconds) of backing up the [Linux code base](https://github.com/torvalds/linux) to each of those supported storages:

-| Storage              | initial backup | 2nd | 3rd | initial restore | 2nd | 3rd |
-|:--------------------:|:------:|:----:|:-----:|:----:|:-----:|:----:|
-| SFTP                 |  31.5  | 6.6  | 20.6  | 22.5  | 7.8  | 18.4 |
-| Amazon S3            |  41.1  | 5.9  | 21.9  | 27.7  | 7.6  | 23.5 |
-| Wasabi               |  38.7  | 5.7  | 31.7  | 25.7  | 6.5  | 23.2 | 
-| DigitalOcean Spaces  |  51.6  | 7.1  | 31.7  | 29.3  | 6.4  | 27.6 | 
-| Backblaze B2         |  106.7 | 24.0 | 88.2  | 67.9  | 14.4 | 39.1 | 
-| Google Cloud Storage |  76.9  | 11.9 | 33.1  | 39.5  | 9.9  | 26.2 | 
-| Google Drive         |  139.3 | 14.7 | 45.2  | 129.4 | 17.8 | 54.4 |
-| Microsoft Azure      |  35.0  | 5.4  | 20.4  | 30.7  | 7.1  | 21.5 | 
-| Microsoft OneDrive   |  250.0 | 31.6 | 80.2  | 333.4 | 26.2 | 82.0 | 
-| Dropbox              |  267.2 | 35.8 | 113.7 | 164.0 | 31.6 | 80.3 |

-For more details please visit https://github.com/gilbertchen/cloud-storage-comparison.
+[![Comparison of Cloud Storages](https://github.com/gilbertchen/duplicacy/blob/master/images/duplicacy_benchmark_cloud.png "Comparison of Cloud Storages")](https://github.com/gilbertchen/cloud-storage-comparison)

-## Feature Comparison with Other Backup Tools
+
+For complete benchmark results please visit https://github.com/gilbertchen/cloud-storage-comparison.
+
+## Comparison with Other Backup Tools

 [duplicity](http://duplicity.nongnu.org) works by applying the rsync algorithm (or more specific, the [librsync](https://github.com/librsync/librsync) library)
 to find the differences from previous backups and only then uploading the differences.  It is the only existing backup tool with extensive cloud support -- the [long list](http://duplicity.nongnu.org/duplicity.1.html#sect7) of storage backends covers almost every cloud provider one can think of.  However, duplicity's biggest flaw lies in its incremental model -- a chain of dependent backups starts with a full backup followed by a number of incremental ones, and ends when another full backup is uploaded.  Deleting one backup will render useless all the subsequent backups on the same chain.  Periodic full backups are required, in order to make previous backups disposable.

 [bup](https://github.com/bup/bup) also uses librsync to split files into chunks but save chunks in the git packfile format.  It doesn't support any cloud storage, or deletion of old backups.

-[Obnam](http://obnam.org) got the incremental backup model right in the sense that every incremental backup is actually a full snapshot.  Although Obnam also splits files into chunks, it does not adopt either the rsync algorithm or the variable-size chunking algorithm.  As a result, deletions or insertions of a few bytes will foil the
-[deduplication](http://obnam.org/faq/dedup).
-Deletion of old backups is possible, but no cloud storages are supported.
-Multiple clients can back up to the same storage, but only sequential access is granted by the [locking on-disk data structures](http://obnam.org/locking/).
-It is unclear if the lack of cloud backends is due to difficulties in porting the locking data structures to cloud storage APIs.
+[Duplicati](https://duplicati.com) is one of the first backup tools that adopt the chunk-based approach to split files into chunks which are then uploaded to the storage.  The chunk-based approach got the incremental backup model right in the sense that every incremental backup is actually a full snapshot.  As Duplicati splits files into fixed-size chunks,  deletions or insertions of a few bytes will foil the deduplication.  Cloud support is extensive, but multiple clients can't back up to the same storage location.

-[Attic](https://attic-backup.org) has been acclaimed by some as the [Holy Grail of backups](https://www.stavros.io/posts/holy-grail-backups).  It follows the same incremental backup model as Obnam, but embraces the variable-size chunk algorithm for better performance and better deduplication.  Deletions of old backup is also supported.  However, no cloud backends are implemented, as in Obnam.  Although concurrent backups from multiple clients to the same storage is in theory possible by the use of locking, it is
+[Attic](https://attic-backup.org) has been acclaimed by some as the [Holy Grail of backups](https://www.stavros.io/posts/holy-grail-backups).  It follows the same incremental backup model like Duplicati, but embraces the variable-size chunk algorithm for better performance and higher deduplication efficiency (not susceptible to byte insertion and deletion any more).  Deletions of old backup is also supported.  However, no cloud backends are implemented.  Although concurrent backups from multiple clients to the same storage is in theory possible by the use of locking, it is
 [not recommended](http://librelist.com/browser//attic/2014/11/11/backing-up-multiple-servers-into-a-single-repository/#e96345aa5a3469a87786675d65da492b) by the developer due to chunk indices being kept in a local cache.
-Concurrent access is not only a convenience; it is a necessity for better deduplication.  For instance, if multiple machines with the same OS installed can back up their entire drives to the same storage, only one copy of the system files needs to be stored, greatly reducing the storage space regardless of the number of machines.  Attic still adopts the traditional approach of using a centralized indexing database to manage chunks, and relies heavily on caching to improve performance.  The presence of exclusive locking makes it hard to be adapted for cloud storage APIs and reduces the level of deduplication.
+Concurrent access is not only a convenience; it is a necessity for better deduplication.  For instance, if multiple machines with the same OS installed can back up their entire drives to the same storage, only one copy of the system files needs to be stored, greatly reducing the storage space regardless of the number of machines.  Attic still adopts the traditional approach of using a centralized indexing database to manage chunks, and relies heavily on caching to improve performance.  The presence of exclusive locking makes it hard to be extended to cloud storages.

-[restic](https://restic.github.io) is a more recent addition.  It is worth mentioning here because, like Duplicacy, it is written in Go.  It uses a format similar to the git packfile format.  Multiple clients backing up to the same storage are still guarded by
-[locks](https://github.com/restic/restic/blob/master/doc/Design.md#locks).  A prune operation will therefore completely block all other clients connected to the storage from doing their regular backups.  Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is.  A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss.  This is the exact problem that we avoided by taking the lock-free approach.
+[restic](https://restic.github.io) is a more recent addition. It uses a format similar to the git packfile format.  Multiple clients backing up to the same storage are still guarded by
+[locks](https://github.com/restic/restic/blob/master/doc/Design.md#locks), and because a chunk database is used, deduplication isn't real-time (different clients sharing the same files will upload different copies of the same chunks).  A prune operation will completely block all other clients connected to the storage from doing their regular backups.  Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is.  A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss.  This is the exact problem that we avoided by taking the lock-free approach.


 The following table compares the feature lists of all these backup tools:


-| Feature/Tool       | duplicity | bup | Obnam             | Attic           | restic            | **Duplicacy** |
+| Feature/Tool       | duplicity | bup | Duplicati         | Attic           | restic            | **Duplicacy** |
 |:------------------:|:---------:|:---:|:-----------------:|:---------------:|:-----------------:|:-------------:|
 | Incremental Backup | Yes       | Yes | Yes               | Yes             | Yes               | **Yes**       |
 | Full Snapshot      | No        | Yes | Yes               | Yes             | Yes               | **Yes**       |
+| Compression        | Yes       | Yes | Yes               | Yes             | No                | **Yes**       |
 | Deduplication      | Weak      | Yes | Weak              | Yes             | Yes               | **Yes**       |
 | Encryption         | Yes       | Yes | Yes               | Yes             | Yes               | **Yes**       |
 | Deletion           | No        | No  | Yes               | Yes             | No                | **Yes**       |
-| Concurrent Access  | No        | No  | Exclusive locking | Not recommended | Exclusive locking | **Lock-free** |
-| Cloud Support      | Extensive | No  | No                | No              | S3, B2, OpenStack | **S3, GCS, Azure, Dropbox, Backblaze B2, Google Drive, OneDrive, and Hubic**|
+| Concurrent Access  | No        | No  | No                | Not recommended | Exclusive locking | **Lock-free** |
+| Cloud Support      | Extensive | No  | Extensive         | No              | Limited           | **Extensive** |
 | Snapshot Migration | No        | No  | No                | No              | No                | **Yes**       |

-
-## Performance Comparison with Other Backup Tools
-
-Duplicacy is not only more feature-rich but also faster than other backup tools.  The following table lists the running times in seconds of backing up the [Linux code base](https://github.com/torvalds/linux) using Duplicacy and 3 other tools.  Clearly Duplicacy is the fastest by a significant margin.
-
-
-|                    |   Duplicacy  |   restic   |   Attic    |  duplicity  |
-|:------------------:|:----------------:|:----------:|:----------:|:-----------:|
-| Initial backup | 13.7 | 20.7 | 26.9 | 44.2 |
-| 2nd backup | 4.8  |  8.0 | 15.4 | 19.5 |
-| 3rd backup | 6.9  | 11.9 | 19.6 | 29.8 |
-| 4th backup | 3.3  | 7.0  | 13.7 | 18.6 |
-| 5th backup | 9.9  | 11.4 | 19.9 | 28.0 |
-| 6th backup | 3.8  | 8.0  | 16.8 | 22.0 |
-| 7th backup | 5.1  | 7.8  | 14.3 | 21.6 |
-| 8th backup | 9.5  | 13.5 | 18.3 | 35.0 |
-| 9th backup | 4.3  | 9.0  | 15.7 | 24.9 |
-| 10th backup | 7.9 | 20.2 | 32.2 | 35.0 |
-| 11th backup | 4.6 | 9.1  | 16.8 | 28.1 |
-| 12th backup | 7.4 | 12.0 | 21.7 | 37.4 |
-
-
-For more details and other speed comparison results, please visit https://github.com/gilbertchen/benchmarking.  There you can also find test scripts that you can use to run your own experiments.
-
 ## License

 * Free for personal use or commercial trial
--- a/duplicacy/duplicacy_main.go
+++ b/duplicacy/duplicacy_main.go
@@ -32,6 +32,7 @@ const (
 )

 var ScriptEnabled bool
+var GitCommit = "unofficial"

 func getRepositoryPreference(context *cli.Context, storageName string) (repository string,
 	preference *duplicacy.Preference) {
@@ -71,6 +72,10 @@ func getRepositoryPreference(context *cli.Context, storageName string) (reposito
 	}

 	if storageName == "" {
+		if duplicacy.Preferences[0].RepositoryPath != "" {
+			repository = duplicacy.Preferences[0].RepositoryPath
+			duplicacy.LOG_INFO("REPOSITORY_SET", "Repository set to %s", repository)
+		}
 		return repository, &duplicacy.Preferences[0]
 	}

@@ -80,6 +85,12 @@ func getRepositoryPreference(context *cli.Context, storageName string) (reposito
 		duplicacy.LOG_ERROR("STORAGE_NONE", "No storage named '%s' is found", storageName)
 		return "", nil
 	}
+
+	if preference.RepositoryPath != "" {
+		repository = preference.RepositoryPath
+		duplicacy.LOG_INFO("REPOSITORY_SET", "Repository set to %s", repository)
+	}
+
 	return repository, preference
 }

@@ -293,9 +304,14 @@ func configRepository(context *cli.Context, init bool) {
 		}
 	}

+	repositoryPath := ""
+	if context.String("repository") != "" {
+		repositoryPath = context.String("repository")
+	}
 	preference := duplicacy.Preference{
 		Name:       storageName,
 		SnapshotID: snapshotID,
+		RepositoryPath: repositoryPath,
 		StorageURL: storageURL,
 		Encrypted:  context.Bool("encrypt"),
 	}
@@ -427,8 +443,12 @@ func configRepository(context *cli.Context, init bool) {

 	duplicacy.SavePreferences()

+	if repositoryPath == "" {
+		repositoryPath = repository
+	}
+
 	duplicacy.LOG_INFO("REPOSITORY_INIT", "%s will be backed up to %s with id %s",
-		repository, preference.StorageURL, preference.SnapshotID)
+		repositoryPath, preference.StorageURL, preference.SnapshotID)
 }

 type TriBool struct {
@@ -514,6 +534,8 @@ func setPreference(context *cli.Context) {
 		newPreference.DoNotSavePassword = triBool.IsTrue()
 	}
 	
+	newPreference.NobackupFile = context.String("nobackup-file")
+
 	key := context.String("key")
 	value := context.String("value")

@@ -689,16 +711,18 @@ func backupRepository(context *cli.Context) {
 	showStatistics := context.Bool("stats")

 	enableVSS := context.Bool("vss")
+	vssTimeout := context.Int("vss-timeout")

 	dryRun := context.Bool("dry-run")
 	uploadRateLimit := context.Int("limit-rate")
+	enumOnly := context.Bool("enum-only")
 	storage.SetRateLimits(0, uploadRateLimit)
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
 	backupManager.SetDryRun(dryRun)
-	backupManager.Backup(repository, quickMode, threads, context.String("t"), showStatistics, enableVSS)
+	backupManager.Backup(repository, quickMode, threads, context.String("t"), showStatistics, enableVSS, vssTimeout, enumOnly)

 	runScript(context, preference.Name, "post")
 }
@@ -782,7 +806,7 @@ func restoreRepository(context *cli.Context) {
 	duplicacy.LOG_DEBUG("REGEX_DEBUG", "There are %d compiled regular expressions stored", len(duplicacy.RegexMap))

 	storage.SetRateLimits(context.Int("limit-rate"), 0)
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
@@ -822,7 +846,7 @@ func listSnapshots(context *cli.Context) {
 	tag := context.String("t")
 	revisions := getRevisions(context)

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	id := preference.SnapshotID
@@ -870,7 +894,7 @@ func checkSnapshots(context *cli.Context) {
 	tag := context.String("t")
 	revisions := getRevisions(context)

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	id := preference.SnapshotID
@@ -925,7 +949,7 @@ func printFile(context *cli.Context) {
 		snapshotID = context.String("id")
 	}

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
@@ -981,11 +1005,11 @@ func diff(context *cli.Context) {
 	}

 	compareByHash := context.Bool("hash")
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
-	backupManager.SnapshotManager.Diff(repository, snapshotID, revisions, path, compareByHash)
+	backupManager.SnapshotManager.Diff(repository, snapshotID, revisions, path, compareByHash, preference.NobackupFile)

 	runScript(context, preference.Name, "post")
 }
@@ -1024,7 +1048,7 @@ func showHistory(context *cli.Context) {

 	revisions := getRevisions(context)
 	showLocalHash := context.Bool("hash")
-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
@@ -1043,12 +1067,17 @@ func pruneSnapshots(context *cli.Context) {
 		os.Exit(ArgumentExitCode)
 	}

+	threads := context.Int("threads")
+	if threads < 1 {
+		threads = 1
+	}
+
 	repository, preference := getRepositoryPreference(context, "")

 	runScript(context, preference.Name, "pre")

 	duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-	storage := duplicacy.CreateStorage(*preference, false, 1)
+	storage := duplicacy.CreateStorage(*preference, false, threads)
 	if storage == nil {
 		return
 	}
@@ -1082,12 +1111,12 @@ func pruneSnapshots(context *cli.Context) {
 		os.Exit(ArgumentExitCode)
 	}

-	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
+	backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password, preference.NobackupFile)
 	duplicacy.SavePassword(*preference, "password", password)

 	backupManager.SetupSnapshotCache(preference.Name)
 	backupManager.SnapshotManager.PruneSnapshots(selfID, snapshotID, revisions, tags, retentions,
-		exhaustive, exclusive, ignoredIDs, dryRun, deleteOnly, collectOnly)
+		exhaustive, exclusive, ignoredIDs, dryRun, deleteOnly, collectOnly, threads)

 	runScript(context, preference.Name, "post")
 }
@@ -1122,7 +1151,7 @@ func copySnapshots(context *cli.Context) {
 		sourcePassword = duplicacy.GetPassword(*source, "password", "Enter source storage password:", false, false)
 	}

-	sourceManager := duplicacy.CreateBackupManager(source.SnapshotID, sourceStorage, repository, sourcePassword)
+	sourceManager := duplicacy.CreateBackupManager(source.SnapshotID, sourceStorage, repository, sourcePassword, source.NobackupFile)
 	sourceManager.SetupSnapshotCache(source.Name)
 	duplicacy.SavePassword(*source, "password", sourcePassword)

@@ -1155,7 +1184,7 @@ func copySnapshots(context *cli.Context) {
 	destinationStorage.SetRateLimits(0, context.Int("upload-limit-rate"))

 	destinationManager := duplicacy.CreateBackupManager(destination.SnapshotID, destinationStorage, repository,
-		destinationPassword)
+		destinationPassword, destination.NobackupFile)
 	duplicacy.SavePassword(*destination, "password", destinationPassword)
 	destinationManager.SetupSnapshotCache(destination.Name)

@@ -1239,6 +1268,50 @@ func infoStorage(context *cli.Context) {

 }

+func benchmark(context *cli.Context) {
+	setGlobalOptions(context)
+	defer duplicacy.CatchLogException()
+
+	fileSize := context.Int("file-size")
+	if fileSize == 0 {
+		fileSize = 256
+	}
+
+	chunkCount := context.Int("chunk-count")
+	if chunkCount == 0 {
+		chunkCount = 64
+	}
+
+	chunkSize := context.Int("chunk-size")
+	if chunkSize == 0 {
+		chunkSize = 4
+	}
+
+	downloadThreads := context.Int("download-threads")
+	if downloadThreads < 1 {
+		downloadThreads = 1
+	}
+
+	uploadThreads := context.Int("upload-threads")
+	if uploadThreads < 1 {
+		uploadThreads = 1
+	}
+
+	threads := downloadThreads
+	if (threads < uploadThreads) {
+		threads = uploadThreads
+	}
+
+	repository, preference := getRepositoryPreference(context, context.String("storage"))
+
+	duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
+	storage := duplicacy.CreateStorage(*preference, false, threads)
+	if storage == nil {
+		return
+	}
+	duplicacy.Benchmark(repository, storage, int64(fileSize) * 1000000, chunkSize * 1024 * 1024, chunkCount, uploadThreads, downloadThreads)
+}
+
 func main() {

 	duplicacy.SetLoggingLevel(duplicacy.INFO)
@@ -1284,6 +1357,11 @@ func main() {
 					Usage:    "assign a name to the storage",
 					Argument: "<name>",
 				},
+				cli.StringFlag{
+					Name:     "repository",
+					Usage:    "initialize a new repository at the specified path rather than the current working directory",
+					Argument: "<path>",
+				},
 			},
 			Usage:     "Initialize the storage if necessary and the current directory as the repository",
 			ArgsUsage: "<snapshot id> <storage url>",
@@ -1323,13 +1401,23 @@ func main() {
 				},
 				cli.BoolFlag{
 					Name:  "vss",
-					Usage: "enable the Volume Shadow Copy service (Windows only)",
+					Usage: "enable the Volume Shadow Copy service (Windows and macOS using APFS only)",
+				},
+				cli.IntFlag{
+					Name:     "vss-timeout",
+					Value:    0,
+					Usage:    "the timeout in seconds to wait for the Volume Shadow Copy operation to complete",
+					Argument: "<timeout>",
 				},
 				cli.StringFlag{
 					Name:     "storage",
 					Usage:    "backup to the specified storage instead of the default one",
 					Argument: "<storage name>",
 				},
+				cli.BoolFlag{
+					Name:  "enum-only",
+					Usage: "enumerate the repository recursively and then exit",
+				},
 			},
 			Usage:     "Save a snapshot of the repository to the storage",
 			ArgsUsage: " ",
@@ -1620,6 +1708,12 @@ func main() {
 					Usage:    "prune snapshots from the specified storage",
 					Argument: "<storage name>",
 				},
+				cli.IntFlag{
+					Name:     "threads",
+					Value:    1,
+					Usage:    "number of threads used to prune unreferenced chunks",
+					Argument: "<n>",
+				},
 			},
 			Usage:     "Prune snapshots by revision, tag, or retention policy",
 			ArgsUsage: " ",
@@ -1682,6 +1776,11 @@ func main() {
 					Name:     "bit-identical",
 					Usage:    "(when using -copy) make the new storage bit-identical to also allow rsync etc.",
 				},
+				cli.StringFlag{
+					Name:     "repository",
+					Usage:    "specify the path of the repository (instead of the current working directory)",
+					Argument: "<path>",
+				},
 			},
 			Usage:     "Add an additional storage to be used for the existing repository",
 			ArgsUsage: "<storage name> <snapshot id> <storage url>",
@@ -1715,6 +1814,12 @@ func main() {
 					Value: &TriBool{},
 					Arg:   "true",
 				},
+				cli.StringFlag{
+					Name:  "nobackup-file",
+					Usage: "Directories containing a file with this name will not be backed up",
+					Argument: "<file name>",
+					Value:   "",
+				},
 				cli.StringFlag{
 					Name:  "key",
 					Usage: "add a key/password whose value is supplied by the -value option",
@@ -1771,7 +1876,7 @@ func main() {
 				cli.IntFlag{
 					Name:     "threads",
 					Value:    1,
-					Usage:    "number of downloading threads",
+					Usage:    "number of uploading threads",
 					Argument: "<n>",
 				},
 			},
@@ -1806,6 +1911,45 @@ func main() {
 			ArgsUsage: "<storage url>",
 			Action:    infoStorage,
 		},
+
+		{
+			Name: "benchmark",
+			Flags: []cli.Flag{
+				cli.IntFlag{
+					Name:     "file-size",
+					Usage:    "the size of the local file to write to and read from (in MB, default to 256)",
+					Argument: "<size>",
+				},
+				cli.IntFlag{
+					Name:     "chunk-count",
+					Usage:    "the number of chunks to upload and download (default to 64)",
+					Argument: "<count>",
+				},
+				cli.IntFlag{
+					Name:     "chunk-size",
+					Usage:    "the size of chunks to upload and download (in MB, default to 4)",
+					Argument: "<size>",
+				},
+				cli.IntFlag{
+					Name:     "upload-threads",
+					Usage:    "the number of upload threads (default to 1)",
+					Argument: "<n>",
+				},
+				cli.IntFlag{
+					Name:     "download-threads",
+					Usage:    "the number of download threads (default to 1)",
+					Argument: "<n>",
+				},
+				cli.StringFlag{
+					Name:     "storage",
+					Usage:    "run the download/upload test agaist the specified storage",
+					Argument: "<storage name>",
+				},
+			},
+			Usage:     "Run a set of benchmarks to test download and upload speeds",
+			ArgsUsage: " ",
+			Action:    benchmark,
+		},
 	}

 	app.Flags = []cli.Flag{
@@ -1839,13 +1983,17 @@ func main() {
 			Usage:    "enable the profiling tool and listen on the specified address:port",
 			Argument: "<address:port>",
 		},
-}
+		cli.StringFlag{
+			Name:	"comment",
+			Usage:	"add a comment to identify the process",
+		},
+	}

 	app.HideVersion = true
 	app.Name = "duplicacy"
 	app.HelpName = "duplicacy"
 	app.Usage = "A new generation cloud backup tool based on lock-free deduplication"
-	app.Version = "2.1.0"
+	app.Version = "2.1.1" + " (" + GitCommit + ")"

 	// If the program is interrupted, call the RunAtError function.
 	c := make(chan os.Signal, 1)
--- a/images/duplicacy_benchmark_cloud.png
+++ b/images/duplicacy_benchmark_cloud.png
--- a/images/duplicacy_benchmark_speed.png
+++ b/images/duplicacy_benchmark_speed.png
--- a/src/duplicacy_acdstorage.go
+++ b/src/duplicacy_acdstorage.go
@@ -55,7 +55,7 @@ func CreateACDStorage(tokenFile string, storagePath string, threads int) (storag
 				return nil, err
 			}
 		} else if !isDir {
-			return nil, fmt.Errorf("%s/%s is not a directory", storagePath+"/"+dir)
+			return nil, fmt.Errorf("%s is not a directory", storagePath+"/"+dir)
 		}
 		storage.idCache[dir] = dirID
 	}
--- a/src/duplicacy_backupmanager.go
+++ b/src/duplicacy_backupmanager.go
@@ -33,6 +33,8 @@ type BackupManager struct {
 	snapshotCache   *FileStorage     // for copies of chunks needed by snapshots

 	config *Config // contains a number of options
+	
+	nobackupFile string // don't backup directory when this file name is found
 }

 func (manager *BackupManager) SetDryRun(dryRun bool) {
@@ -42,7 +44,7 @@ func (manager *BackupManager) SetDryRun(dryRun bool) {
 // CreateBackupManager creates a backup manager using the specified 'storage'.  'snapshotID' is a unique id to
 // identify snapshots created for this repository.  'top' is the top directory of the repository.  'password' is the
 // master key which can be nil if encryption is not enabled.
-func CreateBackupManager(snapshotID string, storage Storage, top string, password string) *BackupManager {
+func CreateBackupManager(snapshotID string, storage Storage, top string, password string, nobackupFile string) *BackupManager {

 	config, _, err := DownloadConfig(storage, password)
 	if err != nil {
@@ -63,6 +65,8 @@ func CreateBackupManager(snapshotID string, storage Storage, top string, passwor
 		SnapshotManager: snapshotManager,

 		config: config,
+		
+		nobackupFile: nobackupFile,
 	}

 	if IsDebugging() {
@@ -159,7 +163,7 @@ func setEntryContent(entries []*Entry, chunkLengths []int, offset int) {
 // unmodified files with last backup).  Otherwise (or if this is the first backup), the entire repository will
 // be scanned to create the snapshot.  'tag' is the tag assigned to the new snapshot.
 func (manager *BackupManager) Backup(top string, quickMode bool, threads int, tag string,
-	showStatistics bool, shadowCopy bool) bool {
+	showStatistics bool, shadowCopy bool, shadowCopyTimeout int, enumOnly bool) bool {

 	var err error
 	top, err = filepath.Abs(top)
@@ -180,16 +184,20 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 		LOG_INFO("BACKUP_START", "Last backup at revision %d found", remoteSnapshot.Revision)
 	}

-	shadowTop := CreateShadowCopy(top, shadowCopy)
+	shadowTop := CreateShadowCopy(top, shadowCopy, shadowCopyTimeout)
 	defer DeleteShadowCopy()

 	LOG_INFO("BACKUP_INDEXING", "Indexing %s", top)
-	localSnapshot, skippedDirectories, skippedFiles, err := CreateSnapshotFromDirectory(manager.snapshotID, shadowTop)
+	localSnapshot, skippedDirectories, skippedFiles, err := CreateSnapshotFromDirectory(manager.snapshotID, shadowTop, manager.nobackupFile)
 	if err != nil {
 		LOG_ERROR("SNAPSHOT_LIST", "Failed to list the directory %s: %v", top, err)
 		return false
 	}

+	if enumOnly {
+		return true
+	}
+
 	// This cache contains all chunks referenced by last snasphot. Any other chunks will lead to a call to
 	// UploadChunk.
 	chunkCache := make(map[string]bool)
@@ -199,7 +207,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 	// A revision number of 0 means this is the initial backup
 	if remoteSnapshot.Revision > 0 {
 		// Add all chunks in the last snapshot to the cache
-		for _, chunkID := range manager.SnapshotManager.GetSnapshotChunks(remoteSnapshot) {
+		for _, chunkID := range manager.SnapshotManager.GetSnapshotChunks(remoteSnapshot, true) {
 			chunkCache[chunkID] = true
 		}
 	} else {
@@ -752,7 +760,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 	remoteSnapshot := manager.SnapshotManager.DownloadSnapshot(manager.snapshotID, revision)
 	manager.SnapshotManager.DownloadSnapshotContents(remoteSnapshot, patterns, true)

-	localSnapshot, _, _, err := CreateSnapshotFromDirectory(manager.snapshotID, top)
+	localSnapshot, _, _, err := CreateSnapshotFromDirectory(manager.snapshotID, top, manager.nobackupFile)
 	if err != nil {
 		LOG_ERROR("SNAPSHOT_LIST", "Failed to list the repository: %v", err)
 		return false
@@ -767,10 +775,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 		for _, file := range remoteSnapshot.Files {

 			if MatchPath(file.Path, patterns) {
-				LOG_TRACE("RESTORE_INCLUDE", "Include %s", file.Path)
 				includedFiles = append(includedFiles, file)
-			} else {
-				LOG_TRACE("RESTORE_EXCLUDE", "Exclude %s", file.Path)
 			}
 		}

@@ -837,7 +842,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 			stat, err := os.Stat(fullPath)

 			if err == nil && !stat.IsDir() {
-				LOG_ERROR("RESTORE_NOTDIR", "The path %s is not a directory: %v", fullPath, err)
+				LOG_ERROR("RESTORE_NOTDIR", "The path %s is not a directory", fullPath)
 				return false
 			}

--- a/src/duplicacy_backupmanager_test.go
+++ b/src/duplicacy_backupmanager_test.go
@@ -239,11 +239,11 @@ func TestBackupManager(t *testing.T) {
 	time.Sleep(time.Duration(delay) * time.Second)

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	backupManager := CreateBackupManager("host1", storage, testDir, password)
+	backupManager := CreateBackupManager("host1", storage, testDir, password, "")
 	backupManager.SetupSnapshotCache("default")

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false, 0, false)
 	time.Sleep(time.Duration(delay) * time.Second)
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
 	backupManager.Restore(testDir+"/repository2", threads, /*inPlace=*/false, /*quickMode=*/false, threads, /*overwrite=*/true,
@@ -267,7 +267,7 @@ func TestBackupManager(t *testing.T) {
 	modifyFile(testDir+"/repository1/dir1/file3", 0.3)

 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "second", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "second", false, false, 0, false)
 	time.Sleep(time.Duration(delay) * time.Second)
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
 	backupManager.Restore(testDir+"/repository2", 2, /*inPlace=*/true, /*quickMode=*/true, threads, /*overwrite=*/true,
@@ -287,7 +287,7 @@ func TestBackupManager(t *testing.T) {
 	os.Mkdir(testDir+"/repository1/dir2/dir3", 0700)
 	os.Mkdir(testDir+"/repository1/dir4", 0700)
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "third", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "third", false, false, 0, false)
 	time.Sleep(time.Duration(delay) * time.Second)

 	// Create some directories and files under repository2 that will be deleted during restore
@@ -343,16 +343,16 @@ func TestBackupManager(t *testing.T) {
 	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1, 2, 3} /*tag*/, "",
 		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
 	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, []int{1} /*tags*/, nil /*retentions*/, nil,
-		/*exhaustive*/ false /*exclusive=*/, false /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false)
+		/*exhaustive*/ false /*exclusive=*/, false /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false, 1)
 	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
 	if numberOfSnapshots != 2 {
 		t.Errorf("Expected 2 snapshots but got %d", numberOfSnapshots)
 	}
 	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3} /*tag*/, "",
 		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
-	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "fourth", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "fourth", false, false, 0, false)
 	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, nil /*tags*/, nil /*retentions*/, nil,
-		/*exhaustive*/ false /*exclusive=*/, true /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false)
+		/*exhaustive*/ false /*exclusive=*/, true /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false, 1)
 	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
 	if numberOfSnapshots != 3 {
 		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
--- a/src/duplicacy_benchmark.go
+++ b/src/duplicacy_benchmark.go
@@ -0,0 +1,235 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+package duplicacy
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"os"
+	"path/filepath"
+	"time"
+)
+
+func benchmarkSplit(reader *bytes.Reader, fileSize int64, chunkSize int, compression bool, encryption bool, annotation string) {
+
+	config := CreateConfig()
+	config.CompressionLevel = DEFAULT_COMPRESSION_LEVEL
+	config.AverageChunkSize = chunkSize
+	config.MaximumChunkSize = chunkSize * 4
+	config.MinimumChunkSize = chunkSize / 4
+	config.ChunkSeed = []byte("duplicacy")
+
+	config.HashKey = DEFAULT_KEY
+	config.IDKey = DEFAULT_KEY
+
+	maker := CreateChunkMaker(config, false)
+
+	startTime := float64(time.Now().UnixNano()) / 1e9
+	numberOfChunks := 0
+	reader.Seek(0, os.SEEK_SET)
+	maker.ForEachChunk(reader,
+		func(chunk *Chunk, final bool) {
+			if compression {
+				key := ""
+				if encryption {
+					key = "0123456789abcdef0123456789abcdef"
+				}
+				err := chunk.Encrypt([]byte(key), "")
+				if err != nil {
+					LOG_ERROR("BENCHMARK_ENCRYPT", "Failed to encrypt the chunk: %v", err)
+				}
+			}
+			config.PutChunk(chunk)
+			numberOfChunks++
+		},
+		func(size int64, hash string) (io.Reader, bool) {
+			return nil, false
+		})
+
+	runningTime := float64(time.Now().UnixNano())/1e9 - startTime
+	speed := int64(float64(fileSize) / runningTime)
+	LOG_INFO("BENCHMARK_SPLIT", "Split %s bytes into %d chunks %s in %.2fs: %s/s", PrettySize(fileSize), numberOfChunks, annotation,
+		runningTime, PrettySize(speed))
+
+	return
+}
+
+func benchmarkRun(threads int, chunkCount int, job func(threadIndex int, chunkIndex int)) {
+	indexChannel := make(chan int, chunkCount)
+	stopChannel := make(chan int, threads)
+	finishChannel := make(chan int, threads)
+
+	// Start the uploading goroutines
+	for i := 0; i < threads; i++ {
+		go func(threadIndex int) {
+			defer CatchLogException()
+			for {
+				select {
+				case chunkIndex := <-indexChannel:
+					job(threadIndex, chunkIndex)
+					finishChannel <- 0
+				case <-stopChannel:
+					return
+				}
+			}
+		}(i)
+	}
+
+	for i := 0; i < chunkCount; i++ {
+		indexChannel <- i
+	}
+
+	for i := 0; i < chunkCount; i++ {
+		<-finishChannel
+	}
+
+	for i := 0; i < threads; i++ {
+		stopChannel <- 0
+	}
+}
+
+func Benchmark(localDirectory string, storage Storage, fileSize int64, chunkSize int, chunkCount int, uploadThreads int, downloadThreads int) bool {
+
+	filename := filepath.Join(localDirectory, "benchmark.dat")
+
+	defer func() {
+		os.Remove(filename)
+	}()
+
+	LOG_INFO("BENCHMARK_GENERATE", "Generating %s byte random data in memory", PrettySize(fileSize))
+	data := make([]byte, fileSize)
+	_, err := rand.Read(data)
+	if err != nil {
+		LOG_ERROR("BENCHMARK_RAND", "Failed to generate random data: %v", err)
+		return false
+	}
+
+	startTime := float64(time.Now().UnixNano()) / 1e9
+	LOG_INFO("BENCHMARK_WRITE", "Writing random data to local disk")
+	err = ioutil.WriteFile(filename, data, 0600)
+	if err != nil {
+		LOG_ERROR("BENCHMARK_WRITE", "Failed to write the random data: %v", err)
+		return false
+	}
+
+	runningTime := float64(time.Now().UnixNano())/1e9 - startTime
+	speed := int64(float64(fileSize) / runningTime)
+	LOG_INFO("BENCHMARK_WRITE", "Wrote %s bytes in %.2fs: %s/s", PrettySize(fileSize), runningTime, PrettySize(speed))
+
+	startTime = float64(time.Now().UnixNano()) / 1e9
+	LOG_INFO("BENCHMARK_READ", "Reading the random data from local disk")
+	file, err := os.Open(filename)
+	if err != nil {
+		LOG_ERROR("BENCHMARK_OPEN", "Failed to open the random data file: %v", err)
+		return false
+	}
+	segment := make([]byte, 1024*1024)
+	for err == nil {
+		_, err = file.Read(segment)
+	}
+	if err != io.EOF {
+		LOG_ERROR("BENCHMARK_OPEN", "Failed to read the random data file: %v", err)
+		return false
+	}
+	file.Close()
+	runningTime = float64(time.Now().UnixNano())/1e9 - startTime
+	speed = int64(float64(fileSize) / runningTime)
+	LOG_INFO("BENCHMARK_READ", "Read %s bytes in %.2fs: %s/s", PrettySize(fileSize), runningTime, PrettySize(speed))
+
+	buffer := bytes.NewReader(data)
+	benchmarkSplit(buffer, fileSize, chunkSize, false, false, "without compression/encryption")
+	benchmarkSplit(buffer, fileSize, chunkSize, true, false, "with compression but without encryption")
+	benchmarkSplit(buffer, fileSize, chunkSize, true, true, "with compression and encryption")
+
+	storage.CreateDirectory(0, "benchmark")
+	existingFiles, _, err := storage.ListFiles(0, "benchmark/")
+	if err != nil {
+		LOG_ERROR("BENCHMARK_LIST", "Failed to list the benchmark directory: %v", err)
+		return false
+	}
+
+	var existingChunks []string
+	for _, f := range existingFiles {
+		if len(f) > 0 && f[len(f)-1] != '/' {
+			existingChunks = append(existingChunks, "benchmark/"+f)
+		}
+	}
+
+	if len(existingChunks) > 0 {
+		LOG_INFO("BENCHMARK_DELETE", "Deleting %d temporary files from previous benchmark runs", len(existingChunks))
+		benchmarkRun(uploadThreads, len(existingChunks), func(threadIndex int, chunkIndex int) {
+			storage.DeleteFile(threadIndex, existingChunks[chunkIndex])
+		})
+	}
+
+	chunks := make([][]byte, chunkCount)
+	chunkHashes := make([]string, chunkCount)
+	LOG_INFO("BENCHMARK_GENERATE", "Generating %d chunks", chunkCount)
+	for i := 0; i < chunkCount; i++ {
+		chunks[i] = make([]byte, chunkSize)
+		_, err = rand.Read(chunks[i])
+		if err != nil {
+			LOG_ERROR("BENCHMARK_RAND", "Failed to generate random data: %v", err)
+			return false
+		}
+		hashInBytes := sha256.Sum256(chunks[i])
+		chunkHashes[i] = hex.EncodeToString(hashInBytes[:])
+
+	}
+
+	startTime = float64(time.Now().UnixNano()) / 1e9
+	benchmarkRun(uploadThreads, chunkCount, func(threadIndex int, chunkIndex int) {
+		err := storage.UploadFile(threadIndex, fmt.Sprintf("benchmark/chunk%d", chunkIndex), chunks[chunkIndex])
+		if err != nil {
+			LOG_ERROR("BENCHMARK_UPLOAD", "Failed to upload the chunk: %v", err)
+			return
+		}
+	})
+
+	runningTime = float64(time.Now().UnixNano())/1e9 - startTime
+	speed = int64(float64(chunkSize*chunkCount) / runningTime)
+	LOG_INFO("BENCHMARK_UPLOAD", "Uploaded %s bytes in %.2fs: %s/s", PrettySize(int64(chunkSize*chunkCount)), runningTime, PrettySize(speed))
+
+	config := CreateConfig()
+
+	startTime = float64(time.Now().UnixNano()) / 1e9
+	hashError := false
+	benchmarkRun(downloadThreads, chunkCount, func(threadIndex int, chunkIndex int) {
+		chunk := config.GetChunk()
+		chunk.Reset(false)
+		err := storage.DownloadFile(threadIndex, fmt.Sprintf("benchmark/chunk%d", chunkIndex), chunk)
+		if err != nil {
+			LOG_ERROR("BENCHMARK_DOWNLOAD", "Failed to download the chunk: %v", err)
+			return
+		}
+
+		hashInBytes := sha256.Sum256(chunk.GetBytes())
+		hash := hex.EncodeToString(hashInBytes[:])
+		if hash != chunkHashes[chunkIndex] {
+			LOG_WARN("BENCHMARK_HASH", "Chunk %d has mismatched hashes: %s != %s", chunkIndex, chunkHashes[chunkIndex], hash)
+			hashError = true
+		}
+
+		config.PutChunk(chunk)
+	})
+
+	runningTime = float64(time.Now().UnixNano())/1e9 - startTime
+	speed = int64(float64(chunkSize*chunkCount) / runningTime)
+	LOG_INFO("BENCHMARK_DOWNLOAD", "Downloaded %s bytes in %.2fs: %s/s", PrettySize(int64(chunkSize*chunkCount)), runningTime, PrettySize(speed))
+
+	if !hashError {
+		benchmarkRun(uploadThreads, chunkCount, func(threadIndex int, chunkIndex int) {
+			storage.DeleteFile(threadIndex, fmt.Sprintf("benchmark/chunk%d", chunkIndex))
+		})
+		LOG_INFO("BENCHMARK_DELETE", "Deleted %d temporary files from the storage", chunkCount)
+	}
+
+	return true
+}
--- a/src/duplicacy_chunk.go
+++ b/src/duplicacy_chunk.go
@@ -5,11 +5,14 @@
 package duplicacy

 import (
+	"os"
 	"bytes"
 	"compress/zlib"
 	"crypto/aes"
 	"crypto/cipher"
 	"crypto/rand"
+	"crypto/hmac"
+	"crypto/sha256"
 	"encoding/hex"
 	"fmt"
 	"hash"
@@ -267,6 +270,17 @@ func (chunk *Chunk) Encrypt(encryptionKey []byte, derivationKey string) (err err

 }

+// This is to ensure compability with Vertical Backup, which still uses HMAC-SHA256 (instead of HMAC-BLAKE2) to
+// derive the key used to encrypt/decrypt files and chunks.
+
+var DecryptWithHMACSHA256 = false
+
+func init() {
+	if value, found := os.LookupEnv("DUPLICACY_DECRYPT_WITH_HMACSHA256"); found && value != "0" {
+		DecryptWithHMACSHA256 = true
+	}
+}
+
 // Decrypt decrypts the encrypted data stored in the chunk buffer.  If derivationKey is not nil, the actual
 // encryption key will be HMAC-SHA256(encryptionKey, derivationKey).
 func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err error) {
@@ -286,7 +300,13 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 		key := encryptionKey

 		if len(derivationKey) > 0 {
-			hasher := chunk.config.NewKeyedHasher([]byte(derivationKey))
+			var hasher hash.Hash
+			if DecryptWithHMACSHA256 {
+				hasher = hmac.New(sha256.New, []byte(derivationKey))
+			} else {
+				hasher = chunk.config.NewKeyedHasher([]byte(derivationKey))
+			}
+
 			hasher.Write(encryptionKey)
 			key = hasher.Sum(nil)
 		}
@@ -325,6 +345,7 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 			return err
 		}

+
 		paddingLength := int(decryptedBytes[len(decryptedBytes)-1])
 		if paddingLength == 0 {
 			paddingLength = 256
--- a/src/duplicacy_chunkdownloader.go
+++ b/src/duplicacy_chunkdownloader.go
@@ -174,7 +174,7 @@ func (downloader *ChunkDownloader) Prefetch(file *Entry) {
 // Reclaim releases the downloaded chunk to the chunk pool
 func (downloader *ChunkDownloader) Reclaim(chunkIndex int) {

-	if downloader.lastChunkIndex == chunkIndex {
+	if downloader.lastChunkIndex >= chunkIndex {
 		return
 	}

@@ -187,13 +187,20 @@ func (downloader *ChunkDownloader) Reclaim(chunkIndex int) {
 		}
 	}

+	for i := downloader.lastChunkIndex; i < chunkIndex; i++ {
+		// These chunks are never downloaded if 'isDownloading' is false; note that 'isDownloading' isn't reset to
+		// false after a chunk has been downloaded
+		if !downloader.taskList[i].isDownloading {
+			atomic.AddInt64(&downloader.totalChunkSize, -int64(downloader.taskList[i].chunkLength))
+		}
+	}
 	downloader.lastChunkIndex = chunkIndex
 }

 // WaitForChunk waits until the specified chunk is ready
 func (downloader *ChunkDownloader) WaitForChunk(chunkIndex int) (chunk *Chunk) {

-	// Reclain any chunk not needed
+	// Reclaim any chunk not needed
 	downloader.Reclaim(chunkIndex)

 	// If we haven't started download the specified chunk, download it now
@@ -317,8 +324,19 @@ func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadT
 			}

 			if !exist {
-				// Retry for the Hubic backend as it may return 404 even when the chunk exists
-				if _, ok := downloader.storage.(*HubicStorage); ok && downloadAttempt < MaxDownloadAttempts {
+
+				retry := false
+
+				// Retry for Hubic or WebDAV as it may return 404 even when the chunk exists
+				if _, ok := downloader.storage.(*HubicStorage); ok {
+					retry = true
+				}
+
+				if _, ok := downloader.storage.(*WebDAVStorage); ok {
+					retry = true
+				}
+
+				if retry && downloadAttempt < MaxDownloadAttempts {
 					LOG_WARN("DOWNLOAD_RETRY", "Failed to find the chunk %s; retrying", chunkID)
 					continue
 				}
--- a/src/duplicacy_chunkoperator.go
+++ b/src/duplicacy_chunkoperator.go
@@ -0,0 +1,209 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+package duplicacy
+
+import (
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+// These are operations that ChunkOperator will perform.
+const (
+	ChunkOperationFind      = 0
+	ChunkOperationDelete    = 1
+	ChunkOperationFossilize = 2
+	ChunkOperationResurrect = 3
+)
+
+// ChunkOperatorTask is used to pass paramaters for different kinds of chunk operations.
+type ChunkOperatorTask struct {
+	operation int    // The type of operation
+	chunkID   string // The chunk id
+	filePath  string // The path of the chunk file; it may be empty
+}
+
+// ChunkOperator is capable of performing multi-threaded operations on chunks.
+type ChunkOperator struct {
+	numberOfActiveTasks int64                  // The number of chunks that are being operated on
+	storage             Storage                // This storage
+	threads             int                    // Number of threads
+	taskQueue           chan ChunkOperatorTask // Operating goroutines are waiting on this channel for input
+	stopChannel         chan bool              // Used to stop all the goroutines
+
+	fossils     []string    // For fossilize operation, the paths of the fossils are stored in this slice
+	fossilsLock *sync.Mutex // The lock for 'fossils'
+}
+
+// CreateChunkOperator creates a new ChunkOperator.
+func CreateChunkOperator(storage Storage, threads int) *ChunkOperator {
+	operator := &ChunkOperator{
+		storage: storage,
+		threads: threads,
+
+		taskQueue:   make(chan ChunkOperatorTask, threads*4),
+		stopChannel: make(chan bool),
+
+		fossils:     make([]string, 0),
+		fossilsLock: &sync.Mutex{},
+	}
+
+	// Start the operator goroutines
+	for i := 0; i < operator.threads; i++ {
+		go func(threadIndex int) {
+			defer CatchLogException()
+			for {
+				select {
+				case task := <-operator.taskQueue:
+					operator.Run(threadIndex, task)
+				case <-operator.stopChannel:
+					return
+				}
+			}
+		}(i)
+	}
+
+	return operator
+}
+
+func (operator *ChunkOperator) Stop() {
+	if atomic.LoadInt64(&operator.numberOfActiveTasks) < 0 {
+		return
+	}
+
+	for atomic.LoadInt64(&operator.numberOfActiveTasks) > 0 {
+		time.Sleep(100 * time.Millisecond)
+	}
+	for i := 0; i < operator.threads; i++ {
+		operator.stopChannel <- false
+	}
+
+	// Assign -1 to numberOfActiveTasks so Stop() can be called multiple times
+	atomic.AddInt64(&operator.numberOfActiveTasks, int64(-1))
+}
+
+func (operator *ChunkOperator) AddTask(operation int, chunkID string, filePath string) {
+
+	task := ChunkOperatorTask{
+		operation: operation,
+		chunkID:   chunkID,
+		filePath:  filePath,
+	}
+	operator.taskQueue <- task
+	atomic.AddInt64(&operator.numberOfActiveTasks, int64(1))
+}
+
+func (operator *ChunkOperator) Find(chunkID string) {
+	operator.AddTask(ChunkOperationFind, chunkID, "")
+}
+
+func (operator *ChunkOperator) Delete(chunkID string, filePath string) {
+	operator.AddTask(ChunkOperationDelete, chunkID, filePath)
+}
+
+func (operator *ChunkOperator) Fossilize(chunkID string, filePath string) {
+	operator.AddTask(ChunkOperationFossilize, chunkID, filePath)
+}
+
+func (operator *ChunkOperator) Resurrect(chunkID string, filePath string) {
+	operator.AddTask(ChunkOperationResurrect, chunkID, filePath)
+}
+
+func (operator *ChunkOperator) Run(threadIndex int, task ChunkOperatorTask) {
+	defer func() {
+		atomic.AddInt64(&operator.numberOfActiveTasks, int64(-1))
+	}()
+
+	// task.filePath may be empty.  If so, find the chunk first.
+	if task.operation == ChunkOperationDelete || task.operation == ChunkOperationFossilize {
+		if task.filePath == "" {
+			filePath, exist, _, err := operator.storage.FindChunk(threadIndex, task.chunkID, false)
+			if err != nil {
+				LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", task.chunkID, err)
+				return
+			} else if !exist {
+				if task.operation == ChunkOperationDelete {
+					LOG_WARN("CHUNK_FIND", "Chunk %s does not exist in the storage", task.chunkID)
+					return
+				}
+
+				fossilPath, exist, _, _ := operator.storage.FindChunk(threadIndex, task.chunkID, true)
+				if exist {
+					LOG_WARN("CHUNK_FOSSILIZE", "Chunk %s is already a fossil", task.chunkID)
+					operator.fossilsLock.Lock()
+					operator.fossils = append(operator.fossils, fossilPath)
+					operator.fossilsLock.Unlock()
+				} else {
+					LOG_ERROR("CHUNK_FIND", "Chunk %s does not exist in the storage", task.chunkID)
+				}
+				return
+			}
+			task.filePath = filePath
+		}
+	}
+
+	if task.operation == ChunkOperationFind {
+		_, exist, _, err := operator.storage.FindChunk(threadIndex, task.chunkID, false)
+		if err != nil {
+			LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", task.chunkID, err)
+		} else if !exist {
+			LOG_ERROR("CHUNK_FIND", "Chunk %s does not exist in the storage", task.chunkID)
+		} else {
+			LOG_DEBUG("CHUNK_FIND", "Chunk %s exists in the storage", task.chunkID)
+		}
+	} else if task.operation == ChunkOperationDelete {
+		err := operator.storage.DeleteFile(threadIndex, task.filePath)
+		if err != nil {
+			LOG_WARN("CHUNK_DELETE", "Failed to remove the file %s: %v", task.filePath, err)
+		} else {
+			if task.chunkID != "" {
+				LOG_INFO("CHUNK_DELETE", "The chunk %s has been permanently removed", task.chunkID)
+			} else {
+				LOG_INFO("CHUNK_DELETE", "Deleted file %s from the storage", task.filePath)
+			}
+		}
+	} else if task.operation == ChunkOperationFossilize {
+
+		fossilPath := task.filePath + ".fsl"
+
+		err := operator.storage.MoveFile(threadIndex, task.filePath, fossilPath)
+		if err != nil {
+			if _, exist, _, _ := operator.storage.FindChunk(threadIndex, task.chunkID, true); exist {
+				err := operator.storage.DeleteFile(threadIndex, task.filePath)
+				if err == nil {
+					LOG_TRACE("CHUNK_DELETE", "Deleted chunk file %s as the fossil already exists", task.chunkID)
+				}
+				operator.fossilsLock.Lock()
+				operator.fossils = append(operator.fossils, fossilPath)
+				operator.fossilsLock.Unlock()
+			} else {
+				LOG_ERROR("CHUNK_DELETE", "Failed to fossilize the chunk %s: %v", task.chunkID, err)
+			}
+		} else {
+			LOG_TRACE("CHUNK_FOSSILIZE", "The chunk %s has been marked as a fossil", task.chunkID)
+			operator.fossilsLock.Lock()
+			operator.fossils = append(operator.fossils, fossilPath)
+			operator.fossilsLock.Unlock()
+		}
+	} else if task.operation == ChunkOperationResurrect {
+		chunkPath, exist, _, err := operator.storage.FindChunk(threadIndex, task.chunkID, false)
+		if err != nil {
+			LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", task.chunkID, err)
+		}
+
+		if exist {
+			operator.storage.DeleteFile(threadIndex, task.filePath)
+			LOG_INFO("FOSSIL_RESURRECT", "The chunk %s already exists", task.chunkID)
+		} else {
+			err := operator.storage.MoveFile(threadIndex, task.filePath, chunkPath)
+			if err != nil {
+				LOG_ERROR("FOSSIL_RESURRECT", "Failed to resurrect the chunk %s from the fossil %s: %v",
+					task.chunkID, task.filePath, err)
+			} else {
+				LOG_INFO("FOSSIL_RESURRECT", "The chunk %s has been resurrected", task.filePath)
+			}
+		}
+	}
+}
--- a/src/duplicacy_chunkuploader_test.go
+++ b/src/duplicacy_chunkuploader_test.go
@@ -111,7 +111,7 @@ func TestUploaderAndDownloader(t *testing.T) {
 	for i, chunk := range chunks {
 		downloaded := chunkDownloader.WaitForChunk(i)
 		if downloaded.GetID() != chunk.GetID() {
-			t.Error("Uploaded: %s, downloaded: %s", chunk.GetID(), downloaded.GetID())
+			t.Errorf("Uploaded: %s, downloaded: %s", chunk.GetID(), downloaded.GetID())
 		}
 	}

--- a/src/duplicacy_entry.go
+++ b/src/duplicacy_entry.go
@@ -22,6 +22,9 @@ import (
 var DUPLICACY_DIRECTORY = ".duplicacy"
 var DUPLICACY_FILE = ".duplicacy"

+// Mask for file permission bits
+var fileModeMask = os.ModePerm | os.ModeSetuid | os.ModeSetgid | os.ModeSticky
+
 // Regex for matching 'StartChunk:StartOffset:EndChunk:EndOffset'
 var contentRegex = regexp.MustCompile(`^([0-9]+):([0-9]+):([0-9]+):([0-9]+)`)

@@ -269,7 +272,7 @@ func (entry *Entry) IsLink() bool {
 }

 func (entry *Entry) GetPermissions() os.FileMode {
-	return os.FileMode(entry.Mode) & os.ModePerm
+	return os.FileMode(entry.Mode)&fileModeMask
 }

 func (entry *Entry) IsSameAs(other *Entry) bool {
@@ -297,7 +300,14 @@ func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo, setO
 		}
 	}

-	if (*fileInfo).Mode()&os.ModePerm != entry.GetPermissions() {
+	// Note that chown can remove setuid/setgid bits so should be called before chmod
+	if setOwner {
+		if !SetOwner(fullPath, entry, fileInfo) {
+			return false
+		}
+	}
+
+	if (*fileInfo).Mode()&fileModeMask != entry.GetPermissions() {
 		err := os.Chmod(fullPath, entry.GetPermissions())
 		if err != nil {
 			LOG_ERROR("RESTORE_CHMOD", "Failed to set the file permissions: %v", err)
@@ -318,11 +328,7 @@ func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo, setO
 		entry.SetAttributesToFile(fullPath)
 	}

-	if setOwner {
-		return SetOwner(fullPath, entry, fileInfo)
-	} else {
 	return true
-	}
 }

 // Return -1 if 'left' should appear before 'right', 1 if opposite, and 0 if they are the same.
@@ -435,7 +441,7 @@ func (files FileInfoCompare) Less(i, j int) bool {

 // ListEntries returns a list of entries representing file and subdirectories under the directory 'path'.  Entry paths
 // are normalized as relative to 'top'.  'patterns' are used to exclude or include certain files.
-func ListEntries(top string, path string, fileList *[]*Entry, patterns []string, discardAttributes bool) (directoryList []*Entry,
+func ListEntries(top string, path string, fileList *[]*Entry, patterns []string, nobackupFile string, discardAttributes bool) (directoryList []*Entry,
 	skippedFiles []string, err error) {

 	LOG_DEBUG("LIST_ENTRIES", "Listing %s", path)
@@ -449,6 +455,15 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns []string,
 		return directoryList, nil, err
 	}
 	
+	// This binary search works because ioutil.ReadDir returns files sorted by Name() by default
+	if nobackupFile != "" {
+		ii := sort.Search(len(files), func(ii int) bool { return strings.Compare(files[ii].Name(), nobackupFile) >= 0})
+		if ii < len(files) && files[ii].Name() == nobackupFile {
+			LOG_DEBUG("LIST_NOBACKUP", "%s is excluded due to nobackup file", path)
+			return directoryList, skippedFiles, nil
+		}
+	}
+
 	normalizedPath := path
 	if len(normalizedPath) > 0 && normalizedPath[len(normalizedPath)-1] != '/' {
 		normalizedPath += "/"
@@ -469,7 +484,6 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns []string,
 		}
 		entry := CreateEntryFromFileInfo(f, normalizedPath)
 		if len(patterns) > 0 && !MatchPath(entry.Path, patterns) {
-			LOG_DEBUG("LIST_EXCLUDE", "%s is excluded", entry.Path)
 			continue
 		}
 		if entry.IsLink() {
@@ -483,7 +497,7 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns []string,

 			if isRegular {
 				entry.Mode ^= uint32(os.ModeSymlink)
-			} else if path == "" && filepath.IsAbs(entry.Link) && !strings.HasPrefix(entry.Link, normalizedTop) {
+			} else if path == "" && (filepath.IsAbs(entry.Link) || filepath.HasPrefix(entry.Link, `\\`)) && !strings.HasPrefix(entry.Link, normalizedTop) {
 				stat, err := os.Stat(filepath.Join(top, entry.Path))
 				if err != nil {
 					LOG_WARN("LIST_LINK", "Failed to read the symlink: %v", err)
--- a/src/duplicacy_entry_test.go
+++ b/src/duplicacy_entry_test.go
@@ -173,7 +173,7 @@ func TestEntryList(t *testing.T) {
 		directory := directories[len(directories)-1]
 		directories = directories[:len(directories)-1]
 		entries = append(entries, directory)
-		subdirectories, _, err := ListEntries(testDir, directory.Path, &entries, nil, false)
+		subdirectories, _, err := ListEntries(testDir, directory.Path, &entries, nil, "", false)
 		if err != nil {
 			t.Errorf("ListEntries(%s, %s) returned an error: %s", testDir, directory.Path, err)
 		}
--- a/src/duplicacy_log.go
+++ b/src/duplicacy_log.go
@@ -129,7 +129,7 @@ func logf(level int, logID string, format string, v ...interface{}) {
 	// fmt.Printf("%s %s %s %s\n", now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)

 	if testingT != nil {
-		if level < WARN {
+		if level <= WARN {
 			if level >= loggingLevel {
 				testingT.Logf("%s %s %s %s\n",
 					now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
--- a/src/duplicacy_preference.go
+++ b/src/duplicacy_preference.go
@@ -17,11 +17,13 @@ import (
 type Preference struct {
 	Name              string            `json:"name"`
 	SnapshotID        string            `json:"id"`
+	RepositoryPath    string            `json:"repository"`
 	StorageURL        string            `json:"storage"`
 	Encrypted         bool              `json:"encrypted"`
 	BackupProhibited  bool              `json:"no_backup"`
 	RestoreProhibited bool              `json:"no_restore"`
 	DoNotSavePassword bool              `json:"no_save_password"`
+	NobackupFile      string            `json:"nobackup_file"`
 	Keys              map[string]string `json:"keys"`
 }

--- a/src/duplicacy_s3storage.go
+++ b/src/duplicacy_s3storage.go
@@ -2,6 +2,11 @@
 // Free for personal use and commercial trial
 // Commercial use requires per-user licenses available from https://duplicacy.com

+// NOTE: The code in the Wasabi storage module relies on all functions
+// in this one except MoveFile(), IsMoveFileImplemented() and
+// IsStrongConsistent().  Changes to the API here will need to be
+// reflected there.
+
 package duplicacy

 import (
--- a/src/duplicacy_shadowcopy.go
+++ b/src/duplicacy_shadowcopy.go
@@ -3,10 +3,11 @@
 // Commercial use requires per-user licenses available from https://duplicacy.com

 // +build !windows
+// +build !darwin

 package duplicacy

-func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
+func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string) {
 	return top
 }

--- a/src/duplicacy_shadowcopy_darwin.go
+++ b/src/duplicacy_shadowcopy_darwin.go
@@ -0,0 +1,170 @@
+//
+// Shadow copy module for Mac OSX using APFS snapshot
+//
+//
+// This module copyright 2018 Adam Marcus (https://github.com/amarcu5)
+// and may be distributed under the same terms as Duplicacy.
+
+package duplicacy
+
+import (
+	"context"
+	"errors"
+	"io/ioutil"
+	"os"
+	"os/exec"
+	"strings"
+	"syscall"
+	"time"
+)
+
+var snapshotPath string
+var snapshotDate string
+
+// Converts char array to string
+func CharsToString(ca []int8) string {
+
+	len := len(ca)
+	ba := make([]byte, len) 
+
+	for i, v := range ca {
+		ba[i] = byte(v)
+		if ba[i] == 0 {
+			len = i
+			break
+		}
+	}
+
+	return string(ba[:len])
+}
+
+// Get ID of device containing path
+func GetPathDeviceId(path string) (deviceId int32, err error) {
+
+	stat := syscall.Stat_t{}
+
+	err = syscall.Stat(path, &stat)
+	if err != nil {
+		return 0, err
+	}
+
+	return stat.Dev, nil
+}
+
+// Executes shell command with timeout and returns stdout
+func CommandWithTimeout(timeoutInSeconds int, name string, arg ...string) (output string, err error) {
+
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutInSeconds) * time.Second)
+	defer cancel() 
+
+	cmd := exec.CommandContext(ctx, name, arg...)
+	out, err := cmd.Output()
+
+	if ctx.Err() == context.DeadlineExceeded {
+		err = errors.New("Command '" + name + "' timed out")
+	}
+
+	output = string(out)
+	return output, err
+}
+
+func DeleteShadowCopy() {
+
+	if snapshotPath == "" {
+		return
+	}
+
+	err := exec.Command("/sbin/umount", "-f", snapshotPath).Run()
+	if err != nil {
+		LOG_ERROR("VSS_DELETE", "Error while unmounting snapshot")
+		return
+	}
+
+	err = exec.Command("tmutil", "deletelocalsnapshots", snapshotDate).Run()
+	if err != nil {
+		LOG_ERROR("VSS_DELETE", "Error while deleting local snapshot")
+		return
+	}
+
+	err = os.RemoveAll(snapshotPath)
+	if err != nil {
+		LOG_ERROR("VSS_DELETE", "Error while deleting temporary mount directory")
+		return
+	}
+	
+	LOG_INFO("VSS_DELETE", "Shadow copy unmounted and deleted at %s", snapshotPath)
+
+	snapshotPath = "" 
+}
+
+func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string) {
+
+	if !shadowCopy {
+		return top
+	}
+
+	// Check repository filesystem is APFS
+	stat := syscall.Statfs_t{}
+	err := syscall.Statfs(top, &stat)
+	if err != nil {
+		LOG_ERROR("VSS_INIT", "Unable to determine filesystem of repository path")
+		return top
+	}
+	if CharsToString(stat.Fstypename[:]) != "apfs" {
+		LOG_WARN("VSS_INIT", "VSS requires APFS filesystem")
+		return top
+	}
+
+	// Check path is local as tmutil snapshots will not support APFS formatted external drives
+	deviceIdLocal, err := GetPathDeviceId("/")
+	if err != nil {
+		LOG_ERROR("VSS_INIT", "Unable to get device ID of path: /")
+		return top
+	}
+	deviceIdRepository, err := GetPathDeviceId(top)
+	if err != nil {
+		LOG_ERROR("VSS_INIT", "Unable to get device ID of path: ", top)
+		return top
+	}
+	if deviceIdLocal != deviceIdRepository {
+		LOG_WARN("VSS_PATH", "VSS not supported for non-local repository path: ", top)
+		return top 
+	}
+
+	if timeoutInSeconds <= 60 {
+		timeoutInSeconds = 60
+	}
+
+	// Create mount point
+	snapshotPath, err = ioutil.TempDir("/tmp/", "snp_")
+	if err != nil {
+		LOG_ERROR("VSS_CREATE", "Failed to create temporary mount directory")
+		return top
+	}
+
+	// Use tmutil to create snapshot
+	tmutilOutput, err := CommandWithTimeout(timeoutInSeconds, "tmutil", "snapshot")
+	if err != nil {
+		LOG_ERROR("VSS_CREATE", "Error while calling tmutil: ", err)
+		return top
+	}
+
+	colonPos := strings.IndexByte(tmutilOutput, ':')
+	if colonPos < 0 {
+		LOG_ERROR("VSS_CREATE", "Snapshot creation failed: ", tmutilOutput)
+		return top
+	}
+	snapshotDate = strings.TrimSpace(tmutilOutput[colonPos+1:])
+
+	// Mount snapshot as readonly and hide from GUI i.e. Finder
+	_, err = CommandWithTimeout(timeoutInSeconds, 
+		"/sbin/mount", "-t", "apfs", "-o", "nobrowse,-r,-s=com.apple.TimeMachine." + snapshotDate, "/", snapshotPath)
+	if err != nil {
+		LOG_ERROR("VSS_CREATE", "Error while mounting snapshot: ", err)
+		return top
+	}
+
+	LOG_INFO("VSS_DONE", "Shadow copy created and mounted at %s", snapshotPath)
+
+	return snapshotPath + top
+}
--- a/src/duplicacy_shadowcopy_windows.go
+++ b/src/duplicacy_shadowcopy_windows.go
@@ -353,12 +353,15 @@ func DeleteShadowCopy() {
 	ole.CoUninitialize()
 }

-func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
+func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string) {

 	if !shadowCopy {
 		return top
 	}

+	if timeoutInSeconds <= 60 {
+		timeoutInSeconds = 60
+	}
 	ole.CoInitialize(0)
 	defer ole.CoUninitialize()

@@ -416,7 +419,7 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 		return top
 	}

-	if !async.Wait(60) {
+	if !async.Wait(timeoutInSeconds) {
 		LOG_ERROR("VSS_GATHER", "Shadow copy creation failed: GatherWriterMetadata didn't finish properly")
 		return top
 	}
@@ -456,7 +459,7 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 		return top
 	}

-	if !async.Wait(60) {
+	if !async.Wait(timeoutInSeconds) {
 		LOG_ERROR("VSS_PREPARE", "Shadow copy creation failed: PrepareForBackup didn't finish properly")
 		return top
 	}
@@ -473,7 +476,7 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 		return top
 	}

-	if !async.Wait(180) {
+	if !async.Wait(timeoutInSeconds) {
 		LOG_ERROR("VSS_SNAPSHOT", "Shadow copy creation failed: DoSnapshotSet didn't finish properly")
 		return top
 	}
--- a/src/duplicacy_snapshot.go
+++ b/src/duplicacy_snapshot.go
@@ -57,7 +57,7 @@ func CreateEmptySnapshot(id string) (snapshto *Snapshot) {

 // CreateSnapshotFromDirectory creates a snapshot from the local directory 'top'.  Only 'Files'
 // will be constructed, while 'ChunkHashes' and 'ChunkLengths' can only be populated after uploading.
-func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, skippedDirectories []string,
+func CreateSnapshotFromDirectory(id string, top string, nobackupFile string) (snapshot *Snapshot, skippedDirectories []string,
 	skippedFiles []string, err error) {

 	snapshot = &Snapshot{
@@ -125,7 +125,7 @@ func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, ski
 		directory := directories[len(directories)-1]
 		directories = directories[:len(directories)-1]
 		snapshot.Files = append(snapshot.Files, directory)
-		subdirectories, skipped, err := ListEntries(top, directory.Path, &snapshot.Files, patterns, snapshot.discardAttributes)
+		subdirectories, skipped, err := ListEntries(top, directory.Path, &snapshot.Files, patterns, nobackupFile, snapshot.discardAttributes)
 		if err != nil {
 			LOG_WARN("LIST_FAILURE", "Failed to list subdirectory: %v", err)
 			skippedDirectories = append(skippedDirectories, directory.Path)
@@ -357,6 +357,11 @@ func (snapshot *Snapshot) LoadChunks(description []byte) (err error) {
 	return err
 }

+// ClearChunks removes loaded chunks from memory
+func (snapshot *Snapshot) ClearChunks() {
+	snapshot.ChunkHashes = nil
+}
+
 // LoadLengths construct 'ChunkLengths' from the json description.
 func (snapshot *Snapshot) LoadLengths(description []byte) (err error) {
 	return json.Unmarshal(description, &snapshot.ChunkLengths)
--- a/src/duplicacy_snapshotmanager.go
+++ b/src/duplicacy_snapshotmanager.go
@@ -26,6 +26,7 @@ import (

 const (
 	secondsInDay = 86400
+	chunkDir     = "chunks/"
 )

 // FossilCollection contains fossils and temporary files found during a snapshot deletions.
@@ -37,6 +38,9 @@ type FossilCollection struct {
 	// The lastest revision for each snapshot id when the fossil collection was created.
 	LastRevisions map[string]int `json:"last_revisions"`

+	// Record the set of snapshots that have been removed by the prune command that created this fossil collection
+	DeletedRevisions map[string][]int `json:"deleted_revisions"`
+
 	// Fossils (i.e., chunks not referenced by any snapshots)
 	Fossils []string `json:"fossils"`

@@ -54,6 +58,7 @@ func CreateFossilCollection(allSnapshots map[string][]*Snapshot) *FossilCollecti

 	return &FossilCollection{
 		LastRevisions: lastRevisions,
+		DeletedRevisions: make(map[string][]int),
 	}
 }

@@ -104,6 +109,8 @@ func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignored
 				continue
 			}

+			newSnapshots = append(newSnapshots, snapshot)
+
 			extraTime := 0
 			if !isStrongConsistent {
 				extraTime = secondsInDay / 2
@@ -113,7 +120,6 @@ func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignored
 			// might be in progress (although very unlikely).  So we only deem it deletable if that is not the case.
 			if snapshot.EndTime > collection.EndTime+int64(extraTime) {
 				hasNewSnapshot[hostID] = true
-				newSnapshots = append(newSnapshots, snapshot)
 				break
 			} else {
 				LOG_TRACE("SNAPSHOT_UNDELETABLE",
@@ -174,6 +180,7 @@ type SnapshotManager struct {
 	snapshotCache *FileStorage

 	chunkDownloader *ChunkDownloader
+	chunkOperator   *ChunkOperator
 }

 // CreateSnapshotManager creates a snapshot manager
@@ -388,7 +395,7 @@ func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, all
 	chunks := make(map[string]bool)

 	if latestSnapshot != nil {
-		for _, chunkID := range manager.GetSnapshotChunks(latestSnapshot) {
+		for _, chunkID := range manager.GetSnapshotChunks(latestSnapshot, true) {
 			chunks[chunkID] = true
 		}
 	}
@@ -456,7 +463,7 @@ func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, all
 		}
 	}

-	allFiles, _ := manager.ListAllFiles(manager.snapshotCache, "chunks/")
+	allFiles, _ := manager.ListAllFiles(manager.snapshotCache, chunkDir)
 	for _, file := range allFiles {
 		if file[len(file)-1] != '/' {
 			chunkID := strings.Replace(file, "/", "", -1)
@@ -598,8 +605,9 @@ func (manager *SnapshotManager) ListAllFiles(storage Storage, top string) (allFi
 	return allFiles, allSizes
 }

-// GetSnapshotChunks returns all chunks referenced by a given snapshot.
-func (manager *SnapshotManager) GetSnapshotChunks(snapshot *Snapshot) (chunks []string) {
+// GetSnapshotChunks returns all chunks referenced by a given snapshot. If
+// keepChunkHashes is true, snapshot.ChunkHashes will be populated.
+func (manager *SnapshotManager) GetSnapshotChunks(snapshot *Snapshot, keepChunkHashes bool) (chunks []string) {

 	for _, chunkHash := range snapshot.FileSequence {
 		chunks = append(chunks, manager.config.GetChunkIDFromHash(chunkHash))
@@ -628,6 +636,10 @@ func (manager *SnapshotManager) GetSnapshotChunks(snapshot *Snapshot) (chunks []
 		chunks = append(chunks, manager.config.GetChunkIDFromHash(chunkHash))
 	}

+	if !keepChunkHashes {
+		snapshot.ClearChunks()
+	}
+
 	return chunks
 }

@@ -716,7 +728,7 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 			}

 			if showChunks {
-				for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+				for _, chunkID := range manager.GetSnapshotChunks(snapshot, false) {
 					LOG_INFO("SNAPSHOT_CHUNKS", "chunk: %s", chunkID)
 				}
 			}
@@ -749,7 +761,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 	chunkSnapshotMap := make(map[string]int)

 	LOG_INFO("SNAPSHOT_CHECK", "Listing all chunks")
-	allChunks, allSizes := manager.ListAllFiles(manager.storage, "chunks/")
+	allChunks, allSizes := manager.ListAllFiles(manager.storage, chunkDir)

 	for i, chunk := range allChunks {
 		if len(chunk) == 0 || chunk[len(chunk)-1] == '/' {
@@ -780,6 +792,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 	}

 	snapshotIDIndex := 0
+	totalMissingChunks := 0
 	for snapshotID, _ = range snapshotMap {

 		revisions := revisionsToCheck
@@ -806,7 +819,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 			}

 			chunks := make(map[string]bool)
-			for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+			for _, chunkID := range manager.GetSnapshotChunks(snapshot, false) {
 				chunks[chunkID] = true
 			}

@@ -865,18 +878,23 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 			}

 			if missingChunks > 0 {
-				LOG_ERROR("SNAPSHOT_CHECK", "Some chunks referenced by snapshot %s at revision %d are missing",
+				LOG_WARN("SNAPSHOT_CHECK", "Some chunks referenced by snapshot %s at revision %d are missing",
 					snapshotID, revision)
-				return false
-			}
-
+				totalMissingChunks += missingChunks
+			} else {
 				LOG_INFO("SNAPSHOT_CHECK", "All chunks referenced by snapshot %s at revision %d exist",
 					snapshotID, revision)
 			}
+		}

 		snapshotIDIndex += 1
 	}

+	if totalMissingChunks > 0 {
+		LOG_ERROR("SNAPSHOT_CHECK", "Some chunks referenced by some snapshots do not exist in the storage")
+		return false
+	}
+
 	if showTabular {
 		manager.ShowStatisticsTabular(snapshotMap, chunkSizeMap, chunkUniqueMap, chunkSnapshotMap)
 	} else if showStatistics {
@@ -896,7 +914,7 @@ func (manager *SnapshotManager) ShowStatistics(snapshotMap map[string][]*Snapsho
 		for _, snapshot := range snapshotList {

 			chunks := make(map[string]bool)
-			for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+			for _, chunkID := range manager.GetSnapshotChunks(snapshot, false) {
 				chunks[chunkID] = true
 				snapshotChunks[chunkID] = true
 			}
@@ -949,7 +967,7 @@ func (manager *SnapshotManager) ShowStatisticsTabular(snapshotMap map[string][]*
 		earliestSeenChunks := make(map[string]int)

 		for _, snapshot := range snapshotList {
-			for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+			for _, chunkID := range manager.GetSnapshotChunks(snapshot, true) {
 				if earliestSeenChunks[chunkID] == 0 {
 					earliestSeenChunks[chunkID] = math.MaxInt32
 				}
@@ -960,7 +978,7 @@ func (manager *SnapshotManager) ShowStatisticsTabular(snapshotMap map[string][]*
 		for _, snapshot := range snapshotList {

 			chunks := make(map[string]bool)
-			for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+			for _, chunkID := range manager.GetSnapshotChunks(snapshot, true) {
 				chunks[chunkID] = true
 				snapshotChunks[chunkID] = true
 			}
@@ -1234,7 +1252,7 @@ func (manager *SnapshotManager) PrintFile(snapshotID string, revision int, path

 // Diff compares two snapshots, or two revision of a file if the file argument is given.
 func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []int,
-	filePath string, compareByHash bool) bool {
+	filePath string, compareByHash bool, nobackupFile string) bool {

 	LOG_DEBUG("DIFF_PARAMETERS", "top: %s, id: %s, revision: %v, path: %s, compareByHash: %t",
 		top, snapshotID, revisions, filePath, compareByHash)
@@ -1247,7 +1265,7 @@ func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []
 	if len(revisions) <= 1 {
 		// Only scan the repository if filePath is not provided
 		if len(filePath) == 0 {
-			rightSnapshot, _, _, err = CreateSnapshotFromDirectory(snapshotID, top)
+			rightSnapshot, _, _, err = CreateSnapshotFromDirectory(snapshotID, top, nobackupFile)
 			if err != nil {
 				LOG_ERROR("SNAPSHOT_LIST", "Failed to list the directory %s: %v", top, err)
 				return false
@@ -1484,7 +1502,7 @@ func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revis
 		}
 		if showLocalHash {
 			localFile.Hash = manager.config.ComputeFileHash(joinPath(top, filePath), make([]byte, 32*1024))
-			if lastVersion.Hash != localFile.Hash {
+			if lastVersion == nil || lastVersion.Hash != localFile.Hash {
 				modifiedFlag = "*"
 			}
 		}
@@ -1497,36 +1515,11 @@ func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revis
 }

 // fossilizeChunk turns the chunk into a fossil.
-func (manager *SnapshotManager) fossilizeChunk(chunkID string, filePath string,
-	exclusive bool, collection *FossilCollection) bool {
+func (manager *SnapshotManager) fossilizeChunk(chunkID string, filePath string, exclusive bool) bool {
 	if exclusive {
-		err := manager.storage.DeleteFile(0, filePath)
-		if err != nil {
-			LOG_ERROR("CHUNK_DELETE", "Failed to remove the chunk %s: %v", chunkID, err)
-			return false
+		manager.chunkOperator.Delete(chunkID, filePath)
 	} else {
-			LOG_TRACE("CHUNK_DELETE", "Deleted chunk file %s", chunkID)
-		}
-
-	} else {
-		fossilPath := filePath + ".fsl"
-
-		err := manager.storage.MoveFile(0, filePath, fossilPath)
-		if err != nil {
-			if _, exist, _, _ := manager.storage.FindChunk(0, chunkID, true); exist {
-				err := manager.storage.DeleteFile(0, filePath)
-				if err == nil {
-					LOG_TRACE("CHUNK_DELETE", "Deleted chunk file %s as the fossil already exists", chunkID)
-				}
-			} else {
-				LOG_ERROR("CHUNK_DELETE", "Failed to fossilize the chunk %s: %v", chunkID, err)
-				return false
-			}
-		} else {
-			LOG_TRACE("CHUNK_FOSSILIZE", "Fossilized chunk %s", chunkID)
-		}
-
-		collection.AddFossil(fossilPath)
+		manager.chunkOperator.Fossilize(chunkID, filePath)
 	}

 	return true
@@ -1575,7 +1568,7 @@ func (manager *SnapshotManager) resurrectChunk(fossilPath string, chunkID string
 func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string, revisionsToBeDeleted []int,
 	tags []string, retentions []string,
 	exhaustive bool, exclusive bool, ignoredIDs []string,
-	dryRun bool, deleteOnly bool, collectOnly bool) bool {
+	dryRun bool, deleteOnly bool, collectOnly bool, threads int) bool {

 	LOG_DEBUG("DELETE_PARAMETERS",
 		"id: %s, revisions: %v, tags: %v, retentions: %v, exhaustive: %t, exclusive: %t, "+
@@ -1587,15 +1580,27 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		LOG_WARN("DELETE_OPTIONS", "Tags or retention policy will be ignored if at least one revision is specified")
 	}

-	preferencePath := GetDuplicacyPreferencePath()
-	logDir := path.Join(preferencePath, "logs")
-	os.Mkdir(logDir, 0700)
+	manager.chunkOperator = CreateChunkOperator(manager.storage, threads)
+	defer manager.chunkOperator.Stop()
+
+	prefPath := GetDuplicacyPreferencePath()
+	logDir := path.Join(prefPath, "logs")
+	err := os.MkdirAll(logDir, 0700)
+	if err != nil {
+		LOG_ERROR("LOG_DIR", "Could not open log directory %s: %v", logDir, err)
+	}
 	logFileName := path.Join(logDir, time.Now().Format("prune-log-20060102-150405"))
 	logFile, err := os.OpenFile(logFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
+	if err != nil {
+		LOG_ERROR("LOG_FILE", "Could not open log file %s: %v", logFileName, err)
+	}

 	defer func() {
 		if logFile != nil {
-			logFile.Close()
+			cerr := logFile.Close()
+			if cerr != nil {
+				LOG_WARN("LOG_FILE", "Could not close log file %s: %v", logFileName, cerr)
+			}
 		}
 	}()

@@ -1674,7 +1679,8 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 	}

 	for _, id := range snapshotIDs {
-		revisions, err := manager.ListSnapshotRevisions(id)
+		var revisions []int
+		revisions, err = manager.ListSnapshotRevisions(id)
 		if err != nil {
 			LOG_ERROR("SNAPSHOT_LIST", "Failed to list all revisions for snapshot %s: %v", id, err)
 			return false
@@ -1694,14 +1700,20 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		}
 	}

-	chunkDir := "chunks/"
-
 	collectionRegex := regexp.MustCompile(`^([0-9]+)$`)

 	collectionDir := "fossils"
-	manager.snapshotCache.CreateDirectory(0, collectionDir)
+	err = manager.snapshotCache.CreateDirectory(0, collectionDir)
+	if err != nil {
+		LOG_ERROR("FOSSIL_COLLECT", "Failed to create collection directory %s: %v", collectionDir, err)
+		return false
+	}

 	collections, _, err := manager.snapshotCache.ListFiles(0, collectionDir)
+	if err != nil {
+		LOG_ERROR("FOSSIL_COLLECT", "Failed to list fossil collection files for dir %s: %v", collectionDir, err)
+		return false
+	}
 	maxCollectionNumber := 0

 	referencedFossils := make(map[string]bool)
@@ -1727,7 +1739,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		collectionFile := path.Join(collectionDir, collectionName)
 		manager.fileChunk.Reset(false)

-		err := manager.snapshotCache.DownloadFile(0, collectionFile, manager.fileChunk)
+		err = manager.snapshotCache.DownloadFile(0, collectionFile, manager.fileChunk)
 		if err != nil {
 			LOG_ERROR("FOSSIL_COLLECT", "Failed to read the fossil collection file %s: %v", collectionFile, err)
 			return false
@@ -1740,6 +1752,29 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 			return false
 		}

+		// Determine if any deleted revisions exist
+		exist := false
+		for snapshotID, revisionList := range collection.DeletedRevisions {
+			for _, revision := range revisionList {
+				for _, snapshot := range allSnapshots[snapshotID] {
+					if revision == snapshot.Revision {
+						LOG_INFO("FOSSIL_GHOSTSNAPSHOT", "Snapshot %s revision %d should have been deleted already", snapshotID, revision)
+						exist = true
+					}
+				}
+			}
+		}
+
+		if exist {
+			err = manager.snapshotCache.DeleteFile(0, collectionFile)
+			if err != nil {
+				LOG_WARN("FOSSIL_FILE", "Failed to remove the fossil collection file %s: %v", collectionFile, err)
+			} else {
+				LOG_INFO("FOSSIL_IGNORE", "The fossil collection file %s has been ignored due to ghost snapshots", collectionFile)
+			}
+			continue
+		}
+
 		for _, fossil := range collection.Fossils {
 			referencedFossils[fossil] = true
 		}
@@ -1756,7 +1791,9 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 			newChunks := make(map[string]bool)

 			for _, newSnapshot := range newSnapshots {
-				for _, chunk := range manager.GetSnapshotChunks(newSnapshot) {
+				fmt.Fprintf(logFile, "Snapshot %s revision %d was created after collection %s\n", newSnapshot.ID, newSnapshot.Revision, collectionName)
+				LOG_INFO("PRUNE_NEWSNAPSHOT", "Snapshot %s revision %d was created after collection %s", newSnapshot.ID, newSnapshot.Revision, collectionName)
+				for _, chunk := range manager.GetSnapshotChunks(newSnapshot, false) {
 					newChunks[chunk] = true
 				}
 			}
@@ -1774,15 +1811,14 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 						continue
 					}

-					manager.resurrectChunk(fossil, chunk)
+					manager.chunkOperator.Resurrect(chunk, fossil)
 					fmt.Fprintf(logFile, "Resurrected fossil %s (collection %s)\n", chunk, collectionName)

 				} else {
 					if dryRun {
 						LOG_INFO("FOSSIL_DELETE", "The chunk %s would be permanently removed", chunk)
 					} else {
-						manager.storage.DeleteFile(0, fossil)
-						LOG_INFO("FOSSIL_DELETE", "The chunk %s has been permanently removed", chunk)
+						manager.chunkOperator.Delete(chunk, fossil)
 						fmt.Fprintf(logFile, "Deleted fossil %s (collection %s)\n", chunk, collectionName)
 					}
 				}
@@ -1794,8 +1830,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 					LOG_INFO("TEMPORARY_DELETE", "The temporary file %s would be deleted", temporary)
 				} else {
 					// Fail silently, since temporary files are supposed to be renamed or deleted after upload is done
-					manager.storage.DeleteFile(0, temporary)
-					LOG_INFO("TEMPORARY_DELETE", "The temporary file %s has been deleted", temporary)
+					manager.chunkOperator.Delete("", temporary)
 					fmt.Fprintf(logFile, "Deleted temporary %s (collection %s)\n", temporary, collectionName)
 				}
 			}
@@ -1912,208 +1947,46 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		}
 	}

-	if toBeDeleted == 0 && exhaustive == false {
+	if toBeDeleted == 0 && !exhaustive {
 		LOG_INFO("SNAPSHOT_NONE", "No snapshot to delete")
 		return false
 	}

-	chunkRegex := regexp.MustCompile(`^[0-9a-f]+$`)
-
-	referencedChunks := make(map[string]bool)
-
-	// Now build all chunks referened by snapshot not deleted
-	for _, snapshots := range allSnapshots {
-
-		if len(snapshots) > 0 {
-			latest := snapshots[len(snapshots)-1]
-			if latest.Flag && !exclusive {
-				LOG_ERROR("SNAPSHOT_DELETE",
-					"The latest snapshot %s at revision %d can't be deleted in non-exclusive mode",
-					latest.ID, latest.Revision)
-				return false
-			}
-		}
-
-		for _, snapshot := range snapshots {
-			if snapshot.Flag {
-				LOG_INFO("SNAPSHOT_DELETE", "Deleting snapshot %s at revision %d", snapshot.ID, snapshot.Revision)
-				continue
-			}
-
-			chunks := manager.GetSnapshotChunks(snapshot)
-
-			for _, chunk := range chunks {
-				// The initial value is 'false'.  When a referenced chunk is found it will change the value to 'true'.
-				referencedChunks[chunk] = false
-			}
-		}
-	}
-
 	collection := CreateFossilCollection(allSnapshots)

+	var success bool
 	if exhaustive {
-
-		// In exhaustive, we scan the entire chunk tree to find dangling chunks and temporaries.
-		allFiles, _ := manager.ListAllFiles(manager.storage, chunkDir)
-		for _, file := range allFiles {
-			if file[len(file)-1] == '/' {
-				continue
+		success = manager.pruneSnapshotsExhaustive(referencedFossils, allSnapshots, collection, logFile, dryRun, exclusive)
+	} else {
+		success = manager.pruneSnapshotsNonExhaustive(allSnapshots, collection, logFile, dryRun, exclusive)
 	}
-
-			if strings.HasSuffix(file, ".tmp") {
-
-				// This is a temporary chunk file.  It can be a result of a restore operation still in progress, or
-				// a left-over from a restore operation that was terminated abruptly.
-				if dryRun {
-					LOG_INFO("CHUNK_TEMPORARY", "Found temporary file %s", file)
-					continue
-				}
-
-				if exclusive {
-					// In exclusive mode, we assume no other restore operation is running concurrently.
-					err := manager.storage.DeleteFile(0, chunkDir+file)
-					if err != nil {
-						LOG_ERROR("CHUNK_TEMPORARY", "Failed to remove the temporary file %s: %v", file, err)
+	if !success {
 		return false
-					} else {
-						LOG_DEBUG("CHUNK_TEMPORARY", "Deleted temporary file %s", file)
-					}
-					fmt.Fprintf(logFile, "Deleted temporary %s\n", file)
-				} else {
-					collection.AddTemporary(file)
-				}
-				continue
-			} else if strings.HasSuffix(file, ".fsl") {
-				// This is a fossil.  If it is unreferenced, it can be a result of failing to save the fossil
-				// collection file after making it a fossil.
-				if _, found := referencedFossils[file]; !found {
-					if dryRun {
-						LOG_INFO("FOSSIL_UNREFERENCED", "Found unreferenced fossil %s", file)
-						continue
 	}

-					chunk := strings.Replace(file, "/", "", -1)
-					chunk = strings.Replace(chunk, ".fsl", "", -1)
-
-					if _, found := referencedChunks[chunk]; found {
-						manager.resurrectChunk(chunkDir+file, chunk)
-					} else {
-						err := manager.storage.DeleteFile(0, chunkDir+file)
-						if err != nil {
-							LOG_WARN("FOSSIL_DELETE", "Failed to remove the unreferenced fossil %s: %v", file, err)
-						} else {
-							LOG_DEBUG("FOSSIL_DELETE", "Deleted unreferenced fossil %s", file)
-						}
-						fmt.Fprintf(logFile, "Deleted unreferenced fossil %s\n", file)
-					}
+	manager.chunkOperator.Stop()
+	for _, fossil := range manager.chunkOperator.fossils {
+		collection.AddFossil(fossil)
 	}

-				continue
-			}
-
-			chunk := strings.Replace(file, "/", "", -1)
-
-			if !chunkRegex.MatchString(chunk) {
-				LOG_WARN("CHUNK_UNKONWN_FILE", "File %s is not a chunk", file)
-				continue
-			}
-
-			if value, found := referencedChunks[chunk]; !found {
-
-				if dryRun {
-					LOG_INFO("CHUNK_UNREFERENCED", "Found unreferenced chunk %s", chunk)
-					continue
-				}
-
-				manager.fossilizeChunk(chunk, chunkDir+file, exclusive, collection)
-				if exclusive {
-					fmt.Fprintf(logFile, "Deleted chunk %s (exclusive mode)\n", chunk)
-				} else {
-					fmt.Fprintf(logFile, "Marked fossil %s\n", chunk)
-				}
-
-			} else if value {
-
-				// Note that the initial value is false.  So if the value is true it means another copy of the chunk
-				// exists in a higher-level directory.
-
-				if dryRun {
-					LOG_INFO("CHUNK_REDUNDANT", "Found redundant chunk %s", chunk)
-					continue
-				}
-
-				// This is a redundant chunk file (for instance D3/495A8D and D3/49/5A8D )
-				err := manager.storage.DeleteFile(0, chunkDir+file)
-				if err != nil {
-					LOG_WARN("CHUNK_DELETE", "Failed to remove the redundant chunk file %s: %v", file, err)
-				} else {
-					LOG_TRACE("CHUNK_DELETE", "Removed the redundant chunk file %s", file)
-				}
-				fmt.Fprintf(logFile, "Deleted redundant chunk %s\n", file)
-
-			} else {
-				referencedChunks[chunk] = true
-				LOG_DEBUG("CHUNK_KEEP", "Chunk %s is referenced", chunk)
-			}
-		}
-	} else {
-		// In non-exhaustive mode, only chunks that exist in the snapshots to be deleted but not other are identified
-		// as unreferenced chunks.
+	// Save the deleted revision in the fossil collection
 	for _, snapshots := range allSnapshots {
 		for _, snapshot := range snapshots {
-
-				if !snapshot.Flag {
-					continue
-				}
-
-				chunks := manager.GetSnapshotChunks(snapshot)
-
-				for _, chunk := range chunks {
-
-					if _, found := referencedChunks[chunk]; found {
-						continue
-					}
-
-					if dryRun {
-						LOG_INFO("CHUNK_UNREFERENCED", "Found unreferenced chunk %s", chunk)
-						continue
-					}
-
-					chunkPath, exist, _, err := manager.storage.FindChunk(0, chunk, false)
-					if err != nil {
-						LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", chunk, err)
-						return false
-					}
-
-					if !exist {
-						LOG_WARN("CHUNK_MISSING", "The chunk %s referenced by snapshot %s revision %d does not exist",
-							chunk, snapshot.ID, snapshot.Revision)
-						continue
-					}
-
-					manager.fossilizeChunk(chunk, chunkPath, exclusive, collection)
-					if exclusive {
-						fmt.Fprintf(logFile, "Deleted chunk %s (exclusive mode)\n", chunk)
-					} else {
-						fmt.Fprintf(logFile, "Marked fossil %s\n", chunk)
-					}
-
-					referencedChunks[chunk] = true
+			if snapshot.Flag {
+				collection.DeletedRevisions[snapshot.ID] = append(collection.DeletedRevisions[snapshot.ID], snapshot.Revision)
 			}
 		}
 	}

-	}
-
 	// Save the fossil collection if it is not empty.
-	if !collection.IsEmpty() && !dryRun {
-
+	if !collection.IsEmpty() && !dryRun && !exclusive {
 		collection.EndTime = time.Now().Unix()

 		collectionNumber := maxCollectionNumber + 1
 		collectionFile := path.Join(collectionDir, fmt.Sprintf("%d", collectionNumber))

-		description, err := json.Marshal(collection)
+		var description []byte
+		description, err = json.Marshal(collection)
 		if err != nil {
 			LOG_ERROR("FOSSIL_COLLECT", "Failed to create a json file for the fossil collection: %v", err)
 			return false
@@ -2142,21 +2015,27 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 				LOG_ERROR("SNAPSHOT_DELETE", "Failed to delete the snapshot %s at revision %d: %v",
 					snapshot.ID, snapshot.Revision, err)
 				return false
-			} else {
+			}
 			LOG_INFO("SNAPSHOT_DELETE", "The snapshot %s at revision %d has been removed",
 				snapshot.ID, snapshot.Revision)
-			}
-			manager.snapshotCache.DeleteFile(0, snapshotPath)
+			err = manager.snapshotCache.DeleteFile(0, snapshotPath)
+			if err != nil {
+				LOG_WARN("SNAPSHOT_DELETE", "The cached snapshot %s at revision %d could not be removed: %v",
+					snapshot.ID, snapshot.Revision, err)
+				fmt.Fprintf(logFile, "Cached snapshot %s at revision %d could not be removed: %v",
+					snapshot.ID, snapshot.Revision, err)
+			} else {
 				fmt.Fprintf(logFile, "Deleted cached snapshot %s at revision %d\n", snapshot.ID, snapshot.Revision)
 			}
 		}
+	}

 	if collection.IsEmpty() && !dryRun && toBeDeleted != 0 && !exclusive {
 		LOG_INFO("FOSSIL_NONE",
 			"No fossil collection has been created since deleted snapshots did not reference any unique chunks")
 	}

-	var latestSnapshot *Snapshot = nil
+	var latestSnapshot *Snapshot
 	if len(allSnapshots[selfID]) > 0 {
 		latestSnapshot = allSnapshots[selfID][len(allSnapshots[selfID])-1]
 	}
@@ -2170,6 +2049,211 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 	return true
 }

+// pruneSnapshots in non-exhaustive mode, only chunks that exist in the
+// snapshots to be deleted but not other are identified as unreferenced chunks.
+func (manager *SnapshotManager) pruneSnapshotsNonExhaustive(allSnapshots map[string][]*Snapshot, collection *FossilCollection, logFile io.Writer, dryRun, exclusive bool) bool {
+	targetChunks := make(map[string]bool)
+
+	// Now build all chunks referened by snapshot not deleted
+	for _, snapshots := range allSnapshots {
+
+		if len(snapshots) > 0 {
+			latest := snapshots[len(snapshots)-1]
+			if latest.Flag && !exclusive {
+				LOG_ERROR("SNAPSHOT_DELETE",
+					"The latest snapshot %s at revision %d can't be deleted in non-exclusive mode",
+					latest.ID, latest.Revision)
+				return false
+			}
+		}
+
+		for _, snapshot := range snapshots {
+			if !snapshot.Flag {
+				continue
+			}
+
+			LOG_INFO("SNAPSHOT_DELETE", "Deleting snapshot %s at revision %d", snapshot.ID, snapshot.Revision)
+			chunks := manager.GetSnapshotChunks(snapshot, false)
+
+			for _, chunk := range chunks {
+				// The initial value is 'false'.  When a referenced chunk is found it will change the value to 'true'.
+				targetChunks[chunk] = false
+			}
+		}
+	}
+
+	for _, snapshots := range allSnapshots {
+		for _, snapshot := range snapshots {
+			if snapshot.Flag {
+				continue
+			}
+
+			chunks := manager.GetSnapshotChunks(snapshot, false)
+
+			for _, chunk := range chunks {
+				if _, found := targetChunks[chunk]; found {
+					targetChunks[chunk] = true
+				}
+			}
+		}
+	}
+
+	for chunk, value := range targetChunks {
+		if value {
+			continue
+		}
+
+		if dryRun {
+			LOG_INFO("CHUNK_UNREFERENCED", "Found unreferenced chunk %s", chunk)
+			continue
+		}
+
+		manager.fossilizeChunk(chunk, "", exclusive)
+		if exclusive {
+			fmt.Fprintf(logFile, "Deleted chunk %s (exclusive mode)\n", chunk)
+		} else {
+			fmt.Fprintf(logFile, "Marked fossil %s\n", chunk)
+		}
+
+		targetChunks[chunk] = true
+	}
+
+	return true
+}
+
+// pruneSnapshotsExhaustive in exhaustive, we scan the entire chunk tree to
+// find dangling chunks and temporaries.
+func (manager *SnapshotManager) pruneSnapshotsExhaustive(referencedFossils map[string]bool, allSnapshots map[string][]*Snapshot, collection *FossilCollection, logFile io.Writer, dryRun, exclusive bool) bool {
+	chunkRegex := regexp.MustCompile(`^[0-9a-f]+$`)
+	referencedChunks := make(map[string]bool)
+
+	// Now build all chunks referened by snapshot not deleted
+	for _, snapshots := range allSnapshots {
+		if len(snapshots) > 0 {
+			latest := snapshots[len(snapshots)-1]
+			if latest.Flag && !exclusive {
+				LOG_ERROR("SNAPSHOT_DELETE",
+					"The latest snapshot %s at revision %d can't be deleted in non-exclusive mode",
+					latest.ID, latest.Revision)
+				return false
+			}
+		}
+
+		for _, snapshot := range snapshots {
+			if snapshot.Flag {
+				LOG_INFO("SNAPSHOT_DELETE", "Deleting snapshot %s at revision %d", snapshot.ID, snapshot.Revision)
+				continue
+			}
+
+			chunks := manager.GetSnapshotChunks(snapshot, false)
+
+			for _, chunk := range chunks {
+				// The initial value is 'false'.  When a referenced chunk is found it will change the value to 'true'.
+				referencedChunks[chunk] = false
+			}
+		}
+	}
+
+	allFiles, _ := manager.ListAllFiles(manager.storage, chunkDir)
+	for _, file := range allFiles {
+		if file[len(file)-1] == '/' {
+			continue
+		}
+
+		if strings.HasSuffix(file, ".tmp") {
+			// This is a temporary chunk file.  It can be a result of a restore operation still in progress, or
+			// a left-over from a restore operation that was terminated abruptly.
+			if dryRun {
+				LOG_INFO("CHUNK_TEMPORARY", "Found temporary file %s", file)
+				continue
+			}
+
+			if exclusive {
+				// In exclusive mode, we assume no other restore operation is running concurrently.
+				manager.chunkOperator.Delete("", chunkDir+file)
+				fmt.Fprintf(logFile, "Deleted temporary %s\n", file)
+			} else {
+				collection.AddTemporary(file)
+			}
+			continue
+		} else if strings.HasSuffix(file, ".fsl") {
+			// This is a fossil.  If it is unreferenced, it can be a result of failing to save the fossil
+			// collection file after making it a fossil.
+			if _, found := referencedFossils[file]; !found {
+
+				chunk := strings.Replace(file, "/", "", -1)
+				chunk = strings.Replace(chunk, ".fsl", "", -1)
+
+				if _, found := referencedChunks[chunk]; found {
+
+					if dryRun {
+						LOG_INFO("FOSSIL_REFERENCED", "Found referenced fossil %s", file)
+						continue
+					}
+
+					manager.chunkOperator.Resurrect(chunk, chunkDir + file)
+					fmt.Fprintf(logFile, "Found referenced fossil %s\n", file)
+
+				} else {
+
+					if dryRun {
+						LOG_INFO("FOSSIL_UNREFERENCED", "Found unreferenced fossil %s", file)
+						continue
+					}
+
+					if exclusive {
+						manager.chunkOperator.Delete(chunk, chunkDir + file)
+					} else {
+						collection.AddFossil(chunkDir + file)
+						LOG_DEBUG("FOSSIL_FIND", "Found unreferenced fossil %s", file)
+					}
+					fmt.Fprintf(logFile, "Found unreferenced fossil %s\n", file)
+				}
+			}
+
+			continue
+		}
+
+		chunk := strings.Replace(file, "/", "", -1)
+
+		if !chunkRegex.MatchString(chunk) {
+			LOG_WARN("CHUNK_UNKONWN_FILE", "File %s is not a chunk", file)
+			continue
+		}
+
+		if value, found := referencedChunks[chunk]; !found {
+			if dryRun {
+				LOG_INFO("CHUNK_UNREFERENCED", "Found unreferenced chunk %s", chunk)
+				continue
+			}
+
+			manager.fossilizeChunk(chunk, chunkDir+file, exclusive)
+			if exclusive {
+				fmt.Fprintf(logFile, "Deleted chunk %s (exclusive mode)\n", chunk)
+			} else {
+				fmt.Fprintf(logFile, "Marked fossil %s\n", chunk)
+			}
+		} else if value {
+			// Note that the initial value is false.  So if the value is true it means another copy of the chunk
+			// exists in a higher-level directory.
+
+			if dryRun {
+				LOG_INFO("CHUNK_REDUNDANT", "Found redundant chunk %s", chunk)
+				continue
+			}
+
+			// This is a redundant chunk file (for instance D3/495A8D and D3/49/5A8D )
+			manager.chunkOperator.Delete(chunk, chunkDir+file)
+			fmt.Fprintf(logFile, "Deleted redundant chunk %s\n", file)
+		} else {
+			referencedChunks[chunk] = true
+			LOG_DEBUG("CHUNK_KEEP", "Chunk %s is referenced", chunk)
+		}
+	}
+
+	return true
+}
+
 // CheckSnapshot performs sanity checks on the given snapshot.
 func (manager *SnapshotManager) CheckSnapshot(snapshot *Snapshot) (err error) {

--- a/src/duplicacy_snapshotmanager_test.go
+++ b/src/duplicacy_snapshotmanager_test.go
@@ -14,6 +14,7 @@ import (
 	"strings"
 	"testing"
 	"time"
+	"io/ioutil"
 )

 func createDummySnapshot(snapshotID string, revision int, endTime int64) *Snapshot {
@@ -143,6 +144,15 @@ func uploadRandomChunk(manager *SnapshotManager, chunkSize int) string {
 	return uploadTestChunk(manager, content)
 }

+func uploadRandomChunks(manager *SnapshotManager, chunkSize int, numberOfChunks int) []string {
+	chunkList := make([]string, 0)
+	for i := 0; i < numberOfChunks; i++ {
+		chunkHash := uploadRandomChunk(manager, chunkSize)
+		chunkList = append(chunkList, chunkHash)
+	}
+	return chunkList
+}
+
 func createTestSnapshot(manager *SnapshotManager, snapshotID string, revision int, startTime int64, endTime int64, chunkHashes []string, tag string) {

 	snapshot := &Snapshot{
@@ -203,7 +213,7 @@ func checkTestSnapshots(manager *SnapshotManager, expectedSnapshots int, expecte
 			snapshot := manager.DownloadSnapshot(snapshotID, revision)
 			numberOfSnapshots++

-			for _, chunk := range manager.GetSnapshotChunks(snapshot) {
+			for _, chunk := range manager.GetSnapshotChunks(snapshot, false) {
 				chunks[chunk] = true
 			}
 		}
@@ -226,7 +236,7 @@ func checkTestSnapshots(manager *SnapshotManager, expectedSnapshots int, expecte
 	}
 }

-func TestSingleRepositoryPrune(t *testing.T) {
+func TestPruneSingleRepository(t *testing.T) {

 	setTestingT(t)

@@ -242,34 +252,35 @@ func TestSingleRepositoryPrune(t *testing.T) {

 	now := time.Now().Unix()
 	day := int64(24 * 3600)
-	t.Logf("Creating 1 snapshot")
-	createTestSnapshot(snapshotManager, "repository1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
-	checkTestSnapshots(snapshotManager, 1, 2)
+	t.Logf("Creating 2 snapshots")
+	createTestSnapshot(snapshotManager, "repository1", 1, now-4*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "repository1", 2, now-4*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Creating 2 snapshots")
-	createTestSnapshot(snapshotManager, "repository1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
-	createTestSnapshot(snapshotManager, "repository1", 3, now-1*day-3600, now-1*day-60, []string{chunkHash3, chunkHash4}, "tag")
-	checkTestSnapshots(snapshotManager, 3, 0)
+	createTestSnapshot(snapshotManager, "repository1", 3, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	createTestSnapshot(snapshotManager, "repository1", 4, now-1*day-3600, now-1*day-60, []string{chunkHash3, chunkHash4}, "tag")
+	checkTestSnapshots(snapshotManager, 4, 0)

-	t.Logf("Removing snapshot repository1 revision 1 with --exclusive")
-	snapshotManager.PruneSnapshots("repository1", "repository1", []int{1}, []string{}, []string{}, false, true, []string{}, false, false, false)
+	t.Logf("Removing snapshot repository1 revisions 1 and 2 with --exclusive")
+	snapshotManager.PruneSnapshots("repository1", "repository1", []int{1, 2}, []string{}, []string{}, false, true, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 0)

-	t.Logf("Removing snapshot repository1 revision 2 without --exclusive")
-	snapshotManager.PruneSnapshots("repository1", "repository1", []int{2}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	t.Logf("Removing snapshot repository1 revision 3 without --exclusive")
+	snapshotManager.PruneSnapshots("repository1", "repository1", []int{3}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 1, 2)

 	t.Logf("Creating 1 snapshot")
 	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
-	createTestSnapshot(snapshotManager, "repository1", 4, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash5}, "tag")
+	createTestSnapshot(snapshotManager, "repository1", 5, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash5}, "tag")
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-	snapshotManager.PruneSnapshots("repository1", "repository1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("repository1", "repository1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 0)
 }

-func TestSingleHostPrune(t *testing.T) {
+func TestPruneSingleHost(t *testing.T) {

 	setTestingT(t)

@@ -292,11 +303,11 @@ func TestSingleHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 3, 0)

 	t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Creating 1 snapshot")
@@ -305,12 +316,12 @@ func TestSingleHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 3, 2)

 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 3, 0)

 }

-func TestMultipleHostPrune(t *testing.T) {
+func TestPruneMultipleHost(t *testing.T) {

 	setTestingT(t)

@@ -333,11 +344,11 @@ func TestMultipleHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 3, 0)

 	t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Creating 1 snapshot")
@@ -346,7 +357,7 @@ func TestMultipleHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 3, 2)

 	t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 3, 2)

 	t.Logf("Creating 1 snapshot")
@@ -355,7 +366,7 @@ func TestMultipleHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 4, 2)

 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 4, 0)
 }

@@ -380,7 +391,7 @@ func TestPruneAndResurrect(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 2, 0)

 	t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 1, 2)

 	t.Logf("Creating 1 snapshot")
@@ -389,11 +400,11 @@ func TestPruneAndResurrect(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Prune without removing any snapshots -- one fossil will be resurrected")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 0)
 }

-func TestInactiveHostPrune(t *testing.T) {
+func TestPruneWithInactiveHost(t *testing.T) {

 	setTestingT(t)

@@ -417,11 +428,11 @@ func TestInactiveHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 3, 0)

 	t.Logf("Removing snapshot vm1@host1 revision 1")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 2, 2)

 	t.Logf("Creating 1 snapshot")
@@ -430,11 +441,11 @@ func TestInactiveHostPrune(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 3, 2)

 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 3, 0)
 }

-func TestRetentionPolicy(t *testing.T) {
+func TestPruneWithRetentionPolicy(t *testing.T) {

 	setTestingT(t)

@@ -458,19 +469,19 @@ func TestRetentionPolicy(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 30, 0)

 	t.Logf("Removing snapshot vm1@host1 0:20 with --exclusive")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 19, 0)

 	t.Logf("Removing snapshot vm1@host1 -k 0:20 with --exclusive")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 19, 0)

 	t.Logf("Removing snapshot vm1@host1 -k 3:14 -k 2:7 with --exclusive")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"3:14", "2:7"}, false, true, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"3:14", "2:7"}, false, true, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 12, 0)
 }

-func TestRetentionPolicyAndTag(t *testing.T) {
+func TestPruneWithRetentionPolicyAndTag(t *testing.T) {

 	setTestingT(t)

@@ -498,6 +509,179 @@ func TestRetentionPolicyAndTag(t *testing.T) {
 	checkTestSnapshots(snapshotManager, 30, 0)

 	t.Logf("Removing snapshot vm1@host1 0:20 with --exclusive and --tag manual")
-	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{"manual"}, []string{"0:7"}, false, true, []string{}, false, false, false)
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{"manual"}, []string{"0:7"}, false, true, []string{}, false, false, false, 1)
 	checkTestSnapshots(snapshotManager, 22, 0)
 }
+
+// Test that an unreferenced fossil shouldn't be removed as it may be the result of another prune job in-progress.
+func TestPruneWithFossils(t *testing.T) {
+	setTestingT(t)
+
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "snapshot_test")
+
+	snapshotManager := createTestSnapshotManager(testDir)
+
+	chunkSize := 1024
+	chunkHash1 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash2 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash3 := uploadRandomChunk(snapshotManager, chunkSize)
+	// Create an unreferenced fossil
+	snapshotManager.storage.UploadFile(0, "chunks/113b6a2350dcfd836829c47304dd330fa6b58b93dd7ac696c6b7b913e6868662.fsl", []byte("this is a test fossil"))
+
+	now := time.Now().Unix()
+	day := int64(24 * 3600)
+	t.Logf("Creating 2 snapshots")
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	checkTestSnapshots(snapshotManager, 2, 1)
+
+	t.Logf("Prune without removing any snapshots but with --exhaustive")
+	// The unreferenced fossil shouldn't be removed
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, true, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 2, 1)
+
+	t.Logf("Prune without removing any snapshots but with --exclusive")
+	// Now the unreferenced fossil should be removed
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, true, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 2, 0)
+}
+
+func TestPruneMultipleThread(t *testing.T) {
+
+	setTestingT(t)
+
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "snapshot_test")
+
+	snapshotManager := createTestSnapshotManager(testDir)
+
+	chunkSize := 1024
+	numberOfChunks := 256
+	numberOfThreads := 4
+
+	chunkList1 := uploadRandomChunks(snapshotManager, chunkSize, numberOfChunks)
+	chunkList2 := uploadRandomChunks(snapshotManager, chunkSize, numberOfChunks)
+
+	now := time.Now().Unix()
+	day := int64(24 * 3600)
+	t.Logf("Creating 2 snapshots")
+	createTestSnapshot(snapshotManager, "repository1", 1, now-4*day-3600, now-3*day-60, chunkList1, "tag")
+	createTestSnapshot(snapshotManager, "repository1", 2, now-3*day-3600, now-2*day-60, chunkList2, "tag")
+	checkTestSnapshots(snapshotManager, 2, 0)
+
+	t.Logf("Removing snapshot revisions 1 with --exclusive")
+	snapshotManager.PruneSnapshots("repository1", "repository1", []int{1}, []string{}, []string{}, false, true, []string{}, false, false, false, numberOfThreads)
+	checkTestSnapshots(snapshotManager, 1, 0)
+
+	t.Logf("Creating 1 more snapshot")
+	chunkList3 := uploadRandomChunks(snapshotManager, chunkSize, numberOfChunks)
+	createTestSnapshot(snapshotManager, "repository1", 3, now-2*day-3600, now-1*day-60, chunkList3, "tag")
+
+	t.Logf("Removing snapshot repository1 revision 2 without --exclusive")
+	snapshotManager.PruneSnapshots("repository1", "repository1", []int{2}, []string{}, []string{}, false, false, []string{}, false, false, false, numberOfThreads)
+
+	t.Logf("Prune without removing any snapshots but with --exclusive")
+	snapshotManager.PruneSnapshots("repository1", "repository1", []int{}, []string{}, []string{}, false, true, []string{}, false, false, false, numberOfThreads)
+	checkTestSnapshots(snapshotManager, 1, 0)
+}
+
+// A snapshot not seen by a fossil collection should always be consider a new snapshot in the fossil deletion step
+func TestPruneNewSnapshots(t *testing.T) {
+	setTestingT(t)
+
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "snapshot_test")
+
+	snapshotManager := createTestSnapshotManager(testDir)
+
+	chunkSize := 1024
+	chunkHash1 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash2 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash3 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash4 := uploadRandomChunk(snapshotManager, chunkSize)
+
+	now := time.Now().Unix()
+	day := int64(24 * 3600)
+	t.Logf("Creating 3 snapshots")
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	createTestSnapshot(snapshotManager, "vm2@host1", 1, now-2*day-3600, now-2*day-60, []string{chunkHash3, chunkHash4}, "tag")
+	checkTestSnapshots(snapshotManager, 3, 0)
+
+	t.Logf("Prune snapshot 1")
+	// chunkHash1 should be marked as fossil
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 2, 2)
+
+	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
+	// Create another snapshot of vm1 that brings back chunkHash1
+	createTestSnapshot(snapshotManager, "vm1@host1", 3, now-0*day-3600, now-0*day-60, []string{chunkHash1, chunkHash3}, "tag")
+	// Create another snapshot of vm2 so the fossil collection will be processed by next prune
+	createTestSnapshot(snapshotManager, "vm2@host1", 2, now + 3600, now + 3600 * 2, []string{chunkHash4, chunkHash5}, "tag")
+
+	// Now chunkHash1 wil be resurrected
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 4, 0)
+	snapshotManager.CheckSnapshots("vm1@host1", []int{2, 3}, "", false, false, false, false, false);
+}
+
+// A fossil collection left by an aborted prune should be ignored if any supposedly deleted snapshot exists
+func TestPruneGhostSnapshots(t *testing.T) {
+	setTestingT(t)
+
+	EnableStackTrace()
+
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "snapshot_test")
+
+	snapshotManager := createTestSnapshotManager(testDir)
+
+	chunkSize := 1024
+	chunkHash1 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash2 := uploadRandomChunk(snapshotManager, chunkSize)
+	chunkHash3 := uploadRandomChunk(snapshotManager, chunkSize)
+
+	now := time.Now().Unix()
+	day := int64(24 * 3600)
+	t.Logf("Creating 2 snapshots")
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	checkTestSnapshots(snapshotManager, 2, 0)
+
+	snapshot1, err := ioutil.ReadFile(path.Join(testDir, "snapshots", "vm1@host1", "1"))
+	if err != nil {
+		t.Errorf("Failed to read snapshot file: %v", err)
+	}
+
+	t.Logf("Prune snapshot 1")
+	// chunkHash1 should be marked as fossil
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 1, 2)
+
+	// Recover the snapshot file for revision 1; this is to simulate a scenario where prune may encounter a network error after
+	// leaving the fossil collection but before deleting any snapshots.
+	err = ioutil.WriteFile(path.Join(testDir, "snapshots", "vm1@host1", "1"), snapshot1, 0644)
+	if err != nil {
+		t.Errorf("Failed to write snapshot file: %v", err)
+	}
+
+	// Create another snapshot of vm1 so the fossil collection becomes eligible for processing.
+	chunkHash4 := uploadRandomChunk(snapshotManager, chunkSize)
+	createTestSnapshot(snapshotManager, "vm1@host1", 3, now - day - 3600, now - day - 60, []string{chunkHash3, chunkHash4}, "tag")
+
+	// Run the prune again but the fossil collection should be igored, since revision 1 still exists
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 3, 2)
+	snapshotManager.CheckSnapshots("vm1@host1", []int{1, 2, 3}, "", false, false, false, true /*searchFossils*/, false);
+
+	// Prune snapshot 1 again
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 2, 2)
+
+	// Create another snapshot
+	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
+	createTestSnapshot(snapshotManager, "vm1@host1", 4, now + 3600, now + 3600 * 2, []string{chunkHash5, chunkHash5}, "tag")
+	checkTestSnapshots(snapshotManager, 3, 2)
+
+	// Run the prune again and this time the fossil collection will be processed and the fossils removed
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false, 1)
+	checkTestSnapshots(snapshotManager, 3, 0)
+	snapshotManager.CheckSnapshots("vm1@host1", []int{2, 3, 4}, "", false, false, false, false, false);
+}
--- a/src/duplicacy_storage.go
+++ b/src/duplicacy_storage.go
@@ -261,7 +261,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		return fileStorage
 	}

-	urlRegex := regexp.MustCompile(`^(\w+)://([\w\-]+@)?([^/]+)(/(.+))?`)
+	urlRegex := regexp.MustCompile(`^([\w-]+)://([\w\-@\.]+@)?([^/]+)(/(.+))?`)

 	matched := urlRegex.FindStringSubmatch(storageURL)

@@ -461,6 +461,42 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		SavePassword(preference, "s3_secret", secretKey)

 		return storage
+
+	} else if matched[1] == "wasabi" {
+
+		region := matched[2]
+		endpoint := matched[3]
+		bucket := matched[5]
+
+		if region != "" {
+			region = region[:len(region)-1]
+		}
+
+		key := GetPassword(preference, "wasabi_key",
+			"Enter Wasabi key:", true, resetPassword)
+		secret := GetPassword(preference, "wasabi_secret",
+			"Enter Wasabi secret:", true, resetPassword)
+
+		storageDir := ""
+		if strings.Contains(bucket, "/") {
+			firstSlash := strings.Index(bucket, "/")
+			storageDir = bucket[firstSlash+1:]
+			bucket = bucket[:firstSlash]
+		}
+
+		storage, err := CreateWasabiStorage(region, endpoint,
+			bucket, storageDir, key, secret, threads)
+
+		if err != nil {
+			LOG_ERROR("STORAGE_CREATE", "Failed to load the Wasabi storage at %s: %v", storageURL, err)
+			return nil
+		}
+
+		SavePassword(preference, "wasabi_key", key)
+		SavePassword(preference, "wasabi_secret", secret)
+
+		return storage
+
 	} else if matched[1] == "dropbox" {
 		storageDir := matched[3] + matched[5]
 		token := GetPassword(preference, "dropbox_token", "Enter Dropbox access token:", true, resetPassword)
@@ -570,6 +606,29 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		}
 		SavePassword(preference, "swift_key", key)
 		return swiftStorage
+	} else if matched[1] == "webdav" || matched[1] == "webdav-http" {
+		server := matched[3]
+		username := matched[2]
+		username = username[:len(username) - 1]
+		storageDir := matched[5]
+		port := 0
+		useHTTP := matched[1] == "webdav-http"
+
+		if strings.Contains(server, ":") {
+			index := strings.Index(server, ":")
+			port, _ = strconv.Atoi(server[index+1:])
+			server = server[:index]
+		}
+
+		prompt := fmt.Sprintf("Enter the WebDAV password:")
+		password := GetPassword(preference, "webdav_password", prompt, true, resetPassword)
+		webDAVStorage, err := CreateWebDAVStorage(server, port, username, password, storageDir, useHTTP, threads)
+		if err != nil {
+			LOG_ERROR("STORAGE_CREATE", "Failed to load the WebDAV storage at %s: %v", storageURL, err)
+			return nil
+		}
+		SavePassword(preference, "webdav_password", password)
+		return webDAVStorage
 	} else {
 		LOG_ERROR("STORAGE_CREATE", "The storage type '%s' is not supported", matched[1])
 		return nil
--- a/src/duplicacy_storage_test.go
+++ b/src/duplicacy_storage_test.go
@@ -78,10 +78,14 @@ func loadStorage(localStoragePath string, threads int) (Storage, error) {
 		storage, err := CreateSFTPStorageWithPassword(config["server"], port, config["username"], config["directory"], 2, config["password"], threads)
 		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
 		return storage, err
-	} else if testStorageName == "s3" || testStorageName == "wasabi" {
+	} else if testStorageName == "s3" {
 		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, false)
-		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
 		return storage, err
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+	} else if testStorageName == "wasabi" {
+		storage, err := CreateWasabiStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
+		return storage, err
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
 	} else if testStorageName == "s3c" {
 		storage, err := CreateS3CStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
 		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
@@ -142,9 +146,18 @@ func loadStorage(localStoragePath string, threads int) (Storage, error) {
 		storage, err := CreateSwiftStorage(config["storage_url"], config["key"], threads)
 		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
 		return storage, err
+	} else if testStorageName == "pcloud" || testStorageName == "box" {
+		storage, err := CreateWebDAVStorage(config["host"], 0, config["username"], config["password"], config["storage_path"], false, threads)
+		if err != nil {
+			return nil, err
+		}
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else {
 		return nil, fmt.Errorf("Invalid storage named: %s", testStorageName)
 	}
+
+	return nil, fmt.Errorf("Invalid storage named: %s", testStorageName)
 }

 func cleanStorage(storage Storage) {
@@ -374,6 +387,7 @@ func TestStorage(t *testing.T) {

 	snapshotIDs := []string{}
 	for _, snapshotDir := range snapshotDirs {
+		LOG_INFO("debug", "snapshot dir: %s", snapshotDir)
 		if len(snapshotDir) > 0 && snapshotDir[len(snapshotDir)-1] == '/' {
 			snapshotIDs = append(snapshotIDs, snapshotDir[:len(snapshotDir)-1])
 		}
@@ -500,7 +514,7 @@ func TestStorage(t *testing.T) {
 	} else {
 		err = storage.DeleteFile(0, filePath)
 		if err != nil {
-			t.Errorf("Failed to delete file %s: %v", filePath)
+			t.Errorf("Failed to delete file %s: %v", filePath, err)
 		} else {
 			time.Sleep(time.Duration(delay) * time.Second)
 			filePath, exist, _, err = storage.FindChunk(0, chunks[1], true)
@@ -587,7 +601,6 @@ func TestCleanStorage(t *testing.T) {
 	storage.DeleteFile(0, "config")
 	LOG_INFO("DELETE_FILE", "Deleted config")

-
 	files, _, err := storage.ListFiles(0, "chunks/")
 	for _, file := range files {
 		if len(file) > 0 && file[len(file)-1] != '/' {
--- a/src/duplicacy_utils.go
+++ b/src/duplicacy_utils.go
@@ -344,11 +344,13 @@ func MatchPath(filePath string, patterns []string) (included bool) {
 	for _, pattern := range patterns {
 		if pattern[0] == '+' {
 			if matchPattern(filePath, pattern[1:]) {
+				LOG_DEBUG("PATTERN_INCLUDE", "%s is included by pattern %s", filePath, pattern)
 				return true
 			}
 		} else if pattern[0] == '-' {
 			allIncludes = false
 			if matchPattern(filePath, pattern[1:]) {
+				LOG_DEBUG("PATTERN_EXCLUDE", "%s is excluded by pattern %s", filePath, pattern)
 				return false
 			}
 		} else if strings.HasPrefix(pattern, "i:") || strings.HasPrefix(pattern, "e:") {
@@ -363,7 +365,14 @@ func MatchPath(filePath string, patterns []string) (included bool) {
 				matched = re.MatchString(filePath)
 			}
 			if matched {
-				return strings.HasPrefix(pattern, "i:")
+				if strings.HasPrefix(pattern, "i:") {
+					LOG_DEBUG("PATTERN_INCLUDE", "%s is included by pattern %s", filePath, pattern)
+					return true
+				} else {
+					LOG_DEBUG("PATTERN_EXCLUDE", "%s is excluded by pattern %s", filePath, pattern)
+					return false
+
+				}
 			} else {
 				if strings.HasPrefix(pattern, "e:") {
 					allIncludes = false
@@ -372,7 +381,13 @@ func MatchPath(filePath string, patterns []string) (included bool) {
 		}
 	}

-	return !allIncludes
+	if allIncludes {
+		LOG_DEBUG("PATTERN_EXCLUDE", "%s is excluded", filePath)
+		return false
+	} else {
+		LOG_DEBUG("PATTERN_INCLUDE", "%s is included", filePath)
+		return true
+	}
 }

 func joinPath(components ...string) string {
--- a/src/duplicacy_utils_test.go
+++ b/src/duplicacy_utils_test.go
@@ -112,7 +112,7 @@ func TestRateLimit(t *testing.T) {
 		return
 	}
 	if int(n) != len(content) {
-		t.Errorf("Wrote %s bytes instead of %s", n, len(content))
+		t.Errorf("Wrote %d bytes instead of %d", n, len(content))
 		return
 	}

@@ -127,7 +127,7 @@ func TestRateLimit(t *testing.T) {
 		return
 	}
 	if int(n) != len(content) {
-		t.Errorf("Copied %s bytes instead of %s", n, len(content))
+		t.Errorf("Copied %d bytes instead of %d", n, len(content))
 		return
 	}

--- a/src/duplicacy_wasabistorage.go
+++ b/src/duplicacy_wasabistorage.go
@@ -0,0 +1,192 @@
+//
+// Storage module for Wasabi (https://www.wasabi.com)
+//
+
+// Wasabi is nominally compatible with AWS S3, but the copy-and-delete
+// method used for renaming objects creates additional expense under
+// Wasabi's billing system.  This module is a pass-through to the
+// existing S3 module for everything other than that one operation.
+//
+// This module copyright 2017 Mark Feit (https://github.com/markfeit)
+// and may be distributed under the same terms as Duplicacy.
+
+package duplicacy
+
+import (
+	"crypto/hmac"
+	"crypto/sha1"
+	"encoding/base64"
+	"errors"
+	"fmt"
+	"net/http"
+	"time"
+)
+
+type WasabiStorage struct {
+	StorageBase
+
+	s3         *S3Storage
+	region     string
+	endpoint   string
+	bucket     string
+	storageDir string
+	key        string
+	secret     string
+	client     *http.Client
+}
+
+// See the Storage interface in duplicacy_storage.go for function
+// descriptions.
+
+func CreateWasabiStorage(
+	regionName string, endpoint string,
+	bucketName string, storageDir string,
+	accessKey string, secretKey string,
+	threads int,
+) (storage *WasabiStorage, err error) {
+
+	s3storage, error := CreateS3Storage(regionName, endpoint, bucketName,
+		storageDir, accessKey, secretKey, threads,
+		true,  // isSSLSupported
+		false, // isMinioCompatible
+	)
+
+	if err != nil {
+		return nil, error
+	}
+
+	wasabi := &WasabiStorage{
+
+		// Pass-through to existing S3 module
+		s3: s3storage,
+
+		// Local copies required for renaming
+		region:     regionName,
+		endpoint:   endpoint,
+		bucket:     bucketName,
+		storageDir: storageDir,
+		key:        accessKey,
+		secret:     secretKey,
+		client:     &http.Client{},
+	}
+
+	wasabi.DerivedStorage = wasabi
+	wasabi.SetDefaultNestingLevels([]int{0}, 0)
+
+	return wasabi, nil
+}
+
+func (storage *WasabiStorage) ListFiles(
+	threadIndex int, dir string,
+) (files []string, sizes []int64, err error) {
+	return storage.s3.ListFiles(threadIndex, dir)
+}
+
+func (storage *WasabiStorage) DeleteFile(
+	threadIndex int, filePath string,
+) (err error) {
+	return storage.s3.DeleteFile(threadIndex, filePath)
+
+}
+
+// This is a lightweight implementation of a call to Wasabi for a
+// rename.  It's designed to get the job done with as few dependencies
+// on other packages as possible rather than being somethng
+// general-purpose and reusable.
+func (storage *WasabiStorage) MoveFile(
+	threadIndex int, from string, to string,
+) (err error) {
+
+	var from_path string
+	// The from path includes the bucket.  Take care not to include an empty storageDir
+	// string as Wasabi's backend will return 404 on URLs with double slashes.
+	if (storage.storageDir == "") {
+		from_path = fmt.Sprintf("/%s/%s", storage.bucket, from)
+	} else {
+		from_path = fmt.Sprintf("/%s/%s/%s", storage.bucket, storage.storageDir, from)
+	}
+
+	object := fmt.Sprintf("https://%s@%s%s",
+		storage.region, storage.endpoint, from_path)
+
+	// The object's new name is relative to the top of the bucket.
+	new_name := fmt.Sprintf("%s/%s", storage.storageDir, to)
+
+	timestamp := time.Now().Format(time.RFC1123Z)
+
+	signing_string := fmt.Sprintf("MOVE\n\n\n%s\n%s", timestamp, from_path)
+
+	signer := hmac.New(sha1.New, []byte(storage.secret))
+	signer.Write([]byte(signing_string))
+
+	signature := base64.StdEncoding.EncodeToString(signer.Sum(nil))
+
+	authorization := fmt.Sprintf("AWS %s:%s", storage.key, signature)
+
+	request, error := http.NewRequest("MOVE", object, nil)
+	if error != nil {
+		return error
+	}
+	request.Header.Add("Authorization", authorization)
+	request.Header.Add("Date", timestamp)
+	request.Header.Add("Destination", new_name)
+	request.Header.Add("Host", storage.endpoint)
+	request.Header.Add("Overwrite", "true")
+
+	response, error := storage.client.Do(request)
+	if error != nil {
+		return error
+	}
+	defer response.Body.Close()
+
+	if response.StatusCode != 200 {
+		return errors.New(response.Status)
+	}
+
+	return nil
+}
+
+func (storage *WasabiStorage) CreateDirectory(
+	threadIndex int, dir string,
+) (err error) {
+	return storage.s3.CreateDirectory(threadIndex, dir)
+}
+
+func (storage *WasabiStorage) GetFileInfo(
+	threadIndex int, filePath string,
+) (exist bool, isDir bool, size int64, err error) {
+	return storage.s3.GetFileInfo(threadIndex, filePath)
+}
+
+func (storage *WasabiStorage) DownloadFile(
+	threadIndex int, filePath string, chunk *Chunk,
+) (err error) {
+	return storage.s3.DownloadFile(threadIndex, filePath, chunk)
+}
+
+func (storage *WasabiStorage) UploadFile(
+	threadIndex int, filePath string, content []byte,
+) (err error) {
+	return storage.s3.UploadFile(threadIndex, filePath, content)
+}
+
+func (storage *WasabiStorage) IsCacheNeeded() bool {
+	return storage.s3.IsCacheNeeded()
+}
+
+func (storage *WasabiStorage) IsMoveFileImplemented() bool {
+	// This is implemented locally since S3 does a copy and delete
+	return true
+}
+
+func (storage *WasabiStorage) IsStrongConsistent() bool {
+	// Wasabi has it, S3 doesn't.
+	return true
+}
+
+func (storage *WasabiStorage) IsFastListing() bool {
+	return storage.s3.IsFastListing()
+}
+
+func (storage *WasabiStorage) EnableTestMode() {
+}
--- a/src/duplicacy_webdavstorage.go
+++ b/src/duplicacy_webdavstorage.go
@@ -0,0 +1,450 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+//
+//
+// This storage backend is based on the work by Yuri Karamani from https://github.com/karamani/webdavclnt,
+// released under the MIT license.
+//
+package duplicacy
+
+import (
+	"bytes"
+	"encoding/xml"
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"net/http"
+	//"net/http/httputil"
+	"strconv"
+	"sync"
+	"time"
+	"strings"
+)
+
+type WebDAVStorage struct {
+	StorageBase
+
+	host       string
+	port       int
+	username   string
+	password   string
+	storageDir string
+	useHTTP    bool
+
+	client             *http.Client
+	threads            int
+	directoryCache     map[string]int // stores directories known to exist by this backend
+	directoryCacheLock sync.Mutex     // lock for accessing directoryCache
+}
+
+var (
+	errWebDAVAuthorizationFailure = errors.New("Authentication failed")
+	errWebDAVMovedPermanently = errors.New("Moved permanently")
+	errWebDAVNotExist = errors.New("Path does not exist")
+	errWebDAVMaximumBackoff = errors.New("Maximum backoff reached")
+	errWebDAVMethodNotAllowed = errors.New("Method not allowed")
+)
+
+func CreateWebDAVStorage(host string, port int, username string, password string, storageDir string, useHTTP bool, threads int) (storage *WebDAVStorage, err error) {
+	if storageDir[len(storageDir)-1] != '/' {
+		storageDir += "/"
+	}
+
+	storage = &WebDAVStorage{
+		host:       host,
+		port:       port,
+		username:   username,
+		password:   password,
+		storageDir: "",
+		useHTTP:    false,
+
+		client:         http.DefaultClient,
+		threads:        threads,
+		directoryCache: make(map[string]int),
+	}
+
+	// Make sure it doesn't follow redirect
+	storage.client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
+        return http.ErrUseLastResponse
+	}
+
+	exist, isDir, _, err := storage.GetFileInfo(0, storageDir)
+	if err != nil {
+		return nil, err
+	}
+	if !exist {
+		return nil, fmt.Errorf("Storage path %s does not exist", storageDir)
+	}
+	if !isDir {
+		return nil, fmt.Errorf("Storage path %s is not a directory", storageDir)
+	}
+	storage.storageDir = storageDir
+
+	for _, dir := range []string{"snapshots", "chunks"} {
+		storage.CreateDirectory(0, dir)
+	}
+
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
+}
+
+func (storage *WebDAVStorage) createConnectionString(uri string) string {
+
+	url := storage.host
+
+	if storage.useHTTP {
+		url = "http://" + url
+	} else {
+		url = "https://" + url
+	}
+
+	if storage.port > 0 {
+		url += fmt.Sprintf(":%d", storage.port)
+	}
+	return url + "/" + storage.storageDir + uri
+}
+
+func (storage *WebDAVStorage) retry(backoff int) int {
+	delay := rand.Intn(backoff*500) + backoff*500
+	time.Sleep(time.Duration(delay) * time.Millisecond)
+	backoff *= 2
+	return backoff
+}
+
+func (storage *WebDAVStorage) sendRequest(method string, uri string, depth int, data []byte) (io.ReadCloser, http.Header, error) {
+
+	backoff := 1
+	for i := 0; i < 8; i++ {
+
+		var dataReader io.Reader
+		headers := make(map[string]string)
+		if method == "PROPFIND" {
+			headers["Content-Type"] = "application/xml"
+			headers["Depth"] = fmt.Sprintf("%d", depth)
+			dataReader = bytes.NewReader(data)
+		} else if method == "PUT" {
+			headers["Content-Type"] = "application/octet-stream"
+			dataReader = CreateRateLimitedReader(data, storage.UploadRateLimit/storage.threads)
+		} else if method == "MOVE" {
+			headers["Destination"] = storage.createConnectionString(string(data))
+			headers["Content-Type"] = "application/octet-stream"
+			dataReader = bytes.NewReader([]byte(""))
+		} else {
+			headers["Content-Type"] = "application/octet-stream"
+			dataReader = bytes.NewReader(data)
+		}
+
+		request, err := http.NewRequest(method, storage.createConnectionString(uri), dataReader)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		if len(storage.username) > 0 {
+			request.SetBasicAuth(storage.username, storage.password)
+		}
+
+		for key, value := range headers {
+			request.Header.Set(key, value)
+		}
+
+		//requestDump, err := httputil.DumpRequest(request, true)
+		//LOG_INFO("debug", "Request: %s", requestDump)
+
+		response, err := storage.client.Do(request)
+		if err != nil {
+			LOG_TRACE("WEBDAV_RETRY", "URL request '%s %s' returned an error (%v)", method, uri, err)
+			backoff = storage.retry(backoff)
+			continue
+		}
+
+		if response.StatusCode < 300 {
+			return response.Body, response.Header, nil
+		}
+
+		if response.StatusCode == 301 {
+			return nil, nil, errWebDAVMovedPermanently
+		}
+
+		response.Body.Close()
+		if response.StatusCode == 404 {
+			// Retry if it is UPLOAD, otherwise return immediately
+			if method != "PUT" {
+				return nil, nil, errWebDAVNotExist
+			}
+		} else if response.StatusCode == 405 {
+			return nil, nil, errWebDAVMethodNotAllowed
+		}
+		LOG_INFO("WEBDAV_RETRY", "URL request '%s %s' returned status code %d", method, uri, response.StatusCode)
+		backoff = storage.retry(backoff)
+	}
+	return nil, nil, errWebDAVMaximumBackoff
+}
+
+type WebDAVProperties map[string]string
+
+type WebDAVPropValue struct {
+	XMLName xml.Name `xml:""`
+	Value   string   `xml:",innerxml"`
+}
+
+type WebDAVProp struct {
+	PropList []WebDAVPropValue `xml:",any"`
+}
+
+type WebDAVPropStat struct {
+	Prop *WebDAVProp `xml:"prop"`
+}
+
+type WebDAVResponse struct {
+	Href     string          `xml:"href"`
+	PropStat *WebDAVPropStat `xml:"propstat"`
+}
+
+type WebDAVMultiStatus struct {
+	Responses []WebDAVResponse `xml:"response"`
+}
+
+func (storage *WebDAVStorage) getProperties(uri string, depth int, properties ...string) (map[string]WebDAVProperties, error) {
+
+	propfind := "<prop>"
+	for _, p := range properties {
+		propfind += fmt.Sprintf("<%s/>", p)
+	}
+	propfind += "</prop>"
+
+	body := fmt.Sprintf(`<?xml version="1.0" encoding="utf-8" ?><propfind xmlns="DAV:">%s</propfind>`, propfind)
+
+	readCloser, _, err := storage.sendRequest("PROPFIND", uri, depth, []byte(body))
+	if err != nil {
+		return nil, err
+	}
+	defer readCloser.Close()
+	content, err := ioutil.ReadAll(readCloser)
+	if err != nil {
+		return nil, err
+	}
+
+	object := WebDAVMultiStatus{}
+	err = xml.Unmarshal(content, &object)
+	if err != nil {
+		return nil, err
+	}
+
+	if object.Responses == nil || len(object.Responses) == 0 {
+		return nil, errors.New("no WebDAV responses")
+	}
+
+	responses := make(map[string]WebDAVProperties)
+
+	for _, responseTag := range object.Responses {
+
+		if responseTag.PropStat == nil || responseTag.PropStat.Prop == nil || responseTag.PropStat.Prop.PropList == nil {
+			return nil, errors.New("no WebDAV properties")
+		}
+
+		properties := make(WebDAVProperties)
+		for _, prop := range responseTag.PropStat.Prop.PropList {
+			properties[prop.XMLName.Local] = prop.Value
+		}
+
+		responseKey := responseTag.Href
+		responses[responseKey] = properties
+
+	}
+
+	return responses, nil
+}
+
+// ListFiles return the list of files and subdirectories under 'dir'.  A subdirectories returned must have a trailing '/', with
+// a size of 0.  If 'dir' is 'snapshots', only subdirectories will be returned.  If 'dir' is 'snapshots/repository_id', then only
+// files will be returned.  If 'dir' is 'chunks', the implementation can return the list either recusively or non-recusively.
+func (storage *WebDAVStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
+	if dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
+	properties, err := storage.getProperties(dir, 1, "getcontentlength", "resourcetype")
+	if err != nil {
+		return nil, nil, err
+	}
+
+	prefixLength := len(storage.storageDir) + len(dir) + 1
+
+	for file, m := range properties {
+		if len(file) <= prefixLength {
+			continue
+		}
+
+		isDir := false
+		size := 0
+		if resourceType, exist := m["resourcetype"]; exist && strings.Contains(resourceType, "collection") {
+			isDir = true
+		} else if length, exist := m["getcontentlength"]; exist {
+			if length == "" {
+				isDir = true
+			} else {
+				size, _ = strconv.Atoi(length)
+			}
+		} else {
+			continue
+		}
+
+		if !isDir {
+			if dir != "snapshots/" {
+				files = append(files, file[prefixLength:])
+				sizes = append(sizes, int64(size))
+			}
+		} else {
+			// This is a dir
+			file := file[prefixLength:]
+			if file[len(file)-1] != '/' {
+				file += "/"
+			}
+			files = append(files, file)
+			sizes = append(sizes, int64(0))
+		}
+	}
+
+	return files, sizes, nil
+}
+
+// GetFileInfo returns the information about the file or directory at 'filePath'.
+func (storage *WebDAVStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
+	properties, err := storage.getProperties(filePath, 0, "getcontentlength", "resourcetype")
+	if err != nil {
+		if err == errWebDAVNotExist {
+			return false, false, 0, nil
+		}
+		if err == errWebDAVMovedPermanently {
+			// This must be a directory
+			return true, true, 0, nil
+		}
+		return false, false, 0, err
+	}
+
+	if m, exist := properties["/" + storage.storageDir + filePath]; !exist {
+		return false, false, 0, nil
+	} else if resourceType, exist := m["resourcetype"]; exist && strings.Contains(resourceType, "collection") {
+		return true, true, 0, nil
+	} else if length, exist := m["getcontentlength"]; exist && length != ""{
+		value, _ := strconv.Atoi(length)
+		return true, false, int64(value), nil
+	} else {
+		return true, true, 0, nil
+	}
+}
+
+// DeleteFile deletes the file or directory at 'filePath'.
+func (storage *WebDAVStorage) DeleteFile(threadIndex int, filePath string) (err error) {
+	readCloser, _, err := storage.sendRequest("DELETE", filePath, 0, []byte(""))
+	if err != nil {
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// MoveFile renames the file.
+func (storage *WebDAVStorage) MoveFile(threadIndex int, from string, to string) (err error) {
+	readCloser, _, err := storage.sendRequest("MOVE", from, 0, []byte(to))
+	if err != nil {
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// createParentDirectory creates the parent directory if it doesn't exist in the cache
+func (storage *WebDAVStorage) createParentDirectory(threadIndex int, dir string) (err error) {
+
+	found := strings.LastIndex(dir, "/")
+	if found == -1 {
+		return nil
+	}
+	parent := dir[:found]
+
+	storage.directoryCacheLock.Lock()
+	_, exist := storage.directoryCache[parent]
+	storage.directoryCacheLock.Unlock()
+
+	if exist {
+		return nil
+	}
+
+	err = storage.CreateDirectory(threadIndex, parent)
+	if err == nil {
+		storage.directoryCacheLock.Lock()
+		storage.directoryCache[parent] = 1
+		storage.directoryCacheLock.Unlock()
+	}
+	return err
+}
+
+// CreateDirectory creates a new directory.
+func (storage *WebDAVStorage) CreateDirectory(threadIndex int, dir string) (err error) {
+	for dir != "" && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
+
+	if dir == "" {
+		return nil
+	}
+
+	// If there is an error in creating the parent directory, proceed anyway
+	storage.createParentDirectory(threadIndex, dir)
+
+	readCloser, _, err := storage.sendRequest("MKCOL", dir, 0, []byte(""))
+	if err != nil {
+		if err == errWebDAVMethodNotAllowed || err == errWebDAVMovedPermanently {
+			// We simply ignore these errors and assume that the directory already exists
+			return nil
+		}
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// DownloadFile reads the file at 'filePath' into the chunk.
+func (storage *WebDAVStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
+	readCloser, _, err := storage.sendRequest("GET", filePath, 0, nil)
+	if err != nil {
+		return err
+	}
+
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.threads)
+	return err
+}
+
+// UploadFile writes 'content' to the file at 'filePath'.
+func (storage *WebDAVStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
+
+	// If there is an error in creating the parent directory, proceed anyway
+	storage.createParentDirectory(threadIndex, filePath)
+
+	readCloser, _, err := storage.sendRequest("PUT", filePath, 0, content)
+	if err != nil {
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
+// managing snapshots.
+func (storage *WebDAVStorage) IsCacheNeeded() bool { return true }
+
+// If the 'MoveFile' method is implemented.
+func (storage *WebDAVStorage) IsMoveFileImplemented() bool { return true }
+
+// If the storage can guarantee strong consistency.
+func (storage *WebDAVStorage) IsStrongConsistent() bool { return false }
+
+// If the storage supports fast listing of files names.
+func (storage *WebDAVStorage) IsFastListing() bool { return false }
+
+// Enable the test mode.
+func (storage *WebDAVStorage) EnableTestMode() {}
Author	SHA1	Message	Date
Gilbert Chen	e8b8922754	Continue to check other snapshots when one snapshot has missing chunks	2018-08-06 21:20:04 -04:00
Gilbert Chen	93cc632021	Record deleted snapshots in the fossil collection and if any deleted snapshot still exist nude the fossil collection	2018-08-04 22:59:25 -04:00
Gilbert Chen	48cc5eaedb	Print git commit number	2018-08-03 23:45:23 -04:00
Gilbert Chen	f304b64b3f	Removed a redundant call to manager.chunkOperator.Resurrect	2018-08-03 11:32:24 -04:00
Gilbert Chen	8ae7d2a97d	Remove extra newline in the PRUNE_NEWSNAPSHOT log message	2018-07-26 21:24:33 -04:00
Gilbert Chen	fce4234861	Rearrange struct members to avoid 64-bit int alignment issues	2018-07-26 21:19:03 -04:00
gilbertchen	e499a24202	Merge pull request #459 from jtackaberry/master Fix "Failed to fossilize chunk" errors in wasabi backend	2018-07-24 21:52:27 -04:00
Gilbert Chen	89769f3906	Add a -storage option to the benchmark command	2018-07-23 22:54:56 -04:00
Gilbert Chen	798cec0714	Bump version to 2.1.1	2018-07-23 22:10:12 -04:00
Gilbert Chen	72dfaa8b6b	Fixed a bug causing a new snapshot to be not counted when deciding which fossils can be deleted	2018-07-23 22:08:08 -04:00
Jason Tackaberry	117cfd997f	Make Wasabi double slash fix more idiomatic	2018-07-13 12:43:27 -04:00
Jason Tackaberry	84f7c513d5	Fix "Failed to fossilize chunk" errors in wasabi backend Fixes #458	2018-07-11 22:55:04 -04:00
gilbertchen	dfdbfed64b	Merge pull request #449 from gilbertchen/benchmark_command Benchmark command	2018-07-02 16:26:28 -04:00
gilbertchen	d4a65ffbcf	Merge pull request #441 from gilbertchen/threaded_prune Implement multithreaded pruning	2018-07-02 16:20:41 -04:00
gilbertchen	736003323a	Merge pull request #417 from amarcu5/fix-permissions Fixed restoration of basic UNIX file permissions	2018-07-02 14:31:10 -04:00
gilbertchen	0af74616b7	Merge pull request #415 from amarcu5/master Add APFS snapshot support	2018-07-02 12:41:19 -04:00
gilbertchen	0f552c8c50	Update ISSUE_TEMPLATE.md	2018-06-20 23:24:59 -04:00
gilbertchen	1adf92e879	Create ISSUE_TEMPLATE.md	2018-06-20 23:24:02 -04:00
Gilbert Chen	f92f1a728c	Check in src/duplicacy_benchmark.go	2018-06-17 22:09:47 -04:00
Gilbert Chen	9a0dcdb0b2	Add a benchmark command to test disk and network speeds	2018-06-17 22:00:01 -04:00
Gilbert Chen	9ae306644d	Avoid filepath.Dir as it returns paths with back slashes on Windows	2018-06-07 16:00:39 -04:00
Gilbert Chen	6f0166be6d	Add a test for multi-threaded pruning	2018-06-06 13:11:19 -04:00
Gilbert Chen	f68eb13584	A few fixes for multi-threaded pruning	2018-06-05 16:09:12 -04:00
Gilbert Chen	dd53b4797e	Implement multi-threaded pruning	2018-06-04 21:52:07 -04:00
Gilbert Chen	7e021f26d3	Don't show the nil error when failing to replace a file with a directory during restore	2018-05-30 13:23:35 -04:00
Gilbert Chen	0e585e4be4	Fixed a crashing bug when showing the history of excluded files	2018-05-30 12:05:40 -04:00
Gilbert Chen	e03cd2a880	Add unreferenced fossils to the fossil collection instead of deleting them immediately	2018-05-29 12:57:38 -04:00
Gilbert Chen	f80a5b1025	Fixed format argument errors	2018-05-24 11:37:52 -04:00
Gilbert Chen	aadd2aa390	Add an -enum-only option to the backup command to enumerate the repository only	2018-05-24 11:34:46 -04:00
Gilbert Chen	72239a31c4	Add -repository optiont to init add add to specifiy an alternate repository path	2018-05-22 15:55:52 -04:00
gilbertchen	c9b60cc0e0	Merge pull request #394 from gilbertchen/webdav Implement the WebDAV backend	2018-05-16 23:31:46 -04:00
gilbertchen	f4cdd1f01b	Merge pull request #392 from fhriley/nobackup_file Add nobackup-file preference.	2018-05-16 23:31:09 -04:00
Gilbert Chen	b1c1b47983	Add an env var DUPLICACY_DECRYPT_WITH_HMACSHA256 to force using HMAC-SHA256 for encryption key in order to be able to manage backups created by Vertical Backup	2018-05-02 22:57:47 -04:00
amarcu5	8c3ef6cae9	Improved cron support Now calls mount and unmount using absolute path	2018-04-30 00:40:34 +01:00
amarcu5	acd7addc9a	Fixed Windows complication	2018-04-30 00:36:11 +01:00
amarcu5	c23ea30da4	Updated VSS flag usage VSS flag usage now indicates support for macOS using APFS in addition to Windows	2018-04-29 23:24:22 +01:00
amarcu5	a4c46624ea	Fix basic UNIX permissions	2018-04-29 20:24:50 +01:00
amarcu5	5747d6763f	Improved error handling Minor improvement to error handling and small formatting changes	2018-04-29 13:33:17 +01:00
amarcu5	ef1d316f33	Cleaned up formatting Fixed tabbing	2018-04-29 04:21:28 +01:00
amarcu5	714a45c34a	Improved macOS VSS checks VSS support now determined by: 1) Checking that repository filesystem is APFS rather than by inspecting macOS version 2) Checking that repository resides on local device (as external APFS formatted drives are not supported) rather than crudely by path name	2018-04-29 04:11:10 +01:00
Gilbert Chen	23a2d91608	Skipped chunks should not be counted in order to get accurate download percentage	2018-04-28 20:19:24 -04:00
amarcu5	23b98a3034	Added macOS VSS checks Restricts VSS under macOS to version 10.13 High Sierra or higher and local volume only	2018-04-28 23:09:59 +01:00
amarcu5	8a3c5847a8	Cleaned up formatting	2018-04-28 21:23:00 +01:00
amarcu5	4c3d5dbc2f	Add APFS snapshot support Add's VSS support for macOS using APFS snapshot's	2018-04-28 17:58:47 +01:00
Gilbert Chen	85bc55e374	The -threads option for the copy command specifies the number of uploading threads	2018-04-11 21:11:24 -04:00
gilbertchen	cd0c7b07a9	Update README.md	2018-04-06 22:27:31 -04:00
gilbertchen	0ea26a92dd	Update README.md	2018-04-06 09:46:42 -04:00
gilbertchen	ca889fca9f	Update README.md	2018-04-06 09:28:05 -04:00
gilbertchen	fbaea6e8b1	Update README.md	2018-04-06 00:06:48 -04:00
gilbertchen	2290c4ace0	Add files via upload	2018-04-05 23:36:57 -04:00
Gilbert Chen	02cd41f4d0	A few improvements to make WebDAV work better with pcloud and box.com	2018-04-05 15:29:41 -04:00
Gilbert Chen	0db8b9831b	Implement the WebDAV backend	2018-04-02 20:03:50 -04:00
Frank	4dd5c43307	Add nobackup-file preference. Directories containing a file with this name will not be backed up. I find it easier to drop a .nobackup file in directories I don't want backed up instead of maintaining a file of exclusions. This is also useful for scripts that create data in the repository but don't want it to be backed up.	2018-04-01 12:50:00 -07:00
Gilbert Chen	6aedc37118	Update the description for the -comment option	2018-03-28 22:19:18 -04:00
gilbertchen	9a56ede07c	Merge pull request #391 from jeffaco/jeff-comment Add -comment capability to allow for duplicacy jobs to be identified with 'ps'	2018-03-28 22:10:58 -04:00
Jeff Coffler	c6e9460b7b	Add 'How to build' reference to README.md.	2018-03-28 11:22:46 -07:00
Jeff Coffler	e74ab809ae	Add new qualifier, `-comment`, to more easily identify specific jobs. This qualifier allows specific text to be associated with a backup job to easily associate what repository (or other context) a duplicacy job is running on behalf of.	2018-03-28 10:16:08 -07:00
Gilbert Chen	5d2242d39d	Preserve the list of chunk hashes for the latest snapshot when cleaning local snapshot cache	2018-03-27 23:34:40 -04:00
Gilbert Chen	b99f4bffec	Follow symlinks that point to paths starting with \\	2018-03-22 22:37:09 -04:00
Gilbert Chen	be2856ebbd	Add a -vss-timeout option to set VSS creation timeout	2018-03-21 22:34:14 -04:00
Gilbert Chen	1ea615fb45	Fix the names of prune tests so they can be run all in once	2018-03-20 15:03:03 -04:00
Gilbert Chen	7d933a2576	Create two identical snapshots in prune test to catch a retrive-after-deletion bug	2018-03-20 14:37:47 -04:00
gilbertchen	13fffc2a11	Merge pull request #329 from pdf/prune_memory Reduce memory consumption for prune operation	2018-03-20 14:32:12 -04:00
gilbertchen	9658463ebe	Merge pull request #331 from markfeit/master Added semi-dedicated Wasabi storage module. #322	2018-03-19 12:03:31 -04:00
gilbertchen	cd77a029ea	Merge branch 'master' into master	2018-03-19 11:53:11 -04:00
Mark Feit	8fdb399e1b	Correct handling of @ in region to be consistent with everythng else.	2018-02-11 07:51:34 -05:00
Mark Feit	9d632c0434	Handle application/testing region string inconsistency.	2018-02-10 10:16:04 -05:00
Mark Feit	cc6e96527e	Add/rearrange returns to keep the compiler from complaining.	2018-02-10 10:15:40 -05:00
Peter Fern	57082cd1d2	Reduce memory consumption for prune operation For non-exhaustive prune, consider only target chunks instead of mapping all chunks in repository.	2018-01-21 10:12:09 +11:00
Peter Fern	bd5a689b7d	Add keepChunkHashes flag to GetSnapshotChunks, allowing reduced memory	2018-01-21 10:11:59 +11:00
Mark Feit	b52d6b3f7f	Incorporated PR feedback; call S3 for IsFastListing() #322	2018-01-18 08:34:33 -05:00
Mark Feit	8aaca37a2b	Added note about Wasabi dependency.	2018-01-18 08:30:00 -05:00
Mark Feit	30f753e499	Cosmetic and key name fixes. #322	2018-01-16 22:19:41 -05:00
Mark Feit	d0771be2dd	Added semi-dedicated Wasabi storage module. #322	2018-01-16 22:06:11 -05:00