Bump version to 2.1.0

Implement OpenStack Swift backend

DESIGN.md

@@ -1,213 +1,5 @@
-## Lock-Free Deduplication
+All documentation has been moved to our wiki page:
 
-The three elements of lock-free deduplication are:
-
-* Use variable-size chunking algorithm to split files into chunks
-* Store each chunk in the storage using a file name derived from its hash, and rely on the file system API to manage chunks without using a centralized indexing database
-* Apply a *two-step fossil collection* algorithm to remove chunks that become unreferenced after a backup is deleted
-
-The variable-size chunking algorithm, also called Content-Defined Chunking, is well-known and has been adopted by many
-backup tools. The main advantage of the variable-size chunking algorithm over the fixed-size chunking algorithm (as used
-by rsync) is that in the former the rolling hash is only used to search for boundaries between chunks, after which a far
-more collision-resistant hash function like MD5 or SHA256 is applied on each chunk. In contrast, in the fixed-size
-chunking algorithm, for the purpose of detecting inserts or deletions, a lookup in the known hash table is required every
-time the rolling hash window is shifted by one byte, thus significantly reducing the chunk splitting performance.
-
-What is novel about lock-free deduplication is the absence of a centralized indexing database for tracking all existing
-chunks and for determining which chunks are not needed any more.  Instead, to check if a chunk has already been uploaded
-before, one can just perform a file lookup via the file storage API using the file name derived from the hash of the chunk.
-This effectively turns a cloud storage offering only a very limited
-set of basic file operations into a powerful modern backup backend capable of both block-level and file-level deduplication.  More importantly, the absence of a centralized indexing database means that there is no need to implement a distributed locking mechanism on top of the file storage.
-
-By eliminating the chunk indexing database, lock-free duplication not only reduces the code complexity but also makes the deduplication less error-prone.  Each chunk is saved individually in its own file, and once saved there is no need for modification.  Data corruption is therefore less likely to occur because of the immutability of chunk files.  Another benefit that comes naturally from lock-free duplication is that when one client creates a new chunk, other clients that happen to have the same original file will notice that the chunk  already exist and therefore will not upload the same chunk again.  This pushes the deduplication to its highest level -- clients without knowledge of each other can share identical chunks with no extra effort.
-
-There is one problem, though.
-Deletion of snapshots without an indexing database, when concurrent access is permitted, turns out to be a hard problem.
-If exclusive access to a file storage by a single client can be guaranteed, the deletion procedure can simply search for
-chunks not referenced by any backup and delete them. However, if concurrent access is required, an unreferenced chunk
-can't be trivially removed, because of the possibility that a backup procedure in progress may reference the same chunk.
-The ongoing backup procedure, still unknown to the deletion procedure, may have already encountered that chunk during its
-file scanning phase, but decided not to upload the chunk again since it already exists in the file storage.
-
-Fortunately, there is a solution to address the deletion problem and make lock-free deduplication practical.  The solution is a *two-step fossil collection* algorithm that deletes unreferenced chunks in two steps: identify and collect them in the first step, and then permanently remove them once certain conditions are met.
-
-## Two-Step Fossil Collection
-
-Interestingly, the two-step fossil collection algorithm hinges on a basic file operation supported almost universally, *file renaming*.
-When the deletion procedure identifies a chunk not referenced by any known snapshots, instead of deleting the chunk file
-immediately, it changes the name of the chunk file (and possibly moves it to a different directory).
-A chunk that has been renamed is called a *fossil*.
-
-The fossil still exists in the file storage.  Two rules are enforced regarding the access of fossils:
-
-* A restore, list, or check procedure that reads existing backups can read the fossil if the original chunk cannot be found.
-* A backup procedure does not check the existence of a fossil. That is, it must upload a chunk if it cannot find the chunk, even if an equivalent fossil exists.
-
-In the first step of the deletion procedure, called the *fossil collection* step, the names of all identified fossils will
-be saved in a fossil collection file. The deletion procedure then exits without performing further actions. This step has not effectively changed any chunk references due to the first fossil access rule.  If a backup procedure references a chunk after it is marked as a fossil, a new chunk will be uploaded because of the second fossil access rule, as shown in Figure 1.
-
-<p align="center">
-  <img src="https://github.com/gilbertchen/duplicacy-beta/blob/master/images/fossil_collection_1.png?raw=true"
-       alt="Reference after Rename"/>
-</p>
-
-The second step, called the *fossil deletion* step, will permanently delete fossils, but only when two conditions are met:
-
-* For each snapshot id, there is a new snapshot that was not seen by the fossil collection step
-* The new snapshot must finish after the fossil collection step
-
-The first condition guarantees that if a backup procedure references a chunk before the deletion procedure turns it into a fossil, the reference will be detected in the fossil deletion step which will then turn the fossil back into a normal chunk.
-
-The second condition guarantees that any backup procedure unknown to the fossil deletion step can start only after the fossil collection step finishes.  Therefore, if it references a chunk that was identified as fossil in the fossil collection step, it should observe the fossil, not the chunk, so it will upload a new chunk, according to the second fossil access rule.
-
-Therefore, if a backup procedure references a chunk before the chunk is marked a fossil, the fossil deletion step will not
-delete the chunk until it sees that backup procedure finishes (as indicated by the appearance of a new snapshot file uploaded to the storage).  This ensures that scenarios depicted in Figure 2 will never happen.
-
-<p align="center">
-  <img src="https://github.com/gilbertchen/duplicacy-beta/blob/master/images/fossil_collection_2.png?raw=true"
-       alt="Reference before Rename"/>
-</p>
-
-
-## Snapshot Format
-
-A snapshot file is a file that the backup procedure uploads to the file storage after it finishes splitting files into
-chunks and uploading all new chunks. It mainly contains metadata for the backup overall, metadata for all the files,
-and chunk references for each file. Here is an example snapshot file for a repository containing 3 files (file1, file2,
-and dir1/file3):
-
-```json
-{
-  "id": "host1",
-  "revision": 1,
-  "tag": "first",
-  "start_time": 1455590487,
-  "end_time": 1455590487,
-  "files": [
-    {
-      "path": "file1",
-      "content": "0:0:2:6108",
-      "hash": "a533c0398194f93b90bd945381ea4f2adb0ad50bd99fd3585b9ec809da395b51",
-      "size": 151901,
-      "time": 1455590487,
-      "mode": 420
-    },
-    {
-      "path": "file2",
-      "content": "2:6108:3:7586",
-      "hash": "f6111c1562fde4df9c0bafe2cf665778c6e25b49bcab5fec63675571293ed644",
-      "size": 172071,
-      "time": 1455590487,
-      "mode": 420
-    },
-    {
-      "path": "dir1/",
-      "size": 102,
-      "time": 1455590487,
-      "mode": 2147484096
-    },
-    {
-      "path": "dir1/file3",
-      "content": "3:7586:4:1734",
-      "hash": "6bf9150424169006388146908d83d07de413de05d1809884c38011b2a74d9d3f",
-      "size": 118457,
-      "time": 1455590487,
-      "mode": 420
-    }
-  ],
-  "chunks": [
-    "9f25db00881a10a8e7bcaa5a12b2659c2358a579118ea45a73c2582681f12919",
-    "6e903aace6cd05e26212fcec1939bb951611c4179c926351f3b20365ef2c212f",
-    "4b0d017bce5491dbb0558c518734429ec19b8a0d7c616f68ddf1b477916621f7",
-    "41841c98800d3b9faa01b1007d1afaf702000da182df89793c327f88a9aba698",
-    "7c11ee13ea32e9bb21a694c5418658b39e8894bbfecd9344927020a9e3129718"
-  ],
-  "lengths": [
-    64638,
-    81155,
-    170593,
-    124309,
-    1734
-  ]
-}
-```
-
-When Duplicacy splits a file in chunks using the variable-size chunking algorithm, if the end of a file is reached and yet the boundary marker for terminating a chunk
-hasn't been found, the next file, if there is one, will be read in and the chunking algorithm continues. It is as if all
-files were packed into a big tar file which is then split into chunks.
-
-The *content* field of a file indicates the indexes of starting and ending chunks and the corresponding offsets. For
-instance, *file1* starts at chunk 0 offset 0 while ends at chunk 2 offset 6108, immediately followed by *file2*.
-
-The backup procedure can run in one of two modes. In the default quick mode, only modified or new files are scanned. Chunks only
-referenced by old files that have been modified are removed from the chunk sequence, and then chunks referenced by new
-files are appended. Indices for unchanged files need to be updated too.
-
-In the safe mode (enabled by the -hash option), all files are scanned and the chunk sequence is regenerated.
-
-The length sequence stores the lengths for all chunks, which are needed when calculating some statistics such as the total
-length of chunks. For a repository containing a large number of files, the size of the snapshot file can be tremendous.
-To make the situation worse, every time a big snapshot file would have been uploaded even if only a few files have been changed since
-last backup. To save space, the variable-size chunking algorithm is also applied to the three dynamic fields of a snapshot
-file, *files*, *chunks*, and *lengths*.
-
-Chunks produced during this step are deduplicated and uploaded in the same way as regular file chunks. The final snapshot file
-contains sequences of chunk hashes and other fixed size fields:
-
-```json
-{
-  "id": "host1",
-  "revision": 1,
-  "start_time": 1455590487,
-  "tag": "first",
-  "end_time": 1455590487,
-  "file_sequence": [
-    "21e4c69f3832e32349f653f31f13cefc7c52d52f5f3417ae21f2ef5a479c3437",
-  ],
-  "chunk_sequence": [
-    "8a36ffb8f4959394fd39bba4f4a464545ff3dd6eed642ad4ccaa522253f2d5d6"
-  ],
-  "length_sequence": [
-    "fc2758ae60a441c244dae05f035136e6dd33d3f3a0c5eb4b9025a9bed1d0c328"
-  ]
-}
-```
-
-In the extreme case where the repository has not been modified since last backup, a new backup procedure will not create any new chunks,
-as shown by the following output from a real use case:
-
-```
-$ duplicacy backup -stats
-Storage set to sftp://gchen@192.168.1.100/Duplicacy
-Last backup at revision 260 found
-Backup for /Users/gchen/duplicacy at revision 261 completed
-Files: 42367 total, 2,204M bytes; 0 new, 0 bytes
-File chunks: 447 total, 2,238M bytes; 0 new, 0 bytes, 0 bytes uploaded
-Metadata chunks: 6 total, 11,753K bytes; 0 new, 0 bytes, 0 bytes uploaded
-All chunks: 453 total, 2,249M bytes; 0 new, 0 bytes, 0 bytes uploaded
-Total running time: 00:00:05
-```
-
-## Encryption
-
-When encryption is enabled (by the -e option with the *init* or *add* command), Duplicacy will generate 4 random 256 bit keys:
-
-* *Hash Key*: for generating a chunk hash from the content of a chunk
-* *ID Key*: for generating a chunk id from a chunk hash
-* *Chunk Key*: for encrypting chunk files
-* *File Key*: for encrypting non-chunk files such as snapshot files.
-
-Here is a diagram showing how these keys are used:
-
-<p align="center">
-  <img src="https://github.com/gilbertchen/duplicacy-beta/blob/master/images/duplicacy_encryption.png?raw=true"
-       alt="encryption"/>
-</p>
-
-Chunk hashes are used internally and stored in the snapshot file.  They are never exposed unless the snapshot file is decrypted.  Chunk ids are used as the file names for the chunks and therefore exposed.  When the *cat* command is used to print out a snapshot file, the chunk hashes stored in the snapshot file will be converted into chunk ids first which are then displayed instead.
-
-Chunk content is encrypted by AES-GCM, with an encryption key that is the HMAC-SHA256 of the chunk Hash with the *Chunk Key* as the secret key.
-
-The snapshot is encrypted by AES-GCM too, using an encrypt key that is the HMAC-SHA256 of the file path with the *File Key* as the secret key.
-
-These four random keys are saved in a file named 'config' in the storage, encrypted with a master key derived from the PBKDF2 function on the storage password chosen by the user.
+* [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication)
+* [Snapshot Format](https://github.com/gilbertchen/duplicacy/wiki/Snapshot-Format)
+* [Encryption](https://github.com/gilbertchen/duplicacy/wiki/Encryption)

GUIDE.md

@@ -1,475 +1,20 @@
-# Duplicacy User Guide
-
-## Commands
-
-#### Init
-
-```
-SYNOPSIS:
-   duplicacy init - Initialize the storage if necessary and the current directory as the repository
-
-USAGE:
-   duplicacy init [command options] <snapshot id> <storage url>
-
-OPTIONS:
-   -encrypt, -e                    encrypt the storage with a password
-   -chunk-size, -c 4M              the average size of chunks
-   -max-chunk-size, -max 16M       the maximum size of chunks (defaults to chunk-size * 4)
-   -min-chunk-size, -min 1M        the minimum size of chunks (defaults to chunk-size / 4)
-   -pref-dir <preference directory path>    Specify alternate location for .duplicacy preferences directory
-```
-
-The *init* command first connects to the storage specified by the storage URL.  If the storage has been already been initialized before, it will download the storage configuration (stored in the file named *config*) and ignore the options provided in the command line.  Otherwise, it will create the configuration file from the options and upload the file.
-
-The initialized storage will then become the default storage for other commands if the `-storage` option is not specified for those commands.  This default storage actually has a name, *default*.
-
-After that, it will prepare the current working directory as the repository to be backed up.  Under the hood, it will create a directory named *.duplicacy* in the repository and put a file named *preferences* that stores the snapshot id and encryption and storage options.
-
-The snapshot id is an id used to distinguish different repositories connected to the same storage.  Each repository must have a unique snapshot id.  A snapshot id must contain only characters valid in Linux and Windows paths (alphabet, digits, underscore, dash, etc), but cannot include `/`, `\`, or `@`.
-
-The `-e` option controls whether or not encryption will be enabled for the storage.  If encryption is enabled, you will be prompted to enter a storage password.
-
-The three chunk size parameters are passed to the variable-size chunking algorithm.  Their values are important to the overall performance, especially for cloud storages.  If the chunk size is too small, a lot of overhead will be in sending requests and receiving responses.  If the chunk size is too large, the effect of de-duplication will be less obvious as more data will need to be transferred with each chunk.
-
-The `-pref-dir` controls the location of the preferences directory. If not specified, a directory named .duplicacy is created in the repository. If specified, it must point to a non-existing directory. The directory is created and a .duplicacy file is created in the repository. The .duplicacy file contains the absolute path name to the preferences directory.
-
-Once a storage has been initialized with these parameters, these parameters cannot be modified any more.
-
-#### Backup
-
-```
-SYNOPSIS:
-   duplicacy backup - Save a snapshot of the repository to the storage
-
-USAGE:
-   duplicacy backup [command options]
-
-OPTIONS:
-   -hash                      detect file differences by hash (rather than size and timestamp)
-   -t <tag>                   assign a tag to the backup
-   -stats                     show statistics during and after backup
-   -threads <n>               number of uploading threads
-   -limit-rate <kB/s>         the maximum upload rate (in kilobytes/sec)
-   -vss                       enable the Volume Shadow Copy service (Windows only)
-   -storage <storage name>    backup to the specified storage instead of the default one
-```
-
-The *backup* command creates a snapshot of the repository and uploads it to the storage.  If `-hash` is not provided,it will upload new or modified files since last backup by comparing file sizes and timestamps.  Otherwise, every file is scanned to detect changes.
-
-You can assign a tag to the snapshot so that later you can refer to it by tag in other commands.
-
-If the `-stats` option is specified, statistical information such as transfer speed, and the number of chunks will be displayed throughout the backup procedure.
-
-The `-threads` option can be used to specify more than one thread to upload chunks.
-
-The `-limit-rate` option sets a cap on the maximum upload rate.
-
-The `-vss` option works on Windows only to turn on the Volume Shadow Copy service such that files opened by other processes with exclusive locks can be read as usual.
-
-When the repository can have multiple storages (added by the *add* command), you can select the storage to back up to by giving a storage name.
-
-You can specify patterns to include/exclude files by putting them in a file named *.duplicacy/filters*.  Please refer to the [Include/Exclude Patterns](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#includeexclude-patterns) section for how to specify the patterns.
-
-#### Restore
-```
-SYNOPSIS:
-   duplicacy restore - Restore the repository to a previously saved snapshot
-
-USAGE:
-   duplicacy restore [command options] [--] [pattern] ...
-
-OPTIONS:
-   -r <revision>            the revision number of the snapshot (required)
-   -hash                    detect file differences by hash (rather than size and timestamp)
-   -overwrite               overwrite existing files in the repository
-   -delete                  delete files not in the snapshot
-   -stats                   show statistics during and after restore
-   -threads <n>             number of downloading threads
-   -limit-rate <kB/s>       the maximum download rate (in kilobytes/sec)
-   -storage <storage name>  restore from the specified storage instead of the default one
-```
-
-The *restore* command restores the repository to a previous revision.  By default the restore procedure will treat files that have the same sizes and timestamps as those in the snapshot as unchanged files, but with the -hash option, every file will be fully scanned to make sure they are in fact unchanged.
-
-By default the restore procedure will not overwriting existing files, unless the `-overwrite` option is specified.
-
-The `-delete` option indicates that files not in the snapshot will be removed.
-
-If the `-stats` option is specified, statistical information such as transfer speed, and number of chunks will be displayed throughout the restore procedure.
-
-The `-threads` option can be used to specify more than one thread to download chunks.
-
-The `-limit-rate` option sets a cap on the maximum upload rate.
-
-When the repository can have multiple storages (added by the *add* command), you can select the storage to restore from by specifying the storage name.
-
-Unlike the *backup* procedure that reading the include/exclude patterns from a file, the *restore* procedure reads them from the command line.  If the patterns can cause confusion to the command line argument parser, -- should be prepended to the patterns.  Please refer to the [Include/Exclude Patterns](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#includeexclude-patterns) section for how to specify patterns.
-
-
-#### List
-```
-SYNOPSIS:
-   duplicacy list - List snapshots
-
-USAGE:
-   duplicacy list [command options]
-
-OPTIONS:
-   -all, -a                    list snapshots with any id
-   -id <snapshot id>           list snapshots with the specified id rather than the default one
-   -r <revision> [+]           the revision number of the snapshot
-   -t <tag>                    list snapshots with the specified tag
-   -files                      print the file list in each snapshot
-   -chunks                     print chunks in each snapshot or all chunks if no snapshot specified
-   -reset-passwords            take passwords from input rather than keychain/keyring or env
-   -storage <storage name>     retrieve snapshots from the specified storage
-```
-
-The *list* command lists information about specified snapshots.  By default it will list snapshots created from the current repository, but you can list all snapshots stored in the storage by specifying the -all option, or list snapshots with a different snapshot id using the `-id` option, and/or snapshots with a particular tag with the `-t` option.
-
-The revision number is a number assigned to the snapshot when it is being created.  This number will keep increasing every time a new snapshot is created from a repository.  You can refer to snapshots by their revision numbers using the `-r` option, which either takes a single revision number `-r 123` or a range `-r 123-456`. There can be multiple `-r` options.
-
-If `-files` is specified, for each snapshot to be listed, this command will also print information about every file contained in the snapshot.
-
-If `-chunks` is specified, the command will also print out every chunk the snapshot references.
-
-The `-reset-password` option is used to reset stored passwords and to allow passwords to be entered again.  Please refer to the [Managing Passwords](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#managing-passwords) section for more information.
-
-When the repository can have multiple storages (added by the *add* command), you can specify the storage to list by specifying the storage name.
-
-#### Check
-```
-SYNOPSIS:
-   duplicacy check - Check the integrity of snapshots
-
-USAGE:
-   duplicacy check [command options]
-
-OPTIONS:
-   -all, -a                 check snapshots with any id
-   -id <snapshot id>        check snapshots with the specified id rather than the default one
-   -r <revision> [+]        the revision number of the snapshot
-   -t <tag>                 check snapshots with the specified tag
-   -fossils                 search fossils if a chunk can't be found
-   -resurrect               turn referenced fossils back into chunks
-   -files                   verify the integrity of every file
-   -stats                   show deduplication statistics (imply -all and all revisions)
-   -storage <storage name>  retrieve snapshots from the specified storage```
-```
-The *check* command checks, for each specified snapshot, that all referenced chunks exist in the storage.
-
-By default the *check* command will check snapshots created from the
-current repository, but you can check all snapshots stored in the storage at once by specifying the `-all` option, or snapshots from a different repository using the `-id` option, and/or snapshots with a particular tag with the `-t` option.
-
-The revision number is a number assigned to the snapshot when it is being created.  This number will keep increasing every time a new snapshot is created from a repository.  You can refer to snapshots by their revision numbers using the `-r` option, which either takes a single revision number `-r 123` or a range `-r 123-456`.  There can be multiple `-r` options.
-
-By default the *check* command only verifies the existence of chunks.  To verify the full integrity of a snapshot, you should specify the `-files` option, which will download chunks and compute file hashes in memory, to make sure that all hashes match.
-
-By default the *check* command does not find fossils. If the `-fossils` option is specified, it will find the fossil if the referenced chunk does not exist.  if the `-resurrect` option is specified, it will turn the fossil back into a chunk.
-
-When the repository can have multiple storages (added by the *add* command), you can specify the storage to check by specifying the storage name.
-
-
-#### Cat
-```
-SYNOPSIS:
-   duplicacy cat - Print to stdout the specified file, or the snapshot content if no file is specified
-
-USAGE:
-   duplicacy cat [command options] [<file>]
-
-OPTIONS:
-   -id <snapshot id>        retrieve from the snapshot with the specified id
-   -r <revision>            the revision number of the snapshot
-   -storage <storage name>  retrieve the file from the specified storage
-```
-
-The *cat* command prints a file or the entire snapshot content if no file is specified.
-
-The file must be specified with a path relative to the repository.
-
-You can specify a different snapshot id rather than the default id.
-
-The `-r` option is optional.  If not specified, the latest revision will be selected.
-
-You can use the `-storage` option to select a different storage other than the default one.
-
-#### Diff
-```
-SYNOPSIS:
-   duplicacy diff - Compare two snapshots or two revisions of a file
-
-USAGE:
-   duplicacy diff [command options] [<file>]
-
-OPTIONS:
-   -id <snapshot id>        diff with the snapshot with the specified id
-   -r <revision> [+]        the revision number of the snapshot
-   -hash                    compute the hashes of on-disk files
-   -storage <storage name>  retrieve files from the specified storage
-```
-The *diff* command compares the same file in two different snapshots if a file is given, otherwise compares the two snapshots.
-
-The file must be specified with a path relative to the repository.
-
-You can specify a different snapshot id rather than the default snapshot id.
-
-If only one revision is given by `-r`, the right hand side of the comparison will be the on-disk file.  The `-hash` option can then instruct this command to compute the hash of the file.
-
-You can use the `-storage` option to select a different storage other than the default one.
-
-#### History
-```
-SYNOPSIS:
-   duplicacy history - Show the history of a file
-
-USAGE:
-   duplicacy history [command options] <file>
-
-OPTIONS:
-   -id <snapshot id>        find the file in the snapshot with the specified id
-   -r <revision> [+]        show history of the specified revisions
-   -hash                    show the hash of the on-disk file
-   -storage <storage name>  retrieve files from the specified storage
-```
-
-The *history* command shows how the hash, size, and timestamp of a file change over the specified set of revisions.
-
-You can specify a different snapshot id rather than the default snapshot id, and multiple `-r` options to specify the set of revisions.
-
-The `-hash` option is to compute the hash of the on-disk file.  Otherwise, only the size and timestamp of the on-disk file will be included.
-
-You can use the `-storage` option to select a different storage other than the default one.
-
-#### Prune
-```
-SYNOPSIS:
-   duplicacy prune - Prune snapshots by revision, tag, or retention policy
-
-USAGE:
-   duplicacy prune [command options]
-
-OPTIONS:
-   -id <snapshot id>        delete snapshots with the specified id instead of the default one
-   -all, -a                 match against all snapshot IDs
-   -r <revision> [+]        delete snapshots with the specified revisions
-   -t <tag> [+]             delete snapshots with the specified tags
-   -keep <n:m> [+]          keep 1 snapshot every n days for snapshots older than m days
-   -exhaustive              find all unreferenced chunks by scanning the storage
-   -exclusive               assume exclusive access to the storage (disable two-step fossil collection)
-   -dry-run, -d             show what would have been deleted
-   -delete-only             delete fossils previously collected (if deletable) and don't collect fossils
-   -collect-only            identify and collect fossils, but don't delete fossils previously collected
-   -ignore <id> [+]         ignore the specified snapshot id when deciding if fossils can be deleted
-   -storage <storage name>  prune snapshots from the specified storage
-```
-
-The *prune* command implements the two-step fossil collection algorithm.  It will first find fossil collection files from previous runs and check if contained fossils are eligible for permanent deletion (the fossil deletion step).  Then it will search for snapshots to be deleted, mark unreferenced chunks as fossils (by renaming) and save them in a new fossil collection file stored locally (the fossil collection step).
-
-If a snapshot id is specified, that snapshot id will be used instead of the default one.  The `-a` option will find snapshots with any id.  Snapshots to be deleted can be specified by revision numbers, by a tag, by retention policies, or by any combination of them.
-
-The retention policies are specified by the `-keep` option, which accepts an argument in the form of two numbers *n:m*, where *n* indicates the number of days between two consecutive snapshots to keep, and *m* means that the policy only applies to snapshots at least *m* day old.  If *n* is zero, any snapshots older than *m* days will be removed.
-
-Here are a few sample retention policies:
-
-```sh
-$ duplicacy prune -keep 1:7       # Keep 1 snapshot per day for snapshots older than 7 days
-$ duplicacy prune -keep 7:30      # Keep 1 snapshot every 7 days for snapshots older than 30 days
-$ duplicacy prune -keep 30:180    # Keep 1 snapshot every 30 days for snapshots older than 180 days
-$ duplicacy prune -keep 0:360     # Keep no snapshots older than 360 days
-```
-
-Multiple `-keep` options must be sorted by their *m* values in decreasing order.  For instance, to combine the above policies into one line, it would become:
-
-```sh
-$ duplicacy prune -keep 0:360 -keep 30:180 -keep 7:30 -keep 1:7
-```
-
-The `-exhaustive` option will scan the list of all chunks in the storage, therefore it will find not only unreferenced chunks from deleted snapshots, but also chunks that become unreferenced for other reasons, such as those from an incomplete backup.  It will also find any file that does not look like a chunk file. In contrast, a default *prune* command will only identify
-chunks referenced by deleted snapshots but not any other snapshots.
-
-The `-exclusive` option will assume that no other clients are accessing the storage, effectively disabling the *two-step fossil collection* algorithm.  With this option, the *prune* command will immediately remove unreferenced chunks.
-
-The `-dry-run` option is used to test what changes the *prune* command would have done.  It is guaranteed not to make any changes on the storage, not even creating the local fossil collection file.  The following command checks if the chunk directory is clean (i.e., if there are any unreferenced chunks, temporary files, or anything else):
-
-```
-$ duplicacy prune -d -exclusive -exhaustive    #  Prints out nothing if the chunk directory is clean
-```
-
-The `-delete-only` option will skip the fossil collection step, while the `-collect-only` option will skip the fossil deletion step.
-
-For fossils collected in the fossil collection step to be eligible for safe deletion in the fossil deletion step, at least one new snapshot from *each* snapshot id must be created between two runs of the *prune* command.  However, some repository may not be set up to back up with a regular schedule, and thus literally blocking other repositories from deleting any fossils.  Duplicacy by default will ignore repositories that have no new backup in the past 7 days.  It also provide an `-ignore` option that can be used to skip certain repositories when deciding the deletion criteria.
-
-You can use the `-storage` option to select a different storage other than the default one.
-
-
-#### Password
-```
-SYNOPSIS:
-   duplicacy password - Change the storage password
-
-USAGE:
-   duplicacy password [command options]
-
-OPTIONS:
-   -storage <storage name>  change the password used to access the specified storage
-```
-
-The *password* command decrypts the storage configuration file *config* using the old password, and re-encrypts the file
-using a new password.  It does not change all the encryption keys used to encrypt and decrypt chunk files,
-snapshot files, etc.
-
-You can specify the storage to change the password for when working with multiple storages.
-
-
-#### Add
-```
-SYNOPSIS:
-   duplicacy add - Add an additional storage to be used for the existing repository
-
-USAGE:
-   duplicacy add [command options] <storage name> <snapshot id> <storage url>
-
-OPTIONS:
-   -encrypt, -e                    Encrypt the storage with a password
-   -chunk-size, -c 4M              the average size of chunks
-   -max-chunk-size, -max 16M       the maximum size of chunks (defaults to chunk-size * 4)
-   -min-chunk-size, -min 1M        the minimum size of chunks (defaults to chunk-size / 4)
-   -compression-level, -l <level>  compression level (defaults to -1)
-   -copy <storage name>            make the new storage copy-compatible with an existing one
-```
-
-The *add* command connects another storage to the current repository.  Like the *init* command, if the storage has not been initialized before, a storage configuration file derived from the command line options will be uploaded, but those options will be ignored if the configuration file already exists in the storage.
-
-A unique storage name must be given in order to distinguish it from other storages.
-
-The `-copy` option is required if later you want to copy snapshots between this storage and another storage. Two storages are copy-compatible if they have the same average chunk size, the same maximum chunk size, the same minimum chunk size, the same chunk seed (used in calculating the rolling hash in the variable-size chunks algorithm), and the same hash key.  If the `-copy` option is specified, these parameters will be copied from the existing storage rather than from the command line.
-
-#### Set
-```
-SYNOPSIS:
-   duplicacy set - Change the options for the default or specified storage
-
-USAGE:
-   duplicacy set [command options]
-
-OPTIONS:
-   -encrypt, e[=true]       encrypt the storage with a password
-   -no-backup[=true]        backup to this storage is prohibited
-   -no-restore[=true]       restore from this storage is prohibited
-   -no-save-password[=true] don't save password or access keys to keychain/keyring
-   -key                     add a key/password whose value is supplied by the -value option
-   -value                   the value of the key/password
-   -storage <storage name>  use the specified storage instead of the default one
-```
-
-The *set* command changes the options for the specified storage.
-
-The `-e` option turns on the storage encryption.  If specified as `-e=false`, it turns off the storage encryption.
-
-The `-no-backup` option will not allow backups from this repository to be created.
-
-The `-no-restore` option will not allow restoring this repository to a different revision.
-
-The `-no-save-password` option will require every password or token to be entered every time and not saved anywhere.
-
-The `-key` and `-value` options are used to store (in plain text) access keys or tokens need by various storages.  Please refer to the [Managing Passwords](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#managing-passwords) section for more details.
-
-You can select a storage to change options for by specifying a storage name.
-
-
-#### Copy
-```
-SYNOPSIS:
-   duplicacy copy - Copy snapshots between compatible storages
-
-USAGE:
-   duplicacy copy [command options]
-
-OPTIONS:
-   -id <snapshot id>     copy snapshots with the specified id instead of all snapshot ids
-   -r <revision> [+]     copy snapshots with the specified revisions
-   -from <storage name>  copy snapshots from the specified storage
-   -to <storage name>    copy snapshots to the specified storage
-```
-
-The *copy* command copies snapshots from one storage to another storage.  They must be copy-compatible, i.e., some configuration parameters must be the same.  One storage must be initialized with the `-copy` option provided by the *add* command.
-
-Instead of copying all snapshots, you can specify a set of snapshots to copy by giving the `-r` options.  The *copy* command preserves the revision numbers, so if a revision number already exists on the destination storage the command will fail.
-
-If no `-from` option is given, the snapshots from the default storage will be copied.  The `-to` option specified the destination storage and is required.
-
-## Include/Exclude Patterns
-
-An include pattern starts with +, and an exclude pattern starts with -.  Patterns may contain wildcard characters * which matches a path string of any length, and ? matches a single character.  Note that both * and ? will match any character including the path separator /.
-
-The path separator is always /, even on Windows.
-
-When matching a path against a list of patterns, the path is compared with the part after + or -, one pattern at a time.  Therefore, the order of the patterns is significant.  If a match with an include pattern is found, the path is said to be included without further comparisons.  If a match with an exclude pattern is found, the path is said to be excluded without further comparison.  If a match is not found, the path will be excluded if all patterns are include patterns, but included otherwise.
-
-Patterns ending with a / apply to directories only, and patterns not ending with a / apply to files only.  Patterns ending with * and ?, however, apply to both directories and files.  When a directory is excluded, all files and subdirectories under it will also be excluded.  Therefore, to include a subdirectory, all parent directories must be explicitly included.  For instance, the following pattern list doesn't do what is intended, since the `foo` directory will be excluded so the `foo/bar` will never be visited:
-
-```
-+foo/bar/*
--*
-```
-
-The correct way is to include `foo` as well:
-
-```
-+foo/bar/*
-+foo/
--*
-```
-
-The following pattern list includes only files under the directory foo/ but not files under the subdirectory foo/bar:
-
-```
--foo/bar/
-+foo/*
--*
-```
-
-For the *backup* command, the include/exclude patterns are read from a file named *filters* under the *.duplicacy* directory.
-
-For the *restore* command, the include/exclude patterns are specified as the command line arguments.
-
-
-## Managing Passwords
-
-Duplicacy will attempt to retrieve in three ways the storage password and the storage-specific access tokens/keys.
-
-* If a secret vault service is available, Duplicacy will store passwords/keys entered by the user in such a secret vault and later retrieve them when needed.  On Mac OS X it is Keychain, and on Linux it is gnome-keyring.  On Windows the passwords/keys are encrypted and decrypted by the Data Protection API, and encrypted passwords/keys are stored in the file *.duplicacy/keyring*.  However, if the -no-save-password option is specified for the storage, then Duplicacy will not save passwords this way.
-* If an environment variable for a password is provided, Duplicacy will always take it.  The table below shows the name of the environment variable for each kind of password.  Note that if the storage is not the default one, the storage name will be included in the name of the environment variable (in uppercase). For example, if your storage name is b2, then the environment variable should be named DUPLICACY_B2_PASSWORD.
-* If a matching key and its value are saved to the preference file (.duplicacy/preferences) by the *set* command, the value will be used as the password.  The last column in the table below lists the name of the preference key for each type of password.
-
-| password type | environment variable (default storage) | environment variable (non-default storage in uppercase) | key in preferences  |
-|:----------------:|:----------------:|:----------------:|:----------------:|
-| storage password | DUPLICACY_PASSWORD | DUPLICACY_&lt;STORAGENAME&gt;_PASSWORD | password |
-| sftp password | DUPLICACY_SSH_PASSWORD | DUPLICACY_&lt;STORAGENAME&gt;_SSH_PASSWORD | ssh_password |
-| sftp key file | DUPLICACY_SSH_KEY_FILE | DUPLICACY_&lt;STORAGENAME&gt;_SSH_KEY_FILE | ssh_key_file |
-| Dropbox Token | DUPLICACY_DROPBOX_TOKEN | DUPLICACY_&lt;STORAGENAME>&gt;_DROPBOX_TOKEN | dropbox_token |
-| S3 Access ID | DUPLICACY_S3_ID | DUPLICACY_&lt;STORAGENAME&gt;_S3_ID | s3_id |
-| S3 Secret Key | DUPLICACY_S3_SECRET | DUPLICACY_&lt;STORAGENAME&gt;_S3_SECRET | s3_secret |
-| BackBlaze Account ID | DUPLICACY_B2_ID | DUPLICACY_&lt;STORAGENAME&gt;_B2_ID | b2_id |
-| Backblaze Application Key | DUPLICACY_B2_KEY | DUPLICACY_&lt;STORAGENAME&gt;_B2_KEY | b2_key |
-| Azure Access Key | DUPLICACY_AZURE_KEY | DUPLICACY_&lt;STORAGENAME&gt;_AZURE_KEY | azure_key |
-| Google Drive Token File | DUPLICACY_GCD_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_GCD_TOKEN | gcd_token |
-| Microsoft OneDrive Token File | DUPLICACY_ONE_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_ONE_TOKEN | one_token |
-| Hubic Token File | DUPLICACY_HUBIC_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_HUBIC_TOKEN | hubic_token |
-
-Note that the passwords stored in the environment variable and the preference need to be in plaintext and thus are insecure and should be avoided whenever possible.
-
-## Cache
-
-Duplicacy maintains a local cache under the `.duplicacy/cache` folder in the repository.  Only snapshot chunks may be stored in this local cache, and file chunks are never cached.
-
-At the end of a backup operation, Duplicacy will clean up the local cache in such a way that only chunks composing the snapshot file from the last backup will stay in the cache.  All other chunks will be removed from the cache.  However, if the *prune* command has been run before (which will leave a the `.duplicacy/collection` folder in the repository, then the *backup* command won't perform any cache cleanup and instead defer that to the *prune* command.
-
-At the end of a prune operation, Duplicacy will remove all chunks from the local cache except those composing the snapshot file from the last backup (those that would be kept by the *backup* command), as well as chunks that contain information about chunks referenced by *all* backups from *all* repositories connected to the same storage url.
-
-Other commands, such as *list*, *check*, does not clean up the local cache at all, so the local cache may keep growing if many of these commands run consecutively.  However, once a *backup* or a *prune* command is invoked, the local cache should shrink to its normal size.
-
-## Scripts
-
-You can instruct Duplicacy to run a script before or after executing a command.  For example, if you create a bash script with the name *pre-prune* under the *.duplicacy/scripts* directory, this bash script will be run before the *prune* command starts.  A script named *post-prune* will be run after the *prune* command finishes.  This rule applies to all commands except *init*.
+All documentation has been moved to our wiki page:
+
+* Commands
+    * [init](https://github.com/gilbertchen/duplicacy/wiki/init)
+    * [backup](https://github.com/gilbertchen/duplicacy/wiki/backup)
+    * [restore](https://github.com/gilbertchen/duplicacy/wiki/restore)
+    * [list](https://github.com/gilbertchen/duplicacy/wiki/list)
+    * [check](https://github.com/gilbertchen/duplicacy/wiki/check)
+    * [prune](https://github.com/gilbertchen/duplicacy/wiki/prune)
+    * [cat](https://github.com/gilbertchen/duplicacy/wiki/cat)
+    * [history](https://github.com/gilbertchen/duplicacy/wiki/history)
+    * [diff](https://github.com/gilbertchen/duplicacy/wiki/diff)
+    * [password](https://github.com/gilbertchen/duplicacy/wiki/password)
+    * [add](https://github.com/gilbertchen/duplicacy/wiki/add)
+    * [set](https://github.com/gilbertchen/duplicacy/wiki/set)
+    * [copy](https://github.com/gilbertchen/duplicacy/wiki/copy)
+* [Include/Exclude Patterns](https://github.com/gilbertchen/duplicacy/wiki/Include-Exclude-Patterns)
+* [Managing Passwords](https://github.com/gilbertchen/duplicacy/wiki/Managing-Passwords)
+* [Cache](https://github.com/gilbertchen/duplicacy/wiki/Cache)
+* [Pre-Command and Post-Command Scripts](https://github.com/gilbertchen/duplicacy/wiki/Pre-Command-and-Post-Command-Scripts)

Gopkg.lock

@@ -0,0 +1,230 @@
+# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
+
+
+[[projects]]
+  name = "cloud.google.com/go"
+  packages = ["compute/metadata","iam","internal","internal/optional","internal/version","storage"]
+  revision = "2d3a6656c17a60b0815b7e06ab0be04eacb6e613"
+  version = "v0.16.0"
+
+[[projects]]
+  name = "github.com/Azure/azure-sdk-for-go"
+  packages = ["version"]
+  revision = "b7fadebe0e7f5c5720986080a01495bd8d27be37"
+  version = "v14.2.0"
+
+[[projects]]
+  name = "github.com/Azure/go-autorest"
+  packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
+  revision = "0ae36a9e544696de46fdadb7b0d5fb38af48c063"
+  version = "v10.2.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/aryann/difflib"
+  packages = ["."]
+  revision = "e206f873d14a916d3d26c40ab667bca123f365a3"
+
+[[projects]]
+  name = "github.com/aws/aws-sdk-go"
+  packages = ["aws","aws/awserr","aws/awsutil","aws/client","aws/client/metadata","aws/corehandlers","aws/credentials","aws/credentials/ec2rolecreds","aws/credentials/endpointcreds","aws/credentials/stscreds","aws/defaults","aws/ec2metadata","aws/endpoints","aws/request","aws/session","aws/signer/v4","internal/shareddefaults","private/protocol","private/protocol/query","private/protocol/query/queryutil","private/protocol/rest","private/protocol/restxml","private/protocol/xml/xmlutil","service/s3","service/sts"]
+  revision = "a32b1dcd091264b5dee7b386149b6cc3823395c9"
+  version = "v1.12.31"
+
+[[projects]]
+  name = "github.com/bkaradzic/go-lz4"
+  packages = ["."]
+  revision = "74ddf82598bc4745b965729e9c6a463bedd33049"
+  version = "v1.0.0"
+
+[[projects]]
+  name = "github.com/dgrijalva/jwt-go"
+  packages = ["."]
+  revision = "dbeaa9332f19a944acb5736b4456cfcc02140e29"
+  version = "v3.1.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/azure-sdk-for-go"
+  packages = ["storage"]
+  revision = "bbf89bd4d716c184f158d1e1428c2dbef4a18307"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/cli"
+  packages = ["."]
+  revision = "1de0a1836ce9c3ae1bf737a0869c4f04f28a7f98"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/go-dropbox"
+  packages = ["."]
+  revision = "90711b603312b1f973f3a5da3793ac4f1e5c2f2a"
+
+[[projects]]
+  name = "github.com/gilbertchen/go-ole"
+  packages = ["."]
+  revision = "0e87ea779d9deb219633b828a023b32e1244dd57"
+  version = "v1.2.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/go.dbus"
+  packages = ["."]
+  revision = "9e442e6378618c083fd3b85b703ffd202721fb17"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/goamz"
+  packages = ["aws","s3"]
+  revision = "eada9f4e8cc2a45db775dee08a2c37597ce4760a"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/gopass"
+  packages = ["."]
+  revision = "bf9dde6d0d2c004a008c27aaee91170c786f6db8"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/keyring"
+  packages = ["."]
+  revision = "8855f5632086e51468cd7ce91056f8da69687ef6"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/xattr"
+  packages = ["."]
+  revision = "68e7a6806b0137a396d7d05601d7403ae1abac58"
+
+[[projects]]
+  name = "github.com/go-ini/ini"
+  packages = ["."]
+  revision = "32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a"
+  version = "v1.32.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/golang/protobuf"
+  packages = ["proto","protoc-gen-go/descriptor","ptypes","ptypes/any","ptypes/duration","ptypes/timestamp"]
+  revision = "1e59b77b52bf8e4b449a57e6f79f21226d571845"
+
+[[projects]]
+  name = "github.com/googleapis/gax-go"
+  packages = ["."]
+  revision = "317e0006254c44a0ac427cc52a0e083ff0b9622f"
+  version = "v2.0.0"
+
+[[projects]]
+  name = "github.com/jmespath/go-jmespath"
+  packages = ["."]
+  revision = "0b12d6b5"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/kr/fs"
+  packages = ["."]
+  revision = "2788f0dbd16903de03cb8186e5c7d97b69ad387b"
+
+[[projects]]
+  name = "github.com/marstr/guid"
+  packages = ["."]
+  revision = "8bd9a64bf37eb297b492a4101fb28e80ac0b290f"
+  version = "v1.1.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/minio/blake2b-simd"
+  packages = ["."]
+  revision = "3f5f724cb5b182a5c278d6d3d55b40e7f8c2efb4"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/ncw/swift"
+  packages = ["."]
+  revision = "ae9f0ea1605b9aa6434ed5c731ca35d83ba67c55"
+
+[[projects]]
+  name = "github.com/pkg/errors"
+  packages = ["."]
+  revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
+  version = "v0.8.0"
+
+[[projects]]
+  name = "github.com/pkg/sftp"
+  packages = ["."]
+  revision = "98203f5a8333288eb3163b7c667d4260fe1333e9"
+  version = "1.0.0"
+
+[[projects]]
+  name = "github.com/satori/go.uuid"
+  packages = ["."]
+  revision = "f58768cc1a7a7e77a3bd49e98cdd21419399b6a3"
+  version = "v1.2.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/vaughan0/go-ini"
+  packages = ["."]
+  revision = "a98ad7ee00ec53921f08832bc06ecf7fd600e6a1"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/crypto"
+  packages = ["curve25519","ed25519","ed25519/internal/edwards25519","pbkdf2","ssh","ssh/agent","ssh/terminal"]
+  revision = "9f005a07e0d31d45e6656d241bb5c0f2efd4bc94"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/net"
+  packages = ["context","context/ctxhttp","http2","http2/hpack","idna","internal/timeseries","lex/httplex","trace"]
+  revision = "9dfe39835686865bff950a07b394c12a98ddc811"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/oauth2"
+  packages = [".","google","internal","jws","jwt"]
+  revision = "f95fa95eaa936d9d87489b15d1d18b97c1ba9c28"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/sys"
+  packages = ["unix","windows"]
+  revision = "82aafbf43bf885069dc71b7e7c2f9d7a614d47da"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/text"
+  packages = ["collate","collate/build","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
+  revision = "88f656faf3f37f690df1a32515b479415e1a6769"
+
+[[projects]]
+  branch = "master"
+  name = "google.golang.org/api"
+  packages = ["drive/v3","gensupport","googleapi","googleapi/internal/uritemplates","googleapi/transport","internal","iterator","option","storage/v1","transport/http"]
+  revision = "17b5f22a248d6d3913171c1a557552ace0d9c806"
+
+[[projects]]
+  name = "google.golang.org/appengine"
+  packages = [".","internal","internal/app_identity","internal/base","internal/datastore","internal/log","internal/modules","internal/remote_api","internal/urlfetch","urlfetch"]
+  revision = "150dc57a1b433e64154302bdc40b6bb8aefa313a"
+  version = "v1.0.0"
+
+[[projects]]
+  branch = "master"
+  name = "google.golang.org/genproto"
+  packages = ["googleapis/api/annotations","googleapis/iam/v1","googleapis/rpc/status"]
+  revision = "891aceb7c239e72692819142dfca057bdcbfcb96"
+
+[[projects]]
+  name = "google.golang.org/grpc"
+  packages = [".","balancer","balancer/roundrobin","codes","connectivity","credentials","encoding","grpclb/grpc_lb_v1/messages","grpclog","internal","keepalive","metadata","naming","peer","resolver","resolver/dns","resolver/passthrough","stats","status","tap","transport"]
+  revision = "5a9f7b402fe85096d2e1d0383435ee1876e863d0"
+  version = "v1.8.0"
+
+[solve-meta]
+  analyzer-name = "dep"
+  analyzer-version = 1
+  inputs-digest = "eff5ae2d9507f0d62cd2e5bdedebb5c59d64f70f476b087c01c35d4a5e1be72d"
+  solver-name = "gps-cdcl"
+  solver-version = 1

Gopkg.toml

@@ -0,0 +1,94 @@
+
+# Gopkg.toml example
+#
+# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
+# for detailed Gopkg.toml documentation.
+#
+# required = ["github.com/user/thing/cmd/thing"]
+# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
+#
+# [[constraint]]
+#   name = "github.com/user/project"
+#   version = "1.0.0"
+#
+# [[constraint]]
+#   name = "github.com/user/project2"
+#   branch = "dev"
+#   source = "github.com/myfork/project2"
+#
+# [[override]]
+#  name = "github.com/x/y"
+#  version = "2.4.0"
+
+
+[[constraint]]
+  name = "cloud.google.com/go"
+  version = "0.16.0"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/aryann/difflib"
+
+[[constraint]]
+  name = "github.com/aws/aws-sdk-go"
+  version = "1.12.31"
+
+[[constraint]]
+  name = "github.com/bkaradzic/go-lz4"
+  version = "1.0.0"
+
+[[constraint]]
+  name = "github.com/gilbertchen/azure-sdk-for-go"
+  branch = "master"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/cli"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/go-dropbox"
+
+[[constraint]]
+  name = "github.com/gilbertchen/go-ole"
+  version = "1.2.0"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/goamz"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/gopass"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/keyring"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/xattr"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/minio/blake2b-simd"
+
+[[constraint]]
+  name = "github.com/pkg/sftp"
+  version = "1.0.0"
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/crypto"
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/net"
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/oauth2"
+
+[[constraint]]
+  branch = "master"
+  name = "google.golang.org/api"

LICENSE.md

@@ -1,6 +1,8 @@
 Copyright © 2017 Acrosync LLC
 
 * Free for personal use or commercial trial
-* Non-trial commercial use requires per-user licenses available from [duplicacy.com](https://duplicacy.com/customer) at a cost of $20 per year
-* Commercial licenses are not required to restore or manage backups; only the backup command requires a valid commercial license
+* Non-trial commercial use requires per-user CLI licenses available from [duplicacy.com](https://duplicacy.com/buy) at a cost of $20 per year
+* A user is defined as the computer account that creates or edits the files to be backed up; if a backup contains files created or edited by multiple users for commercial purposes, one CLI license is required for each user
+* The computer with a valid commercial license for the GUI version may run the CLI version without a CLI license
+* CLI licenses are not required to restore or manage backups; only the backup command requires valid CLI licenses
 * Modification and redistribution are permitted, but commercial use of derivative works is subject to the same requirements of this license

README.md

@@ -1,6 +1,6 @@
 # Duplicacy: A lock-free deduplication cloud backup tool
 
-Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy-cli/blob/master/DESIGN.md).  It is the only cloud backup tool that allows multiple computers to back up to the same storage simultaneously without using any locks (thus readily amenable to various cloud storage services).
+Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).  It is the only cloud backup tool that allows multiple computers to back up to the same storage simultaneously without using any locks (thus readily amenable to various cloud storage services).
 
 The repository hosts source code, design documents, and binary releases of the command line version.  There is also a Duplicacy GUI frontend built for Windows and Mac OS X available from https://duplicacy.com.
 
@@ -18,246 +18,52 @@ Duplicacy currently supports major cloud storage providers (Amazon S3, Google Cl
 * Concurrent access: multiple clients can back up to the same storage at the same time
 * Snapshot migration: all or selected snapshots can be migrated from one storage to another
 
-The key idea of **Lock-Free Deduplication** can be summarized as follows:
+The key idea of **[Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication)** can be summarized as follows:
 
 * Use variable-size chunking algorithm to split files into chunks
 * Store each chunk in the storage using a file name derived from its hash, and rely on the file system API to manage chunks without using a centralized indexing database
 * Apply a *two-step fossil collection* algorithm to remove chunks that become unreferenced after a backup is deleted
 
-The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/DESIGN.md) explains lock-free deduplication in detail.
-
 ## Getting Started
 
-<details>
-<summary>Installation</summary>
-
-Duplicacy is written in Go.  You can run the following command to build the executable (which will be created under `$GOPATH/bin`):
-
-```
-go get -u github.com/gilbertchen/duplicacy/...
-```
-
-You can also visit the [releases page](https://github.com/gilbertchen/duplicacy-cli/releases/latest) to download the pre-built binary suitable for your platform..
-
-</details>
-
-<details>
-<summary>Commands</summary>
-
-Once you have the Duplicacy executable on your path, you can change to the directory that you want to back up (called *repository*) and run the *init* command:
-
-```
-$ cd path/to/your/repository
-$ duplicacy init mywork sftp://user@192.168.1.100/path/to/storage
-```
-
-This *init* command connects the repository with the remote storage at 192.168.1.00 via SFTP.  It will initialize the remote storage if this has not been done before.  It also assigns the snapshot id *mywork* to the repository.  This snapshot id is used to uniquely identify this repository if there are other repositories that also back up to the same storage.
-
-You can now create snapshots of the repository by invoking the *backup* command.  The first snapshot may take a while depending on the size of the repository and the upload bandwidth.  Subsequent snapshots will be much faster, as only new or modified files will be uploaded.  Each snapshot is identified by the snapshot id and an increasing revision number starting from 1.
-
-```sh
-$ duplicacy backup -stats
-```
-
-The *restore* command rolls back the repository to a previous revision:
-```sh
-$ duplicacy restore -r 1
-```
-
-
-
-Duplicacy provides a set of commands, such as list, check, diff, cat history, to manage snapshots:
-
-
-```makefile
-$ duplicacy list            # List all snapshots
-$ duplicacy check           # Check integrity of snapshots
-$ duplicacy diff            # Compare two snapshots, or the same file in two snapshots
-$ duplicacy cat             # Print a file in a snapshot
-$ duplicacy history         # Show how a file changes over time
-```
-
-
-The *prune* command removes snapshots by revisions, or tags, or retention policies:
-
-```sh
-$ duplicacy prune -r 1            # Remove the snapshot with revision number 1
-$ duplicacy prune -t quick        # Remove all snapshots with the tag 'quick'
-$ duplicacy prune -keep 1:7       # Keep 1 snapshot per day for snapshots older than 7 days
-$ duplicacy prune -keep 7:30      # Keep 1 snapshot every 7 days for snapshots older than 30 days
-$ duplicacy prune -keep 0:180     # Remove all snapshots older than 180 days
-```
-
-The first time the *prune* command is called, it removes the specified snapshots but keeps all unreferenced chunks as fossils.
-Since it uses the two-step fossil collection algorithm to clean chunks, you will need to run it again to remove those fossils from the storage:
-
-```sh
-$ duplicacy prune           # Chunks from deleted snapshots will be removed if deletion criteria are met
-```
-
-To back up to multiple storages, use the *add* command to add a new storage.  The *add* command is similar to the *init* command, except that the first argument is a storage name used to distinguish different storages:
-
-```sh
-$ duplicacy add s3 mywork s3://amazon.com/mybucket/path/to/storage
-```
-
-You can back up to any storage by specifying the storage name:
-
-```sh
-$ duplicacy backup -storage s3
-```
-
-However, snapshots created this way will be different on different storages, if the repository has been changed during two backup operations.  A better approach, is to use the *copy* command to copy specified snapshots from one storage to another:
-
-```sh
-$ duplicacy copy -r 1 -to s3   # Copy snapshot at revision 1 to the s3 storage
-$ duplicacy copy -to s3        # Copy every snapshot to the s3 storage
-```
-
-</details>
-
-
-The [User Guide](https://github.com/gilbertchen/duplicacy-cli/blob/master/GUIDE.md) contains a complete reference to
-all commands and other features of Duplicacy.
-
+* [A brief introduction](https://github.com/gilbertchen/duplicacy/wiki/Quick-Start)
+* [Command references](https://github.com/gilbertchen/duplicacy/wiki)
 
 ## Storages
 
-Duplicacy currently supports local file storage, SFTP, and many cloud storage providers.
+With Duplicacy, you can back up files to local or networked drives, SFTP servers, or many cloud storage providers.  The following table compares the costs of all cloud storages supported by Duplicacy.
 
-<details> <summary>Local disk</summary>
+| Type         |   Storage (monthly)    |   Upload           |    Download    |    API Charge   |
+|:------------:|:-------------:|:------------------:|:--------------:|:-----------:|
+| Amazon S3    | $0.023/GB | free | $0.090/GB | [yes](https://aws.amazon.com/s3/pricing/) |
+| Wasabi       | $3.99 first 1TB <br> $0.0039/GB additional | free | $0.04/GB | no |
+| DigitalOcean Spaces| $5 first 250GB <br> $0.020/GB additional | free | first 1TB free <br> $0.01/GB additional| no |
+| Backblaze B2 | 10GB free <br> $0.005/GB | free | 1GB free/day <br> $0.02/GB | [yes](https://www.backblaze.com/b2/b2-transactions-price.html) |
+| Google Cloud Storage| $0.026/GB | free |$ 0.12/GB | [yes](https://cloud.google.com/storage/pricing) |
+| Google Drive | 15GB free <br> $1.99/100GB <br> $9.99/TB | free | free | no |
+| Microsoft Azure | $0.0184/GB | free | free | [yes](https://azure.microsoft.com/en-us/pricing/details/storage/blobs/) |
+| Microsoft OneDrive | 5GB free <br> $1.99/50GB <br> $5.83/TB | free | free | no |
+| Dropbox | 2GB free <br> $8.25/TB | free | free | no |
+| Hubic | 25GB free <br> €1/100GB <br> €5/10TB | free | free | no |
 
-```
-Storage URL:  /path/to/storage (on Linux or Mac OS X)
-              C:\path\to\storage (on Windows)
-```
-</details>
+Please consult the [wiki page](https://github.com/gilbertchen/duplicacy/wiki/Storage-Backends) on how to set up Duplicacy to work with each cloud storage.
 
-<details> <summary>SFTP</summary>
+It should be noted that their performances vary a lot.  A [performance comparison](https://github.com/gilbertchen/cloud-storage-comparison) of these storages measured the running times (in seconds) of backing up and restoring the [Linux code base](https://github.com/torvalds/linux) as follows:
 
-```
-Storage URL:  sftp://username@server/path/to/storage (path relative to the home directory)
-              sftp://username@server//path/to/storage (absolute path)
-```
+| Storage              | initial backup | 2nd | 3rd | initial restore | 2nd | 3rd |
+|:--------------------:|:------:|:----:|:-----:|:----:|:-----:|:----:|
+| SFTP                 |  31.5  | 6.6  | 20.6  | 22.5  | 7.8  | 18.4 |
+| Amazon S3            |  41.1  | 5.9  | 21.9  | 27.7  | 7.6  | 23.5 |
+| Wasabi               |  38.7  | 5.7  | 31.7  | 25.7  | 6.5  | 23.2 | 
+| DigitalOcean Spaces  |  51.6  | 7.1  | 31.7  | 29.3  | 6.4  | 27.6 | 
+| Backblaze B2         |  106.7 | 24.0 | 88.2  | 67.9  | 14.4 | 39.1 | 
+| Google Cloud Storage |  76.9  | 11.9 | 33.1  | 39.5  | 9.9  | 26.2 | 
+| Google Drive         |  139.3 | 14.7 | 45.2  | 129.4 | 17.8 | 54.4 |
+| Microsoft Azure      |  35.0  | 5.4  | 20.4  | 30.7  | 7.1  | 21.5 | 
+| Microsoft OneDrive   |  250.0 | 31.6 | 80.2  | 333.4 | 26.2 | 82.0 | 
+| Dropbox              |  267.2 | 35.8 | 113.7 | 164.0 | 31.6 | 80.3 |
 
-Login methods include password authentication and public key authentication.  Due to a limitation of the underlying Go SSH library, the key pair for public key authentication must be generated without a passphrase.  To work with a key that has a passphrase, you can set up SSH agent forwarding which is also supported by Duplicacy.
-
-</details>
-
-<details> <summary>Dropbox</summary>
-
-```
-Storage URL:  dropbox://path/to/storage
-```
-
-For Duplicacy to access your Dropbox storage, you must provide an access token that can be obtained in one of two ways:
-
-* Create your own app on the [Dropbox Developer](https://www.dropbox.com/developers) page, and then generate the [access token](https://blogs.dropbox.com/developers/2014/05/generate-an-access-token-for-your-own-account/)
-
-* Or authorize Duplicacy to access its app folder inside your Dropbox (following [this link](https://dl.dropboxusercontent.com/u/95866350/start_dropbox_token.html)), and Dropbox will generate the access token (which is not visible to us, as the redirect page showing the token is merely a static html hosted by Dropbox)
-
-Dropbox has two advantages over other cloud providers.  First, if you are already a paid user then to use the unused space as the backup storage is basically free.  Second, unlike other providers Dropbox does not charge bandwidth or API usage fees.
-
-</details>
-
-<details> <summary>Amazon S3</summary>
-
-```
-Storage URL:  s3://amazon.com/bucket/path/to/storage (default region is us-east-1)
-              s3://region@amazon.com/bucket/path/to/storage (other regions must be specified)
-```
-
-You'll need to input an access key and a secret key to access your Amazon S3 storage.
-
-Minio-based S3 compatiable storages are also supported by using the `minio` or `minios` backends:
-```
-Storage URL:  minio://region@host/bucket/path/to/storage (without TLS)
-Storage URL:  minios://region@host/bucket/path/to/storage (with TLS)
-```
-
-There is another backend that works with S3 compatible storage providers that require V2 signing:
-```
-Storage URL:  s3c://region@host/bucket/path/to/storage
-```
-
-</details>
-
-<details>  <summary>Wasabi</summary>
-
-```
-Storage URL: s3://us-east-1@s3.wasabisys.com/bucket/path/to/storage
-```
-
-[Wasabi](https://wasabi.com) is a relatively new cloud storage service providing a S3-compatible API.
-It is well suited for storing backups, because it is much cheaper than Amazon S3 with a storage cost of $.0039/GB/Month and a download fee of $0.04/GB, and no additional charges on API calls.
-
-</details>
-
-<details>  <summary>Google Cloud Storage</summary>
-
-```
-Storage URL:  gcs://bucket/path/to/storage
-```
-
-Starting from version 2.0.0, a new Google Cloud Storage backend is added which is implemented using the [official Google client library](https://godoc.org/cloud.google.com/go/storage).  You must first obtain a credential file by [authorizing](https://duplicacy.com/gcp_start) Duplicacy to access your Google Cloud Storage account or by [downloading](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts) a service account credential file.
-
-You can also use the s3 protocol to access Google Cloud Storage.  To do this, you must enable the [s3 interoperability](https://cloud.google.com/storage/docs/migrating#migration-simple) in your Google Cloud Storage settings and set the storage url as `s3://storage.googleapis.com/bucket/path/to/storage`.
-
-</details>
-
-<details> <summary>Microsoft Azure</summary>
-
-```
-Storage URL:  azure://account/container
-```
-
-You'll need to input the access key once prompted.
-
-</details>
-
-<details> <summary>Backblaze B2</summary>
-
-```
-Storage URL: b2://bucket
-```
-
-You'll need to input the account id and application key.
-
-Backblaze's B2 storage is one of the least expensive (at 0.5 cent per GB per month, with a download fee of 2 cents per GB, plus additional charges for API calls).
-
-</details>
-
-<details> <summary>Google Drive</summary>
-
-```
-Storage URL: gcd://path/to/storage
-```
-
-To use Google Drive as the storage, you first need to download a token file from https://duplicacy.com/gcd_start by authorizing Duplicacy to access your Google Drive, and then enter the path to this token file to Duplicacy when prompted.
-
-</details>
-
-<details> <summary>Microsoft OneDrive</summary>
-
-```
-Storage URL: one://path/to/storage
-```
-
-To use Microsoft OneDrive as the storage, you first need to download a token file from https://duplicacy.com/one_start by authorizing Duplicacy to access your OneDrive, and then enter the path to this token file to Duplicacy when prompted.
-
-</details>
-
-<details> <summary>Hubic</summary>
-
-```
-Storage URL: hubic://path/to/storage
-```
-
-To use Hubic as the storage, you first need to download a token file from https://duplicacy.com/hubic_start by authorizing Duplicacy to access your Hubic drive, and then enter the path to this token file to Duplicacy when prompted.
-
-Hubic offers the most free space (25GB) of all major cloud providers and there is no bandwidth charge (same as Google Drive and OneDrive), so it may be worth a try.
-
-</details>
+For more details please visit https://github.com/gilbertchen/cloud-storage-comparison.
 
 ## Feature Comparison with Other Backup Tools
 
@@ -321,6 +127,8 @@ For more details and other speed comparison results, please visit https://github
 ## License
 
 * Free for personal use or commercial trial
-* Non-trial commercial use requires per-user licenses available from [duplicacy.com](https://duplicacy.com/customer) at a cost of $20 per year
-* Commercial licenses are not required to restore or manage backups; only the backup command requires a valid commercial license
+* Non-trial commercial use requires per-user CLI licenses available from [duplicacy.com](https://duplicacy.com/buy) at a cost of $20 per year
+* A user is defined as the computer account that creates or edits the files to be backed up; if a backup contains files created or edited by multiple users for commercial purposes, one CLI license is required for each user
+* The computer with a valid commercial license for the GUI version may run the CLI version without a CLI license
+* CLI licenses are not required to restore or manage backups; only the backup command requires valid CLI licenses
 * Modification and redistribution are permitted, but commercial use of derivative works is subject to the same requirements of this license

integration_tests/copy_test.sh

@@ -16,6 +16,16 @@ add_file file3
 add_file file4
 ${DUPLICACY} backup 
 ${DUPLICACY} copy -from default -to secondary
-${DUPLICACY} check --files -stats
+${DUPLICACY} check --files -stats -storage default
 ${DUPLICACY} check --files -stats -storage secondary
+# Prune revisions from default storage
+${DUPLICACY} -d -v -log prune -r 1-2 -exclusive -exhaustive -storage default
+# Copy snapshot revisions from secondary back to default
+${DUPLICACY} copy -from secondary -to default
+# Check snapshot revisions again to make sure we're ok!
+${DUPLICACY} check --files -stats -storage default
+${DUPLICACY} check --files -stats -storage secondary
+# Check for orphaned or missing chunks
+${DUPLICACY} prune -exhaustive -exclusive -storage default
+${DUPLICACY} prune -exhaustive -exclusive -storage secondary
 popd

src/duplicacy_acdclient.go

@@ -5,16 +5,16 @@
 package duplicacy
 
 import (
-    "fmt"
-    "time"
 	"bytes"
-    "sync"
-    "io/ioutil"
 	"encoding/json"
+	"fmt"
 	"io"
-    "net/http"
-    "mime/multipart"
+	"io/ioutil"
 	"math/rand"
+	"mime/multipart"
+	"net/http"
+	"sync"
+	"time"
 
 	"golang.org/x/oauth2"
 )
@@ -29,6 +29,7 @@ func (err ACDError) Error() string {
 }
 
 var ACDRefreshTokenURL = "https://duplicacy.com/acd_refresh"
+
 type ACDClient struct {
 	HTTPClient *http.Client
 
@@ -42,7 +43,6 @@ type ACDClient struct {
 	TestMode bool
 }
 
-
 func NewACDClient(tokenFile string) (*ACDClient, error) {
 
 	description, err := ioutil.ReadFile(tokenFile)
@@ -69,7 +69,7 @@ func NewACDClient(tokenFile string) (*ACDClient, error) {
 
 func (client *ACDClient) call(url string, method string, input interface{}, contentType string) (io.ReadCloser, int64, error) {
 
-    LOG_DEBUG("ACD_CALL", "Calling %s", url)
+	//LOG_DEBUG("ACD_CALL", "%s %s", method, url)
 
 	var response *http.Response
 
@@ -106,7 +106,7 @@ func (client *ACDClient) call(url string, method string, input interface{}, cont
 
 		if url != ACDRefreshTokenURL {
 			client.TokenLock.Lock()
-            request.Header.Set("Authorization", "Bearer " + client.Token.AccessToken)
+			request.Header.Set("Authorization", "Bearer "+client.Token.AccessToken)
 			client.TokenLock.Unlock()
 		}
 		if contentType != "" {
@@ -126,20 +126,20 @@ func (client *ACDClient) call(url string, method string, input interface{}, cont
 			buffer := new(bytes.Buffer)
 			buffer.ReadFrom(response.Body)
 			response.Body.Close()
-            return nil, 0, ACDError { Status: response.StatusCode, Message: buffer.String()}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: buffer.String()}
 		}
 
 		if response.StatusCode == 400 {
 			defer response.Body.Close()
 
-            e := &ACDError {
+			e := &ACDError{
 				Status: response.StatusCode,
 			}
 
 			if err := json.NewDecoder(response.Body).Decode(e); err == nil {
 				return nil, 0, e
 			} else {
-                return nil, 0, ACDError { Status: response.StatusCode, Message: "Bad input parameter"}
+				return nil, 0, ACDError{Status: response.StatusCode, Message: "Bad input parameter"}
 			}
 		}
 
@@ -148,7 +148,7 @@ func (client *ACDClient) call(url string, method string, input interface{}, cont
 		if response.StatusCode == 401 {
 
 			if url == ACDRefreshTokenURL {
-                return nil, 0, ACDError { Status: response.StatusCode, Message: "Unauthorized"}
+				return nil, 0, ACDError{Status: response.StatusCode, Message: "Unauthorized"}
 			}
 
 			err = client.RefreshToken()
@@ -158,15 +158,15 @@ func (client *ACDClient) call(url string, method string, input interface{}, cont
 
 			continue
 		} else if response.StatusCode == 403 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Forbidden"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Forbidden"}
 		} else if response.StatusCode == 404 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Resource not found"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Resource not found"}
 		} else if response.StatusCode == 409 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Conflict"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Conflict"}
 		} else if response.StatusCode == 411 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Length required"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Length required"}
 		} else if response.StatusCode == 412 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Precondition failed"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Precondition failed"}
 		} else if response.StatusCode == 429 || response.StatusCode == 500 {
 			reason := "Too many requests"
 			if response.StatusCode == 500 {
@@ -178,9 +178,9 @@ func (client *ACDClient) call(url string, method string, input interface{}, cont
 			backoff *= 2
 			continue
 		} else if response.StatusCode == 503 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Service unavailable"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Service unavailable"}
 		} else {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Unknown error"}
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Unknown error"}
 		}
 	}
 
@@ -231,7 +231,7 @@ func (client *ACDClient) GetEndpoint() (err error) {
 
 	defer readCloser.Close()
 
-    output := &ACDGetEndpointOutput {}
+	output := &ACDGetEndpointOutput{}
 
 	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 		return err
@@ -256,7 +256,7 @@ type ACDListEntriesOutput struct {
 	Entries   []ACDEntry `json:"data"`
 }
 
-func (client *ACDClient) ListEntries(parentID string, listFiles bool) ([]ACDEntry, error) {
+func (client *ACDClient) ListEntries(parentID string, listFiles bool, listDirectories bool) ([]ACDEntry, error) {
 
 	startToken := ""
 
@@ -264,20 +264,22 @@ func (client *ACDClient) ListEntries(parentID string, listFiles bool) ([]ACDEntr
 
 	for {
 
-        url := client.MetadataURL + "nodes/" + parentID + "/children?filters="
+		url := client.MetadataURL + "nodes/" + parentID + "/children?"
 
-        if listFiles {
-            url += "kind:FILE"
-        } else {
-            url += "kind:FOLDER"
+		if listFiles && !listDirectories {
+			url += "filters=kind:FILE&"
+		} else if !listFiles && listDirectories {
+			url += "filters=kind:FOLDER&"
 		}
 
 		if startToken != "" {
-            url += "&startToken=" + startToken
+			url += "startToken=" + startToken + "&"
 		}
 
 		if client.TestMode {
-            url += "&limit=8"
+			url += "limit=8"
+		} else {
+			url += "limit=200"
 		}
 
 		readCloser, _, err := client.call(url, "GET", 0, "")
@@ -287,7 +289,7 @@ func (client *ACDClient) ListEntries(parentID string, listFiles bool) ([]ACDEntr
 
 		defer readCloser.Close()
 
-        output := &ACDListEntriesOutput {}
+		output := &ACDListEntriesOutput{}
 
 		if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 			return nil, err
@@ -321,7 +323,7 @@ func (client *ACDClient) ListByName(parentID string, name string) (string, bool,
 
 	defer readCloser.Close()
 
-    output := &ACDListEntriesOutput {}
+	output := &ACDListEntriesOutput{}
 
 	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 		return "", false, 0, err
@@ -351,7 +353,7 @@ func (client *ACDClient) UploadFile(parentID string, name string, content []byte
 	metadata := make(map[string]interface{})
 	metadata["name"] = name
 	metadata["kind"] = "FILE"
-    metadata["parents"] = []string{ parentID }
+	metadata["parents"] = []string{parentID}
 
 	metadataJSON, err := json.Marshal(metadata)
 	if err != nil {
@@ -389,7 +391,7 @@ func (client *ACDClient) UploadFile(parentID string, name string, content []byte
 
 	defer readCloser.Close()
 
-    entry := ACDEntry {}
+	entry := ACDEntry{}
 	if err = json.NewDecoder(readCloser).Decode(&entry); err != nil {
 		return "", err
 	}
@@ -434,7 +436,7 @@ func (client *ACDClient) CreateDirectory(parentID string, name string) (string,
 	parameters := make(map[string]interface{})
 	parameters["name"] = name
 	parameters["kind"] = "FOLDER"
-    parameters["parents"] = []string {parentID}
+	parameters["parents"] = []string{parentID}
 
 	readCloser, _, err := client.call(url, "POST", parameters, "")
 	if err != nil {
@@ -443,7 +445,7 @@ func (client *ACDClient) CreateDirectory(parentID string, name string) (string,
 
 	defer readCloser.Close()
 
-    entry := ACDEntry {}
+	entry := ACDEntry{}
 	if err = json.NewDecoder(readCloser).Decode(&entry); err != nil {
 		return "", err
 	}

src/duplicacy_acdclient_test.go

@@ -5,11 +5,11 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "testing"
 	"crypto/sha256"
 	"encoding/hex"
+	"fmt"
+	"io"
+	"testing"
 
 	crypto_rand "crypto/rand"
 	"math/rand"
@@ -81,7 +81,7 @@ func TestACDClient(t *testing.T) {
 	maxFileSize := 64 * 1024
 
 	for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
+		content := make([]byte, rand.Int()%maxFileSize+1)
 		_, err = crypto_rand.Read(content)
 		if err != nil {
 			t.Errorf("Error generating random content: %v", err)
@@ -103,7 +103,7 @@ func TestACDClient(t *testing.T) {
 		}
 	}
 
-    entries, err := acdClient.ListEntries(test1ID, true)
+	entries, err := acdClient.ListEntries(test1ID, true, false)
 	if err != nil {
 		t.Errorf("Error list randomly generated files: %v", err)
 		return
@@ -117,7 +117,7 @@ func TestACDClient(t *testing.T) {
 		}
 	}
 
-    entries, err = acdClient.ListEntries(test2ID, true)
+	entries, err = acdClient.ListEntries(test2ID, true, false)
 	if err != nil {
 		t.Errorf("Error list randomly generated files: %v", err)
 		return

src/duplicacy_acdstorage.go

@@ -9,10 +9,11 @@ import (
 	"path"
 	"strings"
 	"sync"
+	"time"
 )
 
 type ACDStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	client          *ACDClient
 	idCache         map[string]string
@@ -28,38 +29,40 @@ func CreateACDStorage(tokenFile string, storagePath string, threads int) (storag
 		return nil, err
 	}
 
-    storage = &ACDStorage {
+	storage = &ACDStorage{
 		client:          client,
 		idCache:         make(map[string]string),
 		idCacheLock:     &sync.Mutex{},
 		numberOfThreads: threads,
 	}
 
-    storagePathID, _, _, err := storage.getIDFromPath(0, storagePath)
+	storagePathID, err := storage.getIDFromPath(0, storagePath, false)
 	if err != nil {
 		return nil, err
 	}
 
+	// Set 'storagePath' as the root of the storage and clean up the id cache accordingly
+	storage.idCache = make(map[string]string)
 	storage.idCache[""] = storagePathID
 
-    for _, dir := range []string { "chunks", "fossils", "snapshots" } {
+	for _, dir := range []string{"chunks", "fossils", "snapshots"} {
 		dirID, isDir, _, err := client.ListByName(storagePathID, dir)
 		if err != nil {
 			return nil, err
 		}
 		if dirID == "" {
-            dirID, err = client.CreateDirectory(storagePathID, dir)
 			if err != nil {
 				return nil, err
 			}
 		} else if !isDir {
-            return nil, fmt.Errorf("%s/%s is not a directory", storagePath + "/" + dir)
+			return nil, fmt.Errorf("%s/%s is not a directory", storagePath+"/"+dir)
 		}
 		storage.idCache[dir] = dirID
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
-
 }
 
 func (storage *ACDStorage) getPathID(path string) string {
@@ -88,56 +91,103 @@ func (storage *ACDStorage) deletePathID(path string) {
 	storage.idCacheLock.Unlock()
 }
 
-
-func (storage *ACDStorage) convertFilePath(filePath string) (string) {
+// convertFilePath converts the path for a fossil in the form of 'chunks/id.fsl' to 'fossils/id'.  This is because
+// ACD doesn't support file renaming. Instead, it only allows one file to be moved from one directory to another.
+// By adding a layer of path conversion we're pretending that we can rename between 'chunks/id' and 'chunks/id.fsl'
+func (storage *ACDStorage) convertFilePath(filePath string) string {
 	if strings.HasPrefix(filePath, "chunks/") && strings.HasSuffix(filePath, ".fsl") {
-        return "fossils/" + filePath[len("chunks/"):len(filePath) - len(".fsl")]
+		return "fossils/" + filePath[len("chunks/"):len(filePath)-len(".fsl")]
 	}
 	return filePath
 }
 
-func (storage *ACDStorage) getIDFromPath(threadIndex int, path string) (fileID string, isDir bool, size int64, err error) {
+// getIDFromPath returns the id of the given path.  If 'createDirectories' is true, create the given path and all its
+// parent directories if they don't exist.  Note that if 'createDirectories' is false, it may return an empty 'fileID'
+// if the file doesn't exist.
+func (storage *ACDStorage) getIDFromPath(threadIndex int, filePath string, createDirectories bool) (fileID string, err error) {
+
+	if fileID, ok := storage.findPathID(filePath); ok {
+		return fileID, nil
+	}
 
 	parentID, ok := storage.findPathID("")
 	if !ok {
-        parentID, isDir, size, err = storage.client.ListByName("", "")
+		parentID, _, _, err = storage.client.ListByName("", "")
 		if err != nil {
-            return "", false, 0, err
+			return "", err
 		}
+		storage.savePathID("", parentID)
 	}
 
-    names := strings.Split(path, "/")
+	names := strings.Split(filePath, "/")
+	current := ""
 	for i, name := range names {
-        parentID, isDir, _, err = storage.client.ListByName(parentID, name)
+
+		current = path.Join(current, name)
+		fileID, ok := storage.findPathID(current)
+		if ok {
+			parentID = fileID
+			continue
+		}
+		isDir := false
+		fileID, isDir, _, err = storage.client.ListByName(parentID, name)
 		if err != nil {
-            return "", false, 0, err
+			return "", err
 		}
-        if parentID == "" {
-            if i == len(names) - 1 {
-                return "", false, 0, nil
+		if fileID == "" {
+			if !createDirectories {
+				return "", nil
+			}
+			// Create the current directory
+			fileID, err = storage.client.CreateDirectory(parentID, name)
+			if err != nil {
+				// Check if the directory has been created by another thread
+				if e, ok := err.(ACDError); !ok || e.Status != 409 {
+					return "", fmt.Errorf("Failed to create directory '%s': %v", current, err)
+				}
+				// A 409 means the directory may have already created by another thread.  Wait 10 seconds
+				// until we seed the directory.
+				for i := 0; i < 10; i++ {
+					var createErr error
+					fileID, isDir, _, createErr = storage.client.ListByName(parentID, name)
+					if createErr != nil {
+						return "", createErr
+					}
+					if fileID == "" {
+						time.Sleep(time.Second)
 					} else {
-                return "", false, 0, fmt.Errorf("File path '%s' does not exist", path)
+						break
 					}
 				}
-        if i != len(names) - 1 && !isDir {
-            return "", false, 0, fmt.Errorf("Invalid path %s", path)
+				if fileID == "" {
+					return "", fmt.Errorf("All attempts to create directory '%s' failed: %v", current, err)
 				}
+			} else {
+				isDir = true
+			}
+		} else {
+			storage.savePathID(current, fileID)
+		}
+		if i != len(names)-1 && !isDir {
+			return "", fmt.Errorf("Path '%s' is not a directory", current)
+		}
+		parentID = fileID
 	}
 
-    return parentID, isDir, size, err
+	return parentID, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
 	var err error
 
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	if dir == "snapshots" {
 
-        entries, err := storage.client.ListEntries(storage.getPathID(dir), false)
+		entries, err := storage.client.ListEntries(storage.getPathID(dir), false, true)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -146,7 +196,7 @@ func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 
 		for _, entry := range entries {
 			storage.savePathID(entry.Name, entry.ID)
-            subDirs = append(subDirs, entry.Name + "/")
+			subDirs = append(subDirs, entry.Name+"/")
 		}
 		return subDirs, nil, nil
 	} else if strings.HasPrefix(dir, "snapshots/") {
@@ -160,9 +210,10 @@ func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 			if pathID == "" {
 				return nil, nil, nil
 			}
+			storage.savePathID(dir, pathID)
 		}
 
-        entries, err := storage.client.ListEntries(pathID, true)
+		entries, err := storage.client.ListEntries(pathID, true, false)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -170,29 +221,40 @@ func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 		files := []string{}
 
 		for _, entry := range entries {
-            storage.savePathID(dir + "/" + entry.Name, entry.ID)
+			storage.savePathID(dir+"/"+entry.Name, entry.ID)
 			files = append(files, entry.Name)
 		}
 		return files, nil, nil
 	} else {
 		files := []string{}
 		sizes := []int64{}
-        for _, parent := range []string {"chunks", "fossils" } {
-            entries, err := storage.client.ListEntries(storage.getPathID(parent), true)
+		parents := []string{"chunks", "fossils"}
+		for i := 0; i < len(parents); i++ {
+			parent := parents[i]
+			pathID, ok := storage.findPathID(parent)
+			if !ok {
+				continue
+			}
+			entries, err := storage.client.ListEntries(pathID, true, true)
 			if err != nil {
 				return nil, nil, err
 			}
-
-
 			for _, entry := range entries {
+				if entry.Kind != "FOLDER" {
 					name := entry.Name
-                if parent == "fossils" {
-                    name += ".fsl"
+					if strings.HasPrefix(parent, "fossils") {
+						name = parent + "/" + name + ".fsl"
+						name = name[len("fossils/"):]
+					} else {
+						name = parent + "/" + name
+						name = name[len("chunks/"):]
 					}
-
-                storage.savePathID(parent + "/" + entry.Name, entry.ID)
 					files = append(files, name)
 					sizes = append(sizes, entry.Size)
+				} else {
+					parents = append(parents, parent+"/"+entry.Name)
+				}
+				storage.savePathID(parent+"/"+entry.Name, entry.ID)
 			}
 		}
 		return files, sizes, nil
@@ -203,18 +265,14 @@ func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *ACDStorage) DeleteFile(threadIndex int, filePath string) (err error) {
 	filePath = storage.convertFilePath(filePath)
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
-        fileID, _, _, err = storage.getIDFromPath(threadIndex, filePath)
+	fileID, err := storage.getIDFromPath(threadIndex, filePath, false)
 	if err != nil {
 		return err
 	}
 	if fileID == "" {
-            LOG_TRACE("ACD_STORAGE", "File %s has disappeared before deletion", filePath)
+		LOG_TRACE("ACD_STORAGE", "File '%s' to be deleted does not exist", filePath)
 		return nil
 	}
-        storage.savePathID(filePath, fileID)
-    }
 
 	err = storage.client.DeleteFile(fileID)
 	if e, ok := err.(ACDError); ok && e.Status == 409 {
@@ -234,11 +292,19 @@ func (storage *ACDStorage) MoveFile(threadIndex int, from string, to string) (er
 		return fmt.Errorf("Attempting to rename file %s with unknown id", from)
 	}
 
-    fromParentID := storage.getPathID("chunks")
-    toParentID := storage.getPathID("fossils")
+	fromParent := path.Dir(from)
+	fromParentID, err := storage.getIDFromPath(threadIndex, fromParent, false)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of the parent directory '%s': %v", fromParent, err)
+	}
+	if fromParentID == "" {
+		return fmt.Errorf("The parent directory '%s' does not exist", fromParent)
+	}
 
-    if strings.HasPrefix(from, "fossils") {
-        fromParentID, toParentID = toParentID, fromParentID
+	toParent := path.Dir(to)
+	toParentID, err := storage.getIDFromPath(threadIndex, toParent, true)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of the parent directory '%s': %v", toParent, err)
 	}
 
 	err = storage.client.MoveFile(fileID, fromParentID, toParentID)
@@ -259,17 +325,21 @@ func (storage *ACDStorage) MoveFile(threadIndex int, from string, to string) (er
 // CreateDirectory creates a new directory.
 func (storage *ACDStorage) CreateDirectory(threadIndex int, dir string) (err error) {
 
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
-    if dir == "chunks" || dir == "snapshots" {
-        return nil
+	parentPath := path.Dir(dir)
+	if parentPath == "." {
+		parentPath = ""
+	}
+	parentID, ok := storage.findPathID(parentPath)
+	if !ok {
+		return fmt.Errorf("Path directory '%s' has unknown id", parentPath)
 	}
 
-    if strings.HasPrefix(dir, "snapshots/") {
-        name := dir[len("snapshots/"):]
-        dirID, err := storage.client.CreateDirectory(storage.getPathID("snapshots"), name)
+	name := path.Base(dir)
+	dirID, err := storage.client.CreateDirectory(parentID, name)
 	if err != nil {
 		if e, ok := err.(ACDError); ok && e.Status == 409 {
 			return nil
@@ -278,9 +348,6 @@ func (storage *ACDStorage) CreateDirectory(threadIndex int, dir string) (err err
 		}
 	}
 	storage.savePathID(dir, dirID)
-        return nil
-
-    }
 
 	return nil
 }
@@ -288,13 +355,26 @@ func (storage *ACDStorage) CreateDirectory(threadIndex int, dir string) (err err
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *ACDStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
 	}
 
 	filePath = storage.convertFilePath(filePath)
-    fileID := ""
-    fileID, isDir, size, err = storage.getIDFromPath(threadIndex, filePath)
+
+	parentPath := path.Dir(filePath)
+	if parentPath == "." {
+		parentPath = ""
+	}
+	parentID, err := storage.getIDFromPath(threadIndex, parentPath, false)
+	if err != nil {
+		return false, false, 0, err
+	}
+	if parentID == "" {
+		return false, false, 0, nil
+	}
+
+	name := path.Base(filePath)
+	fileID, isDir, size, err := storage.client.ListByName(parentID, name)
 	if err != nil {
 		return false, false, 0, err
 	}
@@ -302,44 +382,19 @@ func (storage *ACDStorage) GetFileInfo(threadIndex int, filePath string) (exist
 		return false, false, 0, nil
 	}
 
+	storage.savePathID(filePath, fileID)
 	return true, isDir, size, nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *ACDStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    parentID := ""
-    filePath = "chunks/" + chunkID
-    realPath := filePath
-    if isFossil {
-        parentID = storage.getPathID("fossils")
-        filePath += ".fsl"
-        realPath = "fossils/" + chunkID + ".fsl"
-    } else {
-        parentID = storage.getPathID("chunks")
-    }
-
-    fileID := ""
-    fileID, _, size, err = storage.client.ListByName(parentID, chunkID)
-    if fileID != "" {
-        storage.savePathID(realPath, fileID)
-    }
-    return filePath, fileID != "", size, err
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *ACDStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
-        fileID, _, _, err = storage.getIDFromPath(threadIndex, filePath)
+	fileID, err := storage.getIDFromPath(threadIndex, filePath, false)
 	if err != nil {
 		return err
 	}
 	if fileID == "" {
 		return fmt.Errorf("File path '%s' does not exist", filePath)
 	}
-        storage.savePathID(filePath, fileID)
-    }
 
 	readCloser, _, err := storage.client.DownloadFile(fileID)
 	if err != nil {
@@ -348,32 +403,26 @@ func (storage *ACDStorage) DownloadFile(threadIndex int, filePath string, chunk
 
 	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThreads)
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThreads)
 	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *ACDStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
 	parent := path.Dir(filePath)
-
 	if parent == "." {
 		parent = ""
 	}
 
-    parentID, ok := storage.findPathID(parent)
-
-    if !ok {
-        parentID, _, _, err = storage.getIDFromPath(threadIndex, parent)
+	parentID, err := storage.getIDFromPath(threadIndex, parent, true)
 	if err != nil {
 		return err
 	}
 	if parentID == "" {
 		return fmt.Errorf("File path '%s' does not exist", parent)
 	}
-        storage.savePathID(parent, parentID)
-    }
 
-    fileID, err := storage.client.UploadFile(parentID, path.Base(filePath), content, storage.UploadRateLimit / storage.numberOfThreads)
+	fileID, err := storage.client.UploadFile(parentID, path.Base(filePath), content, storage.UploadRateLimit/storage.numberOfThreads)
 	if err == nil {
 		storage.savePathID(filePath, fileID)
 		return nil
@@ -389,16 +438,16 @@ func (storage *ACDStorage) UploadFile(threadIndex int, filePath string, content
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *ACDStorage) IsCacheNeeded() (bool) { return true }
+func (storage *ACDStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *ACDStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *ACDStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *ACDStorage) IsStrongConsistent() (bool) { return true }
+func (storage *ACDStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *ACDStorage) IsFastListing() (bool) { return true }
+func (storage *ACDStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *ACDStorage) EnableTestMode() {}

src/duplicacy_azurestorage.go

@@ -12,7 +12,7 @@ import (
 )
 
 type AzureStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	containers []*storage.Container
 }
@@ -43,10 +43,12 @@ func CreateAzureStorage(accountName string, accountKey string,
 		return nil, fmt.Errorf("container %s does not exist", containerName)
 	}
 
-    azureStorage = &AzureStorage {
+	azureStorage = &AzureStorage{
 		containers: containers,
 	}
 
+	azureStorage.DerivedStorage = azureStorage
+	azureStorage.SetDefaultNestingLevels([]int{0}, 0)
 	return
 }
 
@@ -62,12 +64,12 @@ func (azureStorage *AzureStorage) ListFiles(threadIndex int, dir string) (files
 		Timeout    uint
 	}
 
-    if len(dir) > 0 && dir[len(dir) - 1] != '/' {
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
 		dir += "/"
 	}
 	dirLength := len(dir)
 
-    parameters := storage.ListBlobsParameters {
+	parameters := storage.ListBlobsParameters{
 		Prefix:    dir,
 		Delimiter: "",
 	}
@@ -84,7 +86,7 @@ func (azureStorage *AzureStorage) ListFiles(threadIndex int, dir string) (files
 		if dir == "snapshots/" {
 			for _, blob := range results.Blobs {
 				name := strings.Split(blob.Name[dirLength:], "/")[0]
-                 subDirs[name + "/"] = true
+				subDirs[name+"/"] = true
 			}
 		} else {
 			for _, blob := range results.Blobs {
@@ -149,23 +151,6 @@ func (storage *AzureStorage) GetFileInfo(threadIndex int, filePath string) (exis
 	return true, false, blob.Properties.ContentLength, nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *AzureStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-    if err != nil {
-        return "", false, 0, err
-    } else {
-        return filePath, exist, size, err
-    }
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *AzureStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	readCloser, err := storage.containers[threadIndex].GetBlobReference(filePath).Get(nil)
@@ -175,13 +160,13 @@ func (storage *AzureStorage) DownloadFile(threadIndex int, filePath string, chun
 
 	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.containers))
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/len(storage.containers))
 	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *AzureStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-   reader := CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.containers))
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.containers))
 	blob := storage.containers[threadIndex].GetBlobReference(filePath)
 	return blob.CreateBlockBlobFromReader(reader, nil)
 
@@ -189,16 +174,16 @@ func (storage *AzureStorage) UploadFile(threadIndex int, filePath string, conten
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *AzureStorage) IsCacheNeeded() (bool) { return true }
+func (storage *AzureStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *AzureStorage)  IsMoveFileImplemented() (bool) { return true }
+func (storage *AzureStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *AzureStorage) IsStrongConsistent() (bool) { return true }
+func (storage *AzureStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *AzureStorage) IsFastListing() (bool) { return true }
+func (storage *AzureStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *AzureStorage) EnableTestMode() {}

src/duplicacy_b2client.go

@@ -5,19 +5,19 @@
 package duplicacy
 
 import (
-    "fmt"
-    "time"
 	"bytes"
-    "strconv"
-    "io/ioutil"
-    "encoding/json"
+	"crypto/sha1"
 	"encoding/base64"
 	"encoding/hex"
+	"encoding/json"
+	"fmt"
 	"io"
-    "net/http"
-    "strings"
-    "crypto/sha1"
+	"io/ioutil"
 	"math/rand"
+	"net/http"
+	"strconv"
+	"strings"
+	"time"
 )
 
 type B2Error struct {
@@ -51,7 +51,6 @@ type B2Client struct {
 	UploadToken string
 
 	TestMode bool
-
 }
 
 func NewB2Client(accountID string, applicationKey string) *B2Client {
@@ -82,40 +81,55 @@ func (client *B2Client) retry(backoff int, response *http.Response) int {
 	return backoff
 }
 
-func (client *B2Client) call(url string, input interface{}) (io.ReadCloser, int64, error) {
+func (client *B2Client) call(url string, method string, requestHeaders map[string]string, input interface{}) (io.ReadCloser, http.Header, int64, error) {
+
+	switch method {
+	case http.MethodGet:
+		break
+	case http.MethodHead:
+		break
+	case http.MethodPost:
+		break
+	default:
+		return nil, nil, 0, fmt.Errorf("unhandled http request method: " + method)
+	}
 
 	var response *http.Response
 
 	backoff := 0
 	for i := 0; i < 8; i++ {
 		var inputReader *bytes.Reader
-        method := "POST"
 
 		switch input.(type) {
 		default:
 			jsonInput, err := json.Marshal(input)
 			if err != nil {
-                return nil, 0, err
+				return nil, nil, 0, err
 			}
 			inputReader = bytes.NewReader(jsonInput)
 		case []byte:
 			inputReader = bytes.NewReader(input.([]byte))
 		case int:
-            method = "GET"
 			inputReader = bytes.NewReader([]byte(""))
 		}
 
 		request, err := http.NewRequest(method, url, inputReader)
 		if err != nil {
-            return nil, 0, err
+			return nil, nil, 0, err
 		}
 
 		if url == B2AuthorizationURL {
-            request.Header.Set("Authorization", "Basic " + base64.StdEncoding.EncodeToString([]byte(client.AccountID + ":" + client.ApplicationKey)))
+			request.Header.Set("Authorization", "Basic "+base64.StdEncoding.EncodeToString([]byte(client.AccountID+":"+client.ApplicationKey)))
 		} else {
 			request.Header.Set("Authorization", client.AuthorizationToken)
 		}
 
+		if requestHeaders != nil {
+			for key, value := range requestHeaders {
+				request.Header.Set(key, value)
+			}
+		}
+
 		if client.TestMode {
 			r := rand.Float32()
 			if r < 0.5 {
@@ -132,28 +146,37 @@ func (client *B2Client) call(url string, input interface{}) (io.ReadCloser, int6
 				backoff = client.retry(backoff, response)
 				continue
 			}
-            return nil, 0, err
+			return nil, nil, 0, err
 		}
 
 		if response.StatusCode < 300 {
-            return response.Body, response.ContentLength, nil
+			return response.Body, response.Header, response.ContentLength, nil
 		}
 
-        LOG_DEBUG("BACKBLAZE_CALL", "URL request '%s' returned status code %d", url, response.StatusCode)
+		LOG_DEBUG("BACKBLAZE_CALL", "URL request '%s %s' returned status code %d", method, url, response.StatusCode)
 
 		io.Copy(ioutil.Discard, response.Body)
 		response.Body.Close()
 		if response.StatusCode == 401 {
 			if url == B2AuthorizationURL {
-                return nil, 0, fmt.Errorf("Authorization failure")
+				return nil, nil, 0, fmt.Errorf("Authorization failure")
 			}
 			client.AuthorizeAccount()
 			continue
 		} else if response.StatusCode == 403 {
 			if !client.TestMode {
-                return nil, 0, fmt.Errorf("B2 cap exceeded")
+				return nil, nil, 0, fmt.Errorf("B2 cap exceeded")
 			}
 			continue
+		} else if response.StatusCode == 404 {
+			if http.MethodHead == method {
+				return nil, nil, 0, nil
+			}
+		} else if response.StatusCode == 416 {
+			if http.MethodHead == method {
+				// 416 Requested Range Not Satisfiable
+				return nil, nil, 0, fmt.Errorf("URL request '%s' returned status code %d", url, response.StatusCode)
+			}
 		} else if response.StatusCode == 429 || response.StatusCode == 408 {
 			backoff = client.retry(backoff, response)
 			continue
@@ -168,17 +191,16 @@ func (client *B2Client) call(url string, input interface{}) (io.ReadCloser, int6
 
 		defer response.Body.Close()
 
-        e := &B2Error {
-        }
+		e := &B2Error{}
 
 		if err := json.NewDecoder(response.Body).Decode(e); err != nil {
-            return nil, 0, err
+			return nil, nil, 0, err
 		}
 
-        return nil, 0, e
+		return nil, nil, 0, e
 	}
 
-    return nil, 0, fmt.Errorf("Maximum backoff reached")
+	return nil, nil, 0, fmt.Errorf("Maximum backoff reached")
 }
 
 type B2AuthorizeAccountOutput struct {
@@ -190,14 +212,14 @@ type B2AuthorizeAccountOutput struct {
 
 func (client *B2Client) AuthorizeAccount() (err error) {
 
-    readCloser, _, err := client.call(B2AuthorizationURL, make(map[string]string))
+	readCloser, _, _, err := client.call(B2AuthorizationURL, http.MethodPost, nil, make(map[string]string))
 	if err != nil {
 		return err
 	}
 
 	defer readCloser.Close()
 
-    output := &B2AuthorizeAccountOutput {}
+	output := &B2AuthorizeAccountOutput{}
 
 	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 		return err
@@ -224,7 +246,7 @@ func (client *B2Client) FindBucket(bucketName string) (err error) {
 
 	url := client.APIURL + "/b2api/v1/b2_list_buckets"
 
-    readCloser, _, err := client.call(url, input)
+	readCloser, _, _, err := client.call(url, http.MethodPost, nil, input)
 	if err != nil {
 		return err
 	}
@@ -289,17 +311,76 @@ func (client *B2Client) ListFileNames(startFileName string, singleFile bool, inc
 
 	for {
 		url := client.APIURL + "/b2api/v1/b2_list_file_names"
+		requestHeaders := map[string]string{}
+		requestMethod := http.MethodPost
+		var requestInput interface{}
+		requestInput = input
 		if includeVersions {
 			url = client.APIURL + "/b2api/v1/b2_list_file_versions"
+		} else if singleFile {
+			// handle a single file with no versions as a special case to download the last byte of the file
+			url = client.DownloadURL + "/file/" + client.BucketName + "/" + startFileName
+			// requesting byte -1 works for empty files where 0-0 fails with a 416 error
+			requestHeaders["Range"] = "bytes=-1"
+			// HEAD request
+			requestMethod = http.MethodHead
+			requestInput = 0
 		}
-        readCloser, _, err := client.call(url, input)
+		var readCloser io.ReadCloser
+		var responseHeader http.Header
+		var err error
+		readCloser, responseHeader, _, err = client.call(url, requestMethod, requestHeaders, requestInput)
 		if err != nil {
 			return nil, err
 		}
 
+		if readCloser != nil {
 			defer readCloser.Close()
+		}
 
-        output := B2ListFileNamesOutput {
+		output := B2ListFileNamesOutput{}
+
+		if singleFile && !includeVersions {
+			if responseHeader == nil {
+				LOG_DEBUG("BACKBLAZE_LIST", "b2_download_file_by_name did not return headers")
+				return []*B2Entry{}, nil
+			}
+			requiredHeaders := []string{
+				"x-bz-file-id",
+				"x-bz-file-name",
+			}
+			missingKeys := []string{}
+			for _, headerKey := range requiredHeaders {
+				if "" == responseHeader.Get(headerKey) {
+					missingKeys = append(missingKeys, headerKey)
+				}
+			}
+			if len(missingKeys) > 0 {
+				return nil, fmt.Errorf("b2_download_file_by_name missing headers: %s", missingKeys)
+			}
+			// construct the B2Entry from the response headers of the download request
+			fileID := responseHeader.Get("x-bz-file-id")
+			fileName := responseHeader.Get("x-bz-file-name")
+			fileAction := "upload"
+			// byte range that is returned: "bytes #-#/#
+			rangeString := responseHeader.Get("Content-Range")
+			// total file size; 1 if file has content, 0 if it's empty
+			lengthString := responseHeader.Get("Content-Length")
+			var fileSize int64
+			if "" != rangeString {
+				fileSize, _ = strconv.ParseInt(rangeString[strings.Index(rangeString, "/")+1:], 0, 64)
+			} else if "" != lengthString {
+				// this should only execute if the requested file is empty and the range request didn't result in a Content-Range header
+				fileSize, _ = strconv.ParseInt(lengthString, 0, 64)
+				if fileSize != 0 {
+					return nil, fmt.Errorf("b2_download_file_by_name returned non-zero file length")
+				}
+			} else {
+				return nil, fmt.Errorf("could not parse b2_download_file_by_name headers")
+			}
+			fileUploadTimestamp, _ := strconv.ParseInt(responseHeader.Get("X-Bz-Upload-Timestamp"), 0, 64)
+
+			return []*B2Entry{&B2Entry{fileID, fileName, fileAction, fileSize, fileUploadTimestamp}}, nil
 		}
 
 		if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
@@ -355,7 +436,7 @@ func (client *B2Client) DeleteFile(fileName string, fileID string) (err error) {
 	input["fileId"] = fileID
 
 	url := client.APIURL + "/b2api/v1/b2_delete_file_version"
-    readCloser, _, err := client.call(url, input)
+	readCloser, _, _, err := client.call(url, http.MethodPost, make(map[string]string), input)
 	if err != nil {
 		return err
 	}
@@ -375,14 +456,14 @@ func (client *B2Client) HideFile(fileName string) (fileID string, err error) {
 	input["fileName"] = fileName
 
 	url := client.APIURL + "/b2api/v1/b2_hide_file"
-    readCloser, _, err := client.call(url, input)
+	readCloser, _, _, err := client.call(url, http.MethodPost, make(map[string]string), input)
 	if err != nil {
 		return "", err
 	}
 
 	defer readCloser.Close()
 
-    output := & B2HideFileOutput {}
+	output := &B2HideFileOutput{}
 
 	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 		return "", err
@@ -396,7 +477,8 @@ func (client *B2Client) DownloadFile(filePath string) (io.ReadCloser, int64, err
 
 	url := client.DownloadURL + "/file/" + client.BucketName + "/" + filePath
 
-    return client.call(url, 0)
+	readCloser, _, len, err := client.call(url, http.MethodGet, make(map[string]string), 0)
+	return readCloser, len, err
 }
 
 type B2GetUploadArgumentOutput struct {
@@ -405,19 +487,19 @@ type B2GetUploadArgumentOutput struct {
 	AuthorizationToken string
 }
 
-func (client *B2Client) getUploadURL() (error) {
+func (client *B2Client) getUploadURL() error {
 	input := make(map[string]string)
 	input["bucketId"] = client.BucketID
 
 	url := client.APIURL + "/b2api/v1/b2_get_upload_url"
-    readCloser, _, err := client.call(url, input)
+	readCloser, _, _, err := client.call(url, http.MethodPost, make(map[string]string), input)
 	if err != nil {
 		return err
 	}
 
 	defer readCloser.Close()
 
-    output := & B2GetUploadArgumentOutput {}
+	output := &B2GetUploadArgumentOutput{}
 
 	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 		return err
@@ -431,7 +513,6 @@ func (client *B2Client) getUploadURL() (error) {
 
 func (client *B2Client) UploadFile(filePath string, content []byte, rateLimit int) (err error) {
 
-
 	hasher := sha1.New()
 	hasher.Write(content)
 	hash := hex.EncodeToString(hasher.Sum(nil))
@@ -498,7 +579,7 @@ func (client *B2Client) UploadFile(filePath string, content []byte, rateLimit in
 		LOG_DEBUG("BACKBLAZE_UPLOAD", "URL request '%s' returned status code %d", client.UploadURL, response.StatusCode)
 
 		if response.StatusCode == 401 {
-            LOG_INFO("BACKBLAZE_UPLOAD", "Re-authorizatoin required")
+			LOG_INFO("BACKBLAZE_UPLOAD", "Re-authorization required")
 			client.UploadURL = ""
 			client.UploadToken = ""
 			continue
@@ -517,4 +598,3 @@ func (client *B2Client) UploadFile(filePath string, content []byte, rateLimit in
 
 	return fmt.Errorf("Maximum backoff reached")
 }
-

src/duplicacy_b2client_test.go

@@ -5,15 +5,15 @@
 package duplicacy
 
 import (
-    "testing"
 	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
+	"testing"
 
 	crypto_rand "crypto/rand"
-    "math/rand"
 	"io"
 	"io/ioutil"
+	"math/rand"
 )
 
 func createB2ClientForTest(t *testing.T) (*B2Client, string) {
@@ -79,7 +79,7 @@ func TestB2Client(t *testing.T) {
 
 	maxSize := 10000
 	for i := 0; i < 20; i++ {
-        size := rand.Int() % maxSize + 1
+		size := rand.Int()%maxSize + 1
 		content := make([]byte, size)
 		_, err := crypto_rand.Read(content)
 		if err != nil {
@@ -90,7 +90,7 @@ func TestB2Client(t *testing.T) {
 		hash := sha256.Sum256(content)
 		name := hex.EncodeToString(hash[:])
 
-        err = b2Client.UploadFile(testDirectory + name, content, 100)
+		err = b2Client.UploadFile(testDirectory+name, content, 100)
 		if err != nil {
 			t.Errorf("Error uploading file '%s': %v", name, err)
 			return
@@ -118,7 +118,7 @@ func TestB2Client(t *testing.T) {
 
 		hash := hex.EncodeToString(hasher.Sum(nil))
 
-        if testDirectory + hash != file.FileName {
+		if testDirectory+hash != file.FileName {
 			t.Errorf("File %s has hash %s", file.FileName, hash)
 		}
 

src/duplicacy_b2storage.go

@@ -9,7 +9,7 @@ import (
 )
 
 type B2Storage struct {
-    RateLimitedStorage
+	StorageBase
 
 	clients []*B2Client
 }
@@ -35,16 +35,19 @@ func CreateB2Storage(accountID string, applicationKey string, bucket string, thr
 		clients = append(clients, client)
 	}
 
-    storage = &B2Storage {
+	storage = &B2Storage{
 		clients: clients,
 	}
+
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *B2Storage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 	length := len(dir) + 1
 
@@ -65,7 +68,7 @@ func (storage *B2Storage) ListFiles(threadIndex int, dir string) (files []string
 		for _, entry := range entries {
 			name := entry.FileName[length:]
 			subDir := strings.Split(name, "/")[0]
-            subDirs[subDir + "/"] = true
+			subDirs[subDir+"/"] = true
 		}
 
 		for subDir, _ := range subDirs {
@@ -79,7 +82,7 @@ func (storage *B2Storage) ListFiles(threadIndex int, dir string) (files []string
 			}
 			lastFile = entry.FileName
 			if entry.Action == "hide" {
-                files = append(files, entry.FileName[length:] + ".fsl")
+				files = append(files, entry.FileName[length:]+".fsl")
 			} else {
 				files = append(files, entry.FileName[length:])
 			}
@@ -98,7 +101,7 @@ func (storage *B2Storage) ListFiles(threadIndex int, dir string) (files []string
 func (storage *B2Storage) DeleteFile(threadIndex int, filePath string) (err error) {
 
 	if strings.HasSuffix(filePath, ".fsl") {
-        filePath = filePath[:len(filePath) - len(".fsl")]
+		filePath = filePath[:len(filePath)-len(".fsl")]
 		entries, err := storage.clients[threadIndex].ListFileNames(filePath, true, true)
 		if err != nil {
 			return err
@@ -107,7 +110,7 @@ func (storage *B2Storage) DeleteFile(threadIndex int, filePath string) (err erro
 		toBeDeleted := false
 
 		for _, entry := range entries {
-            if entry.FileName != filePath || (!toBeDeleted && entry.Action != "hide" ) {
+			if entry.FileName != filePath || (!toBeDeleted && entry.Action != "hide") {
 				continue
 			}
 
@@ -141,12 +144,12 @@ func (storage *B2Storage) MoveFile(threadIndex int, from string, to string) (err
 
 	if strings.HasSuffix(from, ".fsl") {
 		filePath = to
-        if from != to + ".fsl" {
+		if from != to+".fsl" {
 			filePath = ""
 		}
 	} else if strings.HasSuffix(to, ".fsl") {
 		filePath = from
-        if to != from + ".fsl" {
+		if to != from+".fsl" {
 			filePath = ""
 		}
 	}
@@ -182,7 +185,7 @@ func (storage *B2Storage) GetFileInfo(threadIndex int, filePath string) (exist b
 	isFossil := false
 	if strings.HasSuffix(filePath, ".fsl") {
 		isFossil = true
-        filePath = filePath[:len(filePath) - len(".fsl")]
+		filePath = filePath[:len(filePath)-len(".fsl")]
 	}
 
 	entries, err := storage.clients[threadIndex].ListFileNames(filePath, true, isFossil)
@@ -204,20 +207,10 @@ func (storage *B2Storage) GetFileInfo(threadIndex int, filePath string) (exist b
 	return true, false, entries[0].Size, nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *B2Storage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-    return filePath, exist, size, err
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *B2Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
+	filePath = strings.Replace(filePath, " ", "%20", -1)
 	readCloser, _, err := storage.clients[threadIndex].DownloadFile(filePath)
 	if err != nil {
 		return err
@@ -225,27 +218,28 @@ func (storage *B2Storage) DownloadFile(threadIndex int, filePath string, chunk *
 
 	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.clients))
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/len(storage.clients))
 	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *B2Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    return storage.clients[threadIndex].UploadFile(filePath, content, storage.UploadRateLimit / len(storage.clients))
+	filePath = strings.Replace(filePath, " ", "%20", -1)
+	return storage.clients[threadIndex].UploadFile(filePath, content, storage.UploadRateLimit/len(storage.clients))
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *B2Storage) IsCacheNeeded() (bool) { return true }
+func (storage *B2Storage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *B2Storage) IsMoveFileImplemented() (bool) { return true }
+func (storage *B2Storage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *B2Storage) IsStrongConsistent() (bool) { return true }
+func (storage *B2Storage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *B2Storage) IsFastListing() (bool) { return true }
+func (storage *B2Storage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *B2Storage) EnableTestMode() {

src/duplicacy_backupmanager.go

@@ -5,21 +5,21 @@
 package duplicacy
 
 import (
-    "encoding/json"
 	"bytes"
-    "os"
-    "io"
+	"encoding/hex"
+	"encoding/json"
 	"fmt"
+	"io"
+	"os"
 	"path"
-    "time"
+	"path/filepath"
+	"runtime"
 	"sort"
+	"strconv"
+	"strings"
 	"sync"
 	"sync/atomic"
-    "strings"
-    "strconv"
-    "runtime"
-    "encoding/hex"
-    "path/filepath"
+	"time"
 )
 
 // BackupManager performs the two major operations, backup and restore, and passes other operations, mostly related to
@@ -35,7 +35,9 @@ type BackupManager struct {
 	config *Config // contains a number of options
 }
 
-
+func (manager *BackupManager) SetDryRun(dryRun bool) {
+	manager.config.dryRun = dryRun
+}
 
 // CreateBackupManager creates a backup manager using the specified 'storage'.  'snapshotID' is a unique id to
 // identify snapshots created for this repository.  'top' is the top directory of the repository.  'password' is the
@@ -54,7 +56,7 @@ func CreateBackupManager(snapshotID string, storage Storage, top string, passwor
 
 	snapshotManager := CreateSnapshotManager(config, storage)
 
-    backupManager := &BackupManager {
+	backupManager := &BackupManager{
 		snapshotID: snapshotID,
 		storage:    storage,
 
@@ -77,13 +79,13 @@ func (manager *BackupManager) SetupSnapshotCache(storageName string) bool {
 	preferencePath := GetDuplicacyPreferencePath()
 	cacheDir := path.Join(preferencePath, "cache", storageName)
 
-    storage, err := CreateFileStorage(cacheDir, 2, false, 1)
+	storage, err := CreateFileStorage(cacheDir, false, 1)
 	if err != nil {
 		LOG_ERROR("BACKUP_CACHE", "Failed to create the snapshot cache dir: %v", err)
 		return false
 	}
 
-    for _, subdir := range [] string { "chunks", "snapshots" } {
+	for _, subdir := range []string{"chunks", "snapshots"} {
 		err := os.Mkdir(path.Join(cacheDir, subdir), 0744)
 		if err != nil && !os.IsExist(err) {
 			LOG_ERROR("BACKUP_CACHE", "Failed to create the snapshot cache subdir: %v", err)
@@ -91,6 +93,7 @@ func (manager *BackupManager) SetupSnapshotCache(storageName string) bool {
 		}
 	}
 
+	storage.SetDefaultNestingLevels([]int{1}, 1)
 	manager.snapshotCache = storage
 	manager.SnapshotManager.snapshotCache = storage
 	return true
@@ -102,7 +105,7 @@ func (manager *BackupManager) SetupSnapshotCache(storageName string) bool {
 //
 // This function assumes the Size field of each entry is equal to the length of the chunk content that belong
 // to the file.
-func setEntryContent(entries[] *Entry, chunkLengths[]int, offset int) {
+func setEntryContent(entries []*Entry, chunkLengths []int, offset int) {
 	if len(entries) == 0 {
 		return
 	}
@@ -116,7 +119,7 @@ func setEntryContent(entries[] *Entry, chunkLengths[]int, offset int) {
 	entries[i].StartOffset = 0
 	for j, length := range chunkLengths {
 
-        for totalChunkSize + int64(length) >= totalFileSize {
+		for totalChunkSize+int64(length) >= totalFileSize {
 			entries[i].EndChunk = j + offset
 			entries[i].EndOffset = int(totalFileSize - totalChunkSize)
 
@@ -127,7 +130,7 @@ func setEntryContent(entries[] *Entry, chunkLengths[]int, offset int) {
 
 			// If the current file ends at the end of the current chunk, the next file will
 			// start at the next chunk
-            if totalChunkSize + int64(length) == totalFileSize {
+			if totalChunkSize+int64(length) == totalFileSize {
 				entries[i].StartChunk = j + 1 + offset
 				entries[i].StartOffset = 0
 			} else {
@@ -213,7 +216,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 			allChunks, _ := manager.SnapshotManager.ListAllFiles(manager.storage, "chunks/")
 
 			for _, chunk := range allChunks {
-                if len(chunk) == 0 || chunk[len(chunk) - 1] == '/' {
+				if len(chunk) == 0 || chunk[len(chunk)-1] == '/' {
 					continue
 				}
 
@@ -226,7 +229,6 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 			}
 		}
 
-
 		if incompleteSnapshot != nil {
 
 			// This is the last chunk from the incomplete snapshot that can be found in the cache
@@ -240,6 +242,9 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 				}
 			}
 
+			LOG_DEBUG("CHUNK_INCOMPLETE", "The incomplete snapshot contains %d files and %d chunks", len(incompleteSnapshot.Files), len(incompleteSnapshot.ChunkHashes))
+			LOG_DEBUG("CHUNK_INCOMPLETE", "Last chunk in the incomplete snapshot that exist in the storage: %d", lastCompleteChunk)
+
 			// Only keep those files whose chunks exist in the cache
 			var files []*Entry
 			for _, file := range incompleteSnapshot.Files {
@@ -252,8 +257,8 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 			incompleteSnapshot.Files = files
 
 			// Remove incomplete chunks (they may not have been uploaded)
-            incompleteSnapshot.ChunkHashes = incompleteSnapshot.ChunkHashes[:lastCompleteChunk + 1]
-            incompleteSnapshot.ChunkLengths = incompleteSnapshot.ChunkLengths[:lastCompleteChunk + 1]            
+			incompleteSnapshot.ChunkHashes = incompleteSnapshot.ChunkHashes[:lastCompleteChunk+1]
+			incompleteSnapshot.ChunkLengths = incompleteSnapshot.ChunkLengths[:lastCompleteChunk+1]
 			remoteSnapshot = incompleteSnapshot
 			LOG_INFO("FILE_SKIP", "Skipped %d files from previous incomplete backup", len(files))
 		}
@@ -272,14 +277,14 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 	var totalModifiedFileSize int64    // total size of modified files
 	var uploadedModifiedFileSize int64 // portions that have been uploaded (including cache hits)
 
-    var modifiedEntries [] *Entry    // Files that has been modified or newly created
-    var preservedEntries [] *Entry   // Files unchanges
+	var modifiedEntries []*Entry  // Files that has been modified or newly created
+	var preservedEntries []*Entry // Files unchanges
 
 	// If the quick mode is disable and there isn't an incomplete snapshot from last (failed) backup,
 	// we simply treat all files as if they were new, and break them into chunks.
 	// Otherwise, we need to find those that are new or recently modified
 
-    if remoteSnapshot.Revision == 0 && incompleteSnapshot == nil {
+	if (remoteSnapshot.Revision == 0 || !quickMode) && incompleteSnapshot == nil {
 		modifiedEntries = localSnapshot.Files
 		for _, entry := range modifiedEntries {
 			totalModifiedFileSize += entry.Size
@@ -357,7 +362,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 		entry.EndChunk -= deletedChunks
 	}
 
-    var uploadedEntries [] *Entry
+	var uploadedEntries []*Entry
 	var uploadedChunkHashes []string
 	var uploadedChunkLengths []int
 	var uploadedChunkLock = &sync.Mutex{}
@@ -394,7 +399,6 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 		LOG_INFO("SNAPSHOT_FAIL", "Will abort the backup on chunk %d", chunkToFail)
 	}
 
-
 	chunkMaker := CreateChunkMaker(manager.config, false)
 	chunkUploader := CreateChunkUploader(manager.config, manager.storage, nil, threads, nil)
 
@@ -468,11 +472,11 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 				speed := uploadedModifiedFileSize / (now - startUploadingTime)
 				remainingTime := int64(0)
 				if speed > 0 {
-                    remainingTime = (totalModifiedFileSize - uploadedModifiedFileSize) / speed + 1
+					remainingTime = (totalModifiedFileSize-uploadedModifiedFileSize)/speed + 1
 				}
 				percentage := float32(uploadedModifiedFileSize * 1000 / totalModifiedFileSize)
 				LOG_INFO("UPLOAD_PROGRESS", "%s chunk %d size %d, %sB/s %s %.1f%%", action, chunkIndex,
-                            chunkSize, PrettySize(speed), PrettyTime(remainingTime), percentage / 10)
+					chunkSize, PrettySize(speed), PrettyTime(remainingTime), percentage/10)
 			}
 
 			atomic.AddInt64(&numberOfCollectedChunks, 1)
@@ -484,7 +488,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 		// Break files into chunks
 		chunkMaker.ForEachChunk(
 			fileReader.CurrentFile,
-            func (chunk *Chunk, final bool) {
+			func(chunk *Chunk, final bool) {
 
 				hash := chunk.GetHash()
 				chunkID := chunk.GetID()
@@ -494,7 +498,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 
 				_, found := chunkCache[chunkID]
 				if found {
-                    if time.Now().Unix() - lastUploadingTime > keepUploadAlive {
+					if time.Now().Unix()-lastUploadingTime > keepUploadAlive {
 						LOG_INFO("UPLOAD_KEEPALIVE", "Skip chunk cache to keep connection alive")
 						found = false
 					}
@@ -520,7 +524,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 				}
 
 			},
-            func (fileSize int64, hash string) (io.Reader, bool) {
+			func(fileSize int64, hash string) (io.Reader, bool) {
 
 				// Must lock here because the RunAtError function called by other threads may access uploadedEntries
 				uploadedChunkLock.Lock()
@@ -630,7 +634,9 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 	}
 	skippedFiles = append(skippedFiles, fileReader.SkippedFiles...)
 
+	if !manager.config.dryRun {
 		manager.SnapshotManager.CleanSnapshotCache(localSnapshot, nil)
+	}
 	LOG_INFO("BACKUP_END", "Backup for %s at revision %d completed", top, localSnapshot.Revision)
 
 	RunAtError = func() {}
@@ -641,8 +647,8 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 	if showStatistics {
 
 		LOG_INFO("BACKUP_STATS", "Files: %d total, %s bytes; %d new, %s bytes",
-                len(preservedEntries) + len(uploadedEntries),
-                PrettyNumber(preservedFileSize + uploadedFileSize),
+			len(preservedEntries)+len(uploadedEntries),
+			PrettyNumber(preservedFileSize+uploadedFileSize),
 			len(uploadedEntries), PrettyNumber(uploadedFileSize))
 
 		LOG_INFO("BACKUP_STATS", "File chunks: %d total, %s bytes; %d new, %s bytes, %s bytes uploaded",
@@ -656,17 +662,17 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 			PrettyNumber(totalUploadedSnapshotChunkBytes))
 
 		LOG_INFO("BACKUP_STATS", "All chunks: %d total, %s bytes; %d new, %s bytes, %s bytes uploaded",
-                len(localSnapshot.ChunkHashes) + totalSnapshotChunks,
-                PrettyNumber(totalFileChunkLength + totalSnapshotChunkLength),
-                int(numberOfNewFileChunks) + numberOfNewSnapshotChunks,
-                PrettyNumber(totalUploadedFileChunkLength + totalUploadedSnapshotChunkLength),
-                PrettyNumber(totalUploadedFileChunkBytes + totalUploadedSnapshotChunkBytes))
+			len(localSnapshot.ChunkHashes)+totalSnapshotChunks,
+			PrettyNumber(totalFileChunkLength+totalSnapshotChunkLength),
+			int(numberOfNewFileChunks)+numberOfNewSnapshotChunks,
+			PrettyNumber(totalUploadedFileChunkLength+totalUploadedSnapshotChunkLength),
+			PrettyNumber(totalUploadedFileChunkBytes+totalUploadedSnapshotChunkBytes))
 
 		now := time.Now().Unix()
 		if now == startTime {
 			now = startTime + 1
 		}
-        LOG_INFO("BACKUP_STATS", "Total running time: %s", PrettyTime(now - startTime))
+		LOG_INFO("BACKUP_STATS", "Total running time: %s", PrettyTime(now-startTime))
 	}
 
 	skipped := ""
@@ -690,7 +696,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 	}
 
 	if len(skipped) > 0 {
-        if len(skippedDirectories) + len(skippedFiles) == 1 {
+		if len(skippedDirectories)+len(skippedFiles) == 1 {
 			skipped += " was"
 		} else {
 			skipped += " were"
@@ -709,7 +715,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
 // the same as 'top'.  'quickMode' will bypass files with unchanged sizes and timestamps.  'deleteMode' will
 // remove local files that don't exist in the snapshot. 'patterns' is used to include/exclude certain files.
 func (manager *BackupManager) Restore(top string, revision int, inPlace bool, quickMode bool, threads int, overwrite bool,
-                                      deleteMode bool, showStatistics bool, patterns [] string) bool {
+	deleteMode bool, setOwner bool, showStatistics bool, patterns []string) bool {
 
 	startTime := time.Now().Unix()
 
@@ -730,7 +736,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 	_, err := os.Stat(top)
 	if os.IsNotExist(err) {
 		err = os.Mkdir(top, 0744)
-        if err != nil{
+		if err != nil {
 			LOG_ERROR("RESTORE_MKDIR", "Can't create the directory to be restored: %v", err)
 			return false
 		}
@@ -744,7 +750,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 	}
 
 	remoteSnapshot := manager.SnapshotManager.DownloadSnapshot(manager.snapshotID, revision)
-    manager.SnapshotManager.DownloadSnapshotContents(remoteSnapshot, patterns)
+	manager.SnapshotManager.DownloadSnapshotContents(remoteSnapshot, patterns, true)
 
 	localSnapshot, _, _, err := CreateSnapshotFromDirectory(manager.snapshotID, top)
 	if err != nil {
@@ -754,13 +760,14 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 
 	LOG_INFO("RESTORE_START", "Restoring %s to revision %d", top, revision)
 
-    var includedFiles [] *Entry
+	var includedFiles []*Entry
 
 	// Include/exclude some files if needed
 	if len(patterns) > 0 {
 		for _, file := range remoteSnapshot.Files {
 
 			if MatchPath(file.Path, patterns) {
+				LOG_TRACE("RESTORE_INCLUDE", "Include %s", file.Path)
 				includedFiles = append(includedFiles, file)
 			} else {
 				LOG_TRACE("RESTORE_EXCLUDE", "Exclude %s", file.Path)
@@ -774,7 +781,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 	var extraFiles []string
 
 	// These will store files to be downloaded.
-    fileEntries := make([]*Entry, 0, len(remoteSnapshot.Files) / 2)
+	fileEntries := make([]*Entry, 0, len(remoteSnapshot.Files)/2)
 
 	var totalFileSize int64
 	var downloadedFileSize int64
@@ -810,7 +817,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 		if entry.IsLink() {
 			stat, err := os.Lstat(fullPath)
 			if stat != nil {
-                if stat.Mode() & os.ModeSymlink != 0 {
+				if stat.Mode()&os.ModeSymlink != 0 {
 					isRegular, link, err := Readlink(fullPath)
 					if err == nil && link == entry.Link && !isRegular {
 						continue
@@ -875,7 +882,6 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 			if quickMode {
 				if file.IsSameAsFileInfo(stat) {
 					LOG_TRACE("RESTORE_SKIP", "File %s unchanged (by size and timestamp)", file.Path)
-                    file.RestoreMetadata(fullPath, &stat)
 					continue
 				}
 			}
@@ -893,14 +899,14 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 
 		// Handle zero size files.
 		if file.Size == 0 {
-            newFile, err := os.OpenFile(fullPath, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, file.GetPermissions())
+			newFile, err := os.OpenFile(fullPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, file.GetPermissions())
 			if err != nil {
 				LOG_ERROR("DOWNLOAD_OPEN", "Failed to create empty file: %v", err)
 				return false
 			}
 			newFile.Close()
 
-            file.RestoreMetadata(fullPath, nil)
+			file.RestoreMetadata(fullPath, nil, setOwner)
 			if !showStatistics {
 				LOG_INFO("DOWNLOAD_DONE", "Downloaded %s (0)", file.Path)
 			}
@@ -913,15 +919,13 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 			downloadedFileSize += file.Size
 			downloadedFiles = append(downloadedFiles, file)
 		}
-
-        file.RestoreMetadata(fullPath, nil)
+		file.RestoreMetadata(fullPath, nil, setOwner)
 	}
 
-
 	if deleteMode && len(patterns) == 0 {
 		// Reverse the order to make sure directories are empty before being deleted
 		for i := range extraFiles {
-            file := extraFiles[len(extraFiles) - 1 - i]
+			file := extraFiles[len(extraFiles)-1-i]
 			fullPath := joinPath(top, file)
 			os.Remove(fullPath)
 			LOG_INFO("RESTORE_DELETE", "Deleted %s", file)
@@ -931,7 +935,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
 	for _, entry := range remoteSnapshot.Files {
 		if entry.IsDir() && !entry.IsLink() {
 			dir := joinPath(top, entry.Path)
-            entry.RestoreMetadata(dir, nil)
+			entry.RestoreMetadata(dir, nil, setOwner)
 		}
 	}
 
@@ -981,7 +985,7 @@ func (encoder *fileEncoder) NextFile() (io.Reader, bool) {
 	if encoder.currentIndex == len(encoder.files) {
 		return nil, false
 	}
-    if encoder.currentIndex == len(encoder.files) - 1 {
+	if encoder.currentIndex == len(encoder.files)-1 {
 		encoder.buffer.Write([]byte("]"))
 		encoder.currentIndex++
 		return encoder, true
@@ -1012,7 +1016,7 @@ func (encoder *fileEncoder) NextFile() (io.Reader, bool) {
 // UploadSnapshot uploads the specified snapshot to the storage. It turns Files, ChunkHashes, and ChunkLengths into
 // sequences of chunks, and uploads these chunks, and finally the snapshot file.
 func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *ChunkUploader, top string, snapshot *Snapshot,
-                                             chunkCache map[string]bool, ) ( totalSnapshotChunkSize int64,
+	chunkCache map[string]bool) (totalSnapshotChunkSize int64,
 	numberOfNewSnapshotChunks int, totalUploadedSnapshotChunkSize int64,
 	totalUploadedSnapshotChunkBytes int64) {
 
@@ -1039,10 +1043,10 @@ func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *C
 
 	// uploadSequenceFunc uploads chunks read from 'reader'.
 	uploadSequenceFunc := func(reader io.Reader,
-                               nextReader func(size int64, hash string)(io.Reader, bool)) (sequence[]string) {
+		nextReader func(size int64, hash string) (io.Reader, bool)) (sequence []string) {
 
 		chunkMaker.ForEachChunk(reader,
-            func (chunk *Chunk, final bool) {
+			func(chunk *Chunk, final bool) {
 				totalSnapshotChunkSize += int64(chunk.GetLength())
 				chunkID := chunk.GetID()
 				if _, found := chunkCache[chunkID]; found {
@@ -1057,7 +1061,7 @@ func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *C
 		return sequence
 	}
 
-    sequences := []string { "chunks", "lengths" }
+	sequences := []string{"chunks", "lengths"}
 	// The file list is assumed not to be too large when fixed-size chunking is used
 	if chunkMaker.minimumChunkSize == chunkMaker.maximumChunkSize {
 		sequences = append(sequences, "files")
@@ -1074,7 +1078,7 @@ func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *C
 		}
 
 		sequence := uploadSequenceFunc(bytes.NewReader(contents),
-                                       func (fileSize int64, hash string) (io.Reader, bool) {
+			func(fileSize int64, hash string) (io.Reader, bool) {
 				return nil, false
 			})
 		snapshot.SetSequence(sequenceType, sequence)
@@ -1083,7 +1087,7 @@ func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *C
 	// File sequence may be too big to fit into the memory.  So we encode files one by one and take advantages of
 	// the multi-reader capability of the chunk maker.
 	if chunkMaker.minimumChunkSize != chunkMaker.maximumChunkSize {
-        encoder := fileEncoder {
+		encoder := fileEncoder{
 			top:            top,
 			readAttributes: snapshot.discardAttributes,
 			files:          snapshot.Files,
@@ -1093,7 +1097,7 @@ func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *C
 
 		encoder.buffer.Write([]byte("["))
 		sequence := uploadSequenceFunc(encoder,
-                                       func (fileSize int64, hash string) (io.Reader, bool) {
+			func(fileSize int64, hash string) (io.Reader, bool) {
 				return encoder.NextFile()
 			})
 		snapshot.SetSequence("files", sequence)
@@ -1108,8 +1112,9 @@ func (manager *BackupManager) UploadSnapshot(chunkMaker *ChunkMaker, uploader *C
 	}
 
 	path := fmt.Sprintf("snapshots/%s/%d", manager.snapshotID, snapshot.Revision)
+	if !manager.config.dryRun {
 		manager.SnapshotManager.UploadFile(path, path, description)
-
+	}
 	return totalSnapshotChunkSize, numberOfNewSnapshotChunks, totalUploadedSnapshotChunkSize, totalUploadedSnapshotChunkBytes
 }
 
@@ -1139,24 +1144,24 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 		if temporaryPath != fullPath {
 			os.Remove(temporaryPath)
 		}
-    } ()
+	}()
 
 	// These are used to break the existing file into chunks.
-    var existingChunks [] string
-    var existingLengths [] int
+	var existingChunks []string
+	var existingLengths []int
 
 	// These are to enable fast lookup of what chunks are available in the existing file.
-    offsetMap := make(map[string] int64)
-    lengthMap := make(map[string] int)
+	offsetMap := make(map[string]int64)
+	lengthMap := make(map[string]int)
 	var offset int64
 
 	existingFile, err = os.Open(fullPath)
 	if err != nil {
 		if os.IsNotExist(err) {
 			// macOS has no sparse file support
-            if inPlace && entry.Size > 100 * 1024 * 1024 && runtime.GOOS != "darwin" {
+			if inPlace && entry.Size > 100*1024*1024 && runtime.GOOS != "darwin" {
 				// Create an empty sparse file
-                existingFile, err = os.OpenFile(fullPath, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0600)
+				existingFile, err = os.OpenFile(fullPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 				if err != nil {
 					LOG_ERROR("DOWNLOAD_CREATE", "Failed to create the file %s for in-place writing: %v", fullPath, err)
 					return false
@@ -1165,10 +1170,10 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 				n := int64(1)
 				// There is a go bug on Windows (https://github.com/golang/go/issues/21681) that causes Seek to fail
 				// if the lower 32 bit of the offset argument is 0xffffffff.  Therefore we need to avoid that value by increasing n.
-                if uint32(entry.Size) == 0 && (entry.Size >> 32) > 0 {
+				if uint32(entry.Size) == 0 && (entry.Size>>32) > 0 {
 					n = int64(2)
 				}
-                _, err = existingFile.Seek(entry.Size - n, 0)
+				_, err = existingFile.Seek(entry.Size-n, 0)
 				if err != nil {
 					LOG_ERROR("DOWNLOAD_CREATE", "Failed to resize the initial file %s for in-place writing: %v", fullPath, err)
 					return false
@@ -1203,19 +1208,21 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 			// In inplace mode, we only consider chunks in the existing file with the same offsets, so we
 			// break the original file at offsets retrieved from the backup
 			fileHasher := manager.config.NewFileHasher()
-            buffer := make([]byte, 64 * 1024)
+			buffer := make([]byte, 64*1024)
 			err = nil
 			// We set to read one more byte so the file hash will be different if the file to be restored is a
 			// truncated portion of the existing file
-            for i := entry.StartChunk; i <= entry.EndChunk + 1; i++ {
+			for i := entry.StartChunk; i <= entry.EndChunk+1; i++ {
 				hasher := manager.config.NewKeyedHasher(manager.config.HashKey)
-                chunkSize := 1  // the size of extra chunk beyond EndChunk
+				chunkSize := 0
 				if i == entry.StartChunk {
-                    chunkSize -= entry.StartOffset
+					chunkSize = chunkDownloader.taskList[i].chunkLength - entry.StartOffset
 				} else if i == entry.EndChunk {
 					chunkSize = entry.EndOffset
 				} else if i > entry.StartChunk && i < entry.EndChunk {
 					chunkSize = chunkDownloader.taskList[i].chunkLength
+				} else {
+					chunkSize = 1 // the size of extra chunk beyond EndChunk
 				}
 				count := 0
 				for count < chunkSize {
@@ -1256,7 +1263,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 			// we run the chunk maker to split the original file.
 			chunkMaker.ForEachChunk(
 				existingFile,
-                func (chunk *Chunk, final bool) {
+				func(chunk *Chunk, final bool) {
 					hash := chunk.GetHash()
 					chunkSize := chunk.GetLength()
 					existingChunks = append(existingChunks, hash)
@@ -1265,7 +1272,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 					lengthMap[hash] = chunkSize
 					offset += int64(chunkSize)
 				},
-                func (fileSize int64, hash string) (io.Reader, bool) {
+				func(fileSize int64, hash string) (io.Reader, bool) {
 					fileHash = hash
 					return nil, false
 				})
@@ -1290,7 +1297,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 
 		if existingFile == nil {
 			// Create an empty file
-            existingFile, err = os.OpenFile(fullPath, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0600)
+			existingFile, err = os.OpenFile(fullPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 			if err != nil {
 				LOG_ERROR("DOWNLOAD_CREATE", "Failed to create the file %s for in-place writing", fullPath)
 			}
@@ -1349,12 +1356,12 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 				}
 			} else {
 				chunk := chunkDownloader.WaitForChunk(i)
-                _, err = existingFile.Write(chunk.GetBytes()[start: end])
+				_, err = existingFile.Write(chunk.GetBytes()[start:end])
 				if err != nil {
 					LOG_ERROR("DOWNLOAD_WRITE", "Failed to write to the file: %v", err)
 					return false
 				}
-                hasher.Write(chunk.GetBytes()[start: end])
+				hasher.Write(chunk.GetBytes()[start:end])
 			}
 
 			offset += int64(end - start)
@@ -1377,7 +1384,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 	} else {
 
 		// Create the temporary file.
-        newFile, err = os.OpenFile(temporaryPath, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0600)
+		newFile, err = os.OpenFile(temporaryPath, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 		if err != nil {
 			LOG_ERROR("DOWNLOAD_OPEN", "Failed to open file for writing: %v", err)
 			return false
@@ -1427,7 +1434,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 				if i == entry.EndChunk {
 					end = entry.EndOffset
 				}
-                data = chunk.GetBytes()[start: end]
+				data = chunk.GetBytes()[start:end]
 			}
 
 			_, err = newFile.Write(data)
@@ -1447,7 +1454,6 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
 			return false
 		}
 
-
 		if existingFile != nil {
 			existingFile.Close()
 			existingFile = nil
@@ -1500,9 +1506,8 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 		revisionMap[snapshotID][revision] = true
 	}
 
-    var snapshots [] *Snapshot
-    var otherSnapshots [] *Snapshot
-    var snapshotIDs [] string
+	var snapshots []*Snapshot
+	var snapshotIDs []string
 	var err error
 
 	if snapshotID == "" {
@@ -1512,7 +1517,7 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 			return false
 		}
 	} else {
-        snapshotIDs = [] string { snapshotID }
+		snapshotIDs = []string{snapshotID}
 	}
 
 	for _, id := range snapshotIDs {
@@ -1557,17 +1562,6 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 			snapshots = append(snapshots, snapshot)
 		}
 
-        otherRevisions, err := otherManager.SnapshotManager.ListSnapshotRevisions(id)
-        if err != nil {
-            LOG_ERROR("SNAPSHOT_LIST", "Failed to list all revisions at the destination for snapshot %s: %v", id, err)
-            return false
-        }
-
-        for _, otherRevision := range otherRevisions {
-            otherSnapshot := otherManager.SnapshotManager.DownloadSnapshot(id, otherRevision)
-            otherSnapshots = append(otherSnapshots, otherSnapshot)
-        }
-
 	}
 
 	if len(snapshots) == 0 {
@@ -1576,6 +1570,7 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 	}
 
 	chunks := make(map[string]bool)
+	otherChunks := make(map[string]bool)
 
 	for _, snapshot := range snapshots {
 
@@ -1586,16 +1581,22 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 		LOG_TRACE("SNAPSHOT_COPY", "Copying snapshot %s at revision %d", snapshot.ID, snapshot.Revision)
 
 		for _, chunkHash := range snapshot.FileSequence {
+			if _, found := chunks[chunkHash]; !found {
 				chunks[chunkHash] = true
 			}
+		}
 
 		for _, chunkHash := range snapshot.ChunkSequence {
+			if _, found := chunks[chunkHash]; !found {
 				chunks[chunkHash] = true
 			}
+		}
 
 		for _, chunkHash := range snapshot.LengthSequence {
+			if _, found := chunks[chunkHash]; !found {
 				chunks[chunkHash] = true
 			}
+		}
 
 		description := manager.SnapshotManager.DownloadSequence(snapshot.ChunkSequence)
 		err := snapshot.LoadChunks(description)
@@ -1606,44 +1607,42 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 		}
 
 		for _, chunkHash := range snapshot.ChunkHashes {
+			if _, found := chunks[chunkHash]; !found {
 				chunks[chunkHash] = true
 			}
 		}
+	}
 
-    for _, otherSnapshot := range otherSnapshots {
+	otherChunkFiles, otherChunkSizes := otherManager.SnapshotManager.ListAllFiles(otherManager.storage, "chunks/")
 
-        for _, chunkHash := range otherSnapshot.FileSequence {
-            if  _, found := chunks[chunkHash]; found {
-                chunks[chunkHash] = false
+	for i, otherChunkID := range otherChunkFiles {
+		otherChunkID = strings.Replace(otherChunkID, "/", "", -1)
+		if len(otherChunkID) != 64 {
+			continue
+		}
+		if otherChunkSizes[i] == 0 {
+			LOG_DEBUG("SNAPSHOT_COPY", "Chunk %s has length = 0", otherChunkID)
+			continue
+		}
+		otherChunks[otherChunkID] = false
+	}
+
+	LOG_DEBUG("SNAPSHOT_COPY", "Found %d chunks on destination storage", len(otherChunks))
+
+	chunksToCopy := 0
+	chunksToSkip := 0
+
+	for chunkHash, _ := range chunks {
+		otherChunkID := otherManager.config.GetChunkIDFromHash(chunkHash)
+		if _, found := otherChunks[otherChunkID]; found {
+			chunksToSkip++
+		} else {
+			chunksToCopy++
 		}
 	}
 
-        for _, chunkHash := range otherSnapshot.ChunkSequence {
-            if _, found := chunks[chunkHash]; found {
-                chunks[chunkHash] = false
-            }
-        }
-
-        for _, chunkHash := range otherSnapshot.LengthSequence {
-            if _, found := chunks[chunkHash]; found {
-                chunks[chunkHash] = false
-            }
-        }
-
-        description := otherManager.SnapshotManager.DownloadSequence(otherSnapshot.ChunkSequence)
-        err := otherSnapshot.LoadChunks(description)
-        if err != nil {
-            LOG_ERROR("SNAPSHOT_CHUNK", "Failed to load chunks for destination snapshot %s at revision %d: %v",
-                      otherSnapshot.ID, otherSnapshot.Revision, err)
-            return false
-        }
-
-        for _, chunkHash := range otherSnapshot.ChunkHashes {
-            if _, found := chunks[chunkHash]; found {
-                chunks[chunkHash] = false
-            }
-        }
-    }
+	LOG_DEBUG("SNAPSHOT_COPY", "Chunks to copy = %d, to skip = %d, total = %d", chunksToCopy, chunksToSkip, chunksToCopy+chunksToSkip)
+	LOG_DEBUG("SNAPSHOT_COPY", "Total chunks in source snapshot revisions = %d\n", len(chunks))
 
 	chunkDownloader := CreateChunkDownloader(manager.config, manager.storage, nil, false, threads)
 
@@ -1663,11 +1662,11 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 	totalSkipped := 0
 	chunkIndex := 0
 
-    for chunkHash, needsCopy := range chunks {
+	for chunkHash, _ := range chunks {
 		chunkIndex++
 		chunkID := manager.config.GetChunkIDFromHash(chunkHash)
-        if needsCopy {
 		newChunkID := otherManager.config.GetChunkIDFromHash(chunkHash)
+		if _, found := otherChunks[newChunkID]; !found {
 			LOG_DEBUG("SNAPSHOT_COPY", "Copying chunk %s to %s", chunkID, newChunkID)
 			i := chunkDownloader.AddChunk(chunkHash)
 			chunk := chunkDownloader.WaitForChunk(i)
@@ -1685,7 +1684,7 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
 	chunkDownloader.Stop()
 	chunkUploader.Stop()
 
-    LOG_INFO("SNAPSHOT_COPY", "Total chunks copied = %d, skipped = %d.", totalCopied, totalSkipped)
+	LOG_INFO("SNAPSHOT_COPY", "Copy complete, %d total chunks, %d chunks copied, %d skipped", totalCopied+totalSkipped, totalCopied, totalSkipped)
 
 	for _, snapshot := range snapshots {
 		if revisionMap[snapshot.ID][snapshot.Revision] == false {

src/duplicacy_backupmanager_test.go

@@ -5,21 +5,21 @@
 package duplicacy
 
 import (
-    "os"
+	crypto_rand "crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
 	"io"
+	"math/rand"
+	"os"
 	"path"
 	"testing"
-    "math/rand"
-    "encoding/hex"
 	"time"
-    "crypto/sha256"
-    crypto_rand "crypto/rand"
 
 	"runtime/debug"
 )
 
 func createRandomFile(path string, maxSize int) {
-    file, err := os.OpenFile(path, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0644)
+	file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
 	if err != nil {
 		LOG_ERROR("RANDOM_FILE", "Can't open %s for writing: %v", path, err)
 		return
@@ -27,9 +27,9 @@ func createRandomFile(path string, maxSize int) {
 
 	defer file.Close()
 
-    size := maxSize / 2 + rand.Int() % (maxSize / 2)
+	size := maxSize/2 + rand.Int()%(maxSize/2)
 
-    buffer := make([]byte, 32 * 1024)
+	buffer := make([]byte, 32*1024)
 	for size > 0 {
 		bytes := size
 		if bytes > cap(buffer) {
@@ -65,7 +65,7 @@ func modifyFile(path string, portion float32) {
 		if file != nil {
 			file.Close()
 		}
-    } ()
+	}()
 
 	size, err := file.Seek(0, 2)
 	if err != nil {
@@ -73,7 +73,7 @@ func modifyFile(path string, portion float32) {
 		return
 	}
 
-    length := int (float32(size) * portion)
+	length := int(float32(size) * portion)
 	start := rand.Int() % (int(size) - length)
 
 	_, err = file.Seek(int64(start), 0)
@@ -186,28 +186,28 @@ func TestBackupManager(t *testing.T) {
 				debug.PrintStack()
 			}
 		}
-    } ()
+	}()
 
 	testDir := path.Join(os.TempDir(), "duplicacy_test")
 	os.RemoveAll(testDir)
 	os.MkdirAll(testDir, 0700)
 
-    os.Mkdir(testDir + "/repository1", 0700)
-    os.Mkdir(testDir + "/repository1/dir1", 0700)
-    os.Mkdir(testDir + "/repository1/.duplicacy", 0700)
-    os.Mkdir(testDir + "/repository2", 0700)
-    os.Mkdir(testDir + "/repository2/.duplicacy", 0700)
+	os.Mkdir(testDir+"/repository1", 0700)
+	os.Mkdir(testDir+"/repository1/dir1", 0700)
+	os.Mkdir(testDir+"/repository1/.duplicacy", 0700)
+	os.Mkdir(testDir+"/repository2", 0700)
+	os.Mkdir(testDir+"/repository2/.duplicacy", 0700)
 
 	maxFileSize := 1000000
 	//maxFileSize := 200000
 
-    createRandomFile(testDir + "/repository1/file1", maxFileSize)
-    createRandomFile(testDir + "/repository1/file2", maxFileSize)
-    createRandomFile(testDir + "/repository1/dir1/file3", maxFileSize)
+	createRandomFile(testDir+"/repository1/file1", maxFileSize)
+	createRandomFile(testDir+"/repository1/file2", maxFileSize)
+	createRandomFile(testDir+"/repository1/dir1/file3", maxFileSize)
 
 	threads := 1
 
-    storage, err := loadStorage(testDir + "/storage", threads)
+	storage, err := loadStorage(testDir+"/storage", threads)
 	if err != nil {
 		t.Errorf("Failed to create storage: %v", err)
 		return
@@ -227,16 +227,15 @@ func TestBackupManager(t *testing.T) {
 
 	time.Sleep(time.Duration(delay) * time.Second)
 	if testFixedChunkSize {
-        if !ConfigStorage(storage, 100, 64 * 1024, 64 * 1024, 64 * 1024, password, nil) {
+		if !ConfigStorage(storage, 16384, 100, 64*1024, 64*1024, 64*1024, password, nil, false) {
 			t.Errorf("Failed to initialize the storage")
 		}
 	} else {
-        if !ConfigStorage(storage, 100, 64 * 1024, 256 * 1024, 16 * 1024, password, nil) {
+		if !ConfigStorage(storage, 16384, 100, 64*1024, 256*1024, 16*1024, password, nil, false) {
 			t.Errorf("Failed to initialize the storage")
 		}
 	}
 
-
 	time.Sleep(time.Duration(delay) * time.Second)
 
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
@@ -244,13 +243,13 @@ func TestBackupManager(t *testing.T) {
 	backupManager.SetupSnapshotCache("default")
 
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-    backupManager.Backup(testDir + "/repository1", /*quickMode=*/true, threads, "first", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false)
 	time.Sleep(time.Duration(delay) * time.Second)
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-    backupManager.Restore(testDir + "/repository2", threads, /*inPlace=*/false, /*quickMode=*/false, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/nil)
+	backupManager.Restore(testDir+"/repository2", threads, /*inPlace=*/false, /*quickMode=*/false, threads, /*overwrite=*/true,
+		/*deleteMode=*/false, /*setowner=*/false, /*showStatistics=*/false, /*patterns=*/ nil)
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
 		if _, err := os.Stat(testDir + "/repository2/" + f); os.IsNotExist(err) {
 			t.Errorf("File %s does not exist", f)
 			continue
@@ -263,18 +262,18 @@ func TestBackupManager(t *testing.T) {
 		}
 	}
 
-    modifyFile(testDir + "/repository1/file1", 0.1)
-    modifyFile(testDir + "/repository1/file2", 0.2)
-    modifyFile(testDir + "/repository1/dir1/file3", 0.3)
+	modifyFile(testDir+"/repository1/file1", 0.1)
+	modifyFile(testDir+"/repository1/file2", 0.2)
+	modifyFile(testDir+"/repository1/dir1/file3", 0.3)
 
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-    backupManager.Backup(testDir + "/repository1", /*quickMode=*/true, threads, "second", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "second", false, false)
 	time.Sleep(time.Duration(delay) * time.Second)
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-    backupManager.Restore(testDir + "/repository2", 2, /*inPlace=*/true, /*quickMode=*/true, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/nil)
+	backupManager.Restore(testDir+"/repository2", 2, /*inPlace=*/true, /*quickMode=*/true, threads, /*overwrite=*/true,
+		/*deleteMode=*/false, /*setowner=*/false, /*showStatistics=*/false, /*patterns=*/nil)
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
 		hash1 := getFileHash(testDir + "/repository1/" + f)
 		hash2 := getFileHash(testDir + "/repository2/" + f)
 		if hash1 != hash2 {
@@ -284,25 +283,25 @@ func TestBackupManager(t *testing.T) {
 
 	// Truncate file2 and add a few empty directories
 	truncateFile(testDir + "/repository1/file2")
-    os.Mkdir(testDir + "/repository1/dir2", 0700)
-    os.Mkdir(testDir + "/repository1/dir2/dir3", 0700)
-    os.Mkdir(testDir + "/repository1/dir4", 0700)
+	os.Mkdir(testDir+"/repository1/dir2", 0700)
+	os.Mkdir(testDir+"/repository1/dir2/dir3", 0700)
+	os.Mkdir(testDir+"/repository1/dir4", 0700)
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-    backupManager.Backup(testDir + "/repository1", /*quickMode=*/false, threads, "third", false, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "third", false, false)
 	time.Sleep(time.Duration(delay) * time.Second)
 
 	// Create some directories and files under repository2 that will be deleted during restore
-    os.Mkdir(testDir + "/repository2/dir5", 0700)
-    os.Mkdir(testDir + "/repository2/dir5/dir6", 0700)
-    os.Mkdir(testDir + "/repository2/dir7", 0700)
-    createRandomFile(testDir + "/repository2/file4", 100)
-    createRandomFile(testDir + "/repository2/dir5/file5", 100)
+	os.Mkdir(testDir+"/repository2/dir5", 0700)
+	os.Mkdir(testDir+"/repository2/dir5/dir6", 0700)
+	os.Mkdir(testDir+"/repository2/dir7", 0700)
+	createRandomFile(testDir+"/repository2/file4", 100)
+	createRandomFile(testDir+"/repository2/dir5/file5", 100)
 
 	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
-    backupManager.Restore(testDir + "/repository2", 3, /*inPlace=*/true, /*quickMode=*/false, threads, /*overwrite=*/true,
-                          /*deleteMode=*/true, /*showStatistics=*/false, /*patterns=*/nil)
+	backupManager.Restore(testDir+"/repository2", 3, /*inPlace=*/true, /*quickMode=*/false, threads, /*overwrite=*/true,
+		/*deleteMode=*/true, /*setowner=*/false, /*showStatistics=*/false, /*patterns=*/nil)
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
 		hash1 := getFileHash(testDir + "/repository1/" + f)
 		hash2 := getFileHash(testDir + "/repository2/" + f)
 		if hash1 != hash2 {
@@ -311,25 +310,25 @@ func TestBackupManager(t *testing.T) {
 	}
 
 	// These files/dirs should not exist because deleteMode == true
-    checkExistence(t, testDir + "/repository2/dir5", false, false);
-    checkExistence(t, testDir + "/repository2/dir5/dir6", false, false);
-    checkExistence(t, testDir + "/repository2/dir7", false, false);
-    checkExistence(t, testDir + "/repository2/file4", false, false);
-    checkExistence(t, testDir + "/repository2/dir5/file5", false, false);
+	checkExistence(t, testDir+"/repository2/dir5", false, false)
+	checkExistence(t, testDir+"/repository2/dir5/dir6", false, false)
+	checkExistence(t, testDir+"/repository2/dir7", false, false)
+	checkExistence(t, testDir+"/repository2/file4", false, false)
+	checkExistence(t, testDir+"/repository2/dir5/file5", false, false)
 
 	// These empty dirs should exist
-    checkExistence(t, testDir + "/repository2/dir2", true, true);
-    checkExistence(t, testDir + "/repository2/dir2/dir3", true, true);
-    checkExistence(t, testDir + "/repository2/dir4", true, true);
+	checkExistence(t, testDir+"/repository2/dir2", true, true)
+	checkExistence(t, testDir+"/repository2/dir2/dir3", true, true)
+	checkExistence(t, testDir+"/repository2/dir4", true, true)
 
 	// Remove file2 and dir1/file3 and restore them from revision 3
 	os.Remove(testDir + "/repository1/file2")
 	os.Remove(testDir + "/repository1/dir1/file3")
 	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
-    backupManager.Restore(testDir + "/repository1", 3, /*inPlace=*/true, /*quickMode=*/false, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/[]string{"+file2", "+dir1/file3", "-*"})
+	backupManager.Restore(testDir+"/repository1", 3, /*inPlace=*/true, /*quickMode=*/false, threads, /*overwrite=*/true,
+		/*deleteMode=*/false,  /*setowner=*/false, /*showStatistics=*/false, /*patterns=*/[]string{"+file2", "+dir1/file3", "-*"})
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
 		hash1 := getFileHash(testDir + "/repository1/" + f)
 		hash2 := getFileHash(testDir + "/repository2/" + f)
 		if hash1 != hash2 {
@@ -337,6 +336,30 @@ func TestBackupManager(t *testing.T) {
 		}
 	}
 
+	numberOfSnapshots := backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
+	if numberOfSnapshots != 3 {
+		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
+	}
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1, 2, 3} /*tag*/, "",
+		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
+	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, []int{1} /*tags*/, nil /*retentions*/, nil,
+		/*exhaustive*/ false /*exclusive=*/, false /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false)
+	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
+	if numberOfSnapshots != 2 {
+		t.Errorf("Expected 2 snapshots but got %d", numberOfSnapshots)
+	}
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3} /*tag*/, "",
+		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "fourth", false, false)
+	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, nil /*tags*/, nil /*retentions*/, nil,
+		/*exhaustive*/ false /*exclusive=*/, true /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false)
+	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
+	if numberOfSnapshots != 3 {
+		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
+	}
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3, 4} /*tag*/, "",
+		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
+
 	/*buf := make([]byte, 1<<16)
 	  runtime.Stack(buf, true)
 	  fmt.Printf("%s", buf)*/

src/duplicacy_chunk.go

@@ -5,28 +5,27 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "hash"
 	"bytes"
-    "runtime"
-    "crypto/cipher"
+	"compress/zlib"
 	"crypto/aes"
+	"crypto/cipher"
 	"crypto/rand"
 	"encoding/hex"
-    "compress/zlib"
+	"fmt"
+	"hash"
+	"io"
+	"runtime"
 
 	"github.com/bkaradzic/go-lz4"
-
 )
 
 // A chunk needs to acquire a new buffer and return the old one for every encrypt/decrypt operation, therefore
 // we maintain a pool of previously used buffers.
-var chunkBufferPool chan *bytes.Buffer = make(chan *bytes.Buffer, runtime.NumCPU() * 16)
+var chunkBufferPool chan *bytes.Buffer = make(chan *bytes.Buffer, runtime.NumCPU()*16)
 
 func AllocateChunkBuffer() (buffer *bytes.Buffer) {
 	select {
-    case buffer = <- chunkBufferPool:
+	case buffer = <-chunkBufferPool:
 	default:
 		buffer = new(bytes.Buffer)
 	}
@@ -76,9 +75,9 @@ func CreateChunk(config *Config, bufferNeeded bool) *Chunk {
 		}
 	}
 
-    return &Chunk {
-        buffer     : buffer,
-        config     : config,
+	return &Chunk{
+		buffer: buffer,
+		config: config,
 	}
 }
 
@@ -92,7 +91,7 @@ func (chunk *Chunk) GetLength() int {
 }
 
 // GetBytes returns data available in this chunk
-func (chunk *Chunk) GetBytes() [] byte {
+func (chunk *Chunk) GetBytes() []byte {
 	return chunk.buffer.Bytes()
 }
 
@@ -114,7 +113,7 @@ func (chunk *Chunk) Reset(hashNeeded bool) {
 }
 
 // Write implements the Writer interface.
-func (chunk *Chunk) Write(p []byte) (int, error){
+func (chunk *Chunk) Write(p []byte) (int, error) {
 
 	// buffer may be nil, when the chunk is used for computing the hash only.
 	if chunk.buffer == nil {
@@ -132,7 +131,7 @@ func (chunk *Chunk) Write(p []byte) (int, error){
 
 // GetHash returns the chunk hash.
 func (chunk *Chunk) GetHash() string {
-    if (len(chunk.hash) == 0) {
+	if len(chunk.hash) == 0 {
 		chunk.hash = chunk.hasher.Sum(nil)
 	}
 
@@ -179,7 +178,7 @@ func (chunk *Chunk) Encrypt(encryptionKey []byte, derivationKey string) (err err
 	encryptedBuffer.Reset()
 	defer func() {
 		ReleaseChunkBuffer(encryptedBuffer)
-    } ()
+	}()
 
 	if len(encryptionKey) > 0 {
 
@@ -229,7 +228,7 @@ func (chunk *Chunk) Encrypt(encryptionKey []byte, derivationKey string) (err err
 		if availableLength < maximumLength {
 			encryptedBuffer.Grow(maximumLength - availableLength)
 		}
-        written, err := lz4.Encode(encryptedBuffer.Bytes()[offset + 4:], chunk.buffer.Bytes())
+		written, err := lz4.Encode(encryptedBuffer.Bytes()[offset+4:], chunk.buffer.Bytes())
 		if err != nil {
 			return fmt.Errorf("LZ4 compression error: %v", err)
 		}
@@ -258,7 +257,7 @@ func (chunk *Chunk) Encrypt(encryptionKey []byte, derivationKey string) (err err
 
 	// The encrypted data will be appended to the duplicacy header and the once.
 	encryptedBytes := gcm.Seal(encryptedBuffer.Bytes()[:offset], nonce,
-                               encryptedBuffer.Bytes()[offset: offset + dataLength + paddingLength], nil)
+		encryptedBuffer.Bytes()[offset:offset+dataLength+paddingLength], nil)
 
 	encryptedBuffer.Truncate(len(encryptedBytes))
 
@@ -278,7 +277,7 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 	encryptedBuffer.Reset()
 	defer func() {
 		ReleaseChunkBuffer(encryptedBuffer)
-    } ()
+	}()
 
 	chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
 
@@ -309,15 +308,15 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 			return fmt.Errorf("No enough encrypted data (%d bytes) provided", len(encryptedBuffer.Bytes()))
 		}
 
-        if string(encryptedBuffer.Bytes()[:headerLength - 1]) != ENCRYPTION_HEADER[:headerLength - 1] {
+		if string(encryptedBuffer.Bytes()[:headerLength-1]) != ENCRYPTION_HEADER[:headerLength-1] {
 			return fmt.Errorf("The storage doesn't seem to be encrypted")
 		}
 
-        if encryptedBuffer.Bytes()[headerLength - 1] != 0 {
-            return fmt.Errorf("Unsupported encryption version %d", encryptedBuffer.Bytes()[headerLength - 1])
+		if encryptedBuffer.Bytes()[headerLength-1] != 0 {
+			return fmt.Errorf("Unsupported encryption version %d", encryptedBuffer.Bytes()[headerLength-1])
 		}
 
-        nonce := encryptedBuffer.Bytes()[headerLength: offset]
+		nonce := encryptedBuffer.Bytes()[headerLength:offset]
 
 		decryptedBytes, err := gcm.Open(encryptedBuffer.Bytes()[:offset], nonce,
 			encryptedBuffer.Bytes()[offset:], nil)
@@ -326,7 +325,7 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 			return err
 		}
 
-        paddingLength := int(decryptedBytes[len(decryptedBytes) - 1])
+		paddingLength := int(decryptedBytes[len(decryptedBytes)-1])
 		if paddingLength == 0 {
 			paddingLength = 256
 		}
@@ -335,10 +334,10 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 		}
 
 		for i := 0; i < paddingLength; i++ {
-            padding := decryptedBytes[len(decryptedBytes) - 1 - i]
+			padding := decryptedBytes[len(decryptedBytes)-1-i]
 			if padding != byte(paddingLength) {
 				return fmt.Errorf("Incorrect padding of length %d: %x", paddingLength,
-                                  decryptedBytes[len(decryptedBytes) - paddingLength:])
+					decryptedBytes[len(decryptedBytes)-paddingLength:])
 			}
 		}
 
@@ -379,4 +378,3 @@ func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err err
 	return nil
 
 }
-

src/duplicacy_chunk_test.go

@@ -5,10 +5,10 @@
 package duplicacy
 
 import (
-    "testing"
 	"bytes"
 	crypto_rand "crypto/rand"
 	"math/rand"
+	"testing"
 )
 
 func TestChunk(t *testing.T) {
@@ -67,7 +67,6 @@ func TestChunk(t *testing.T) {
 			t.Errorf("Original data:\n%x\nDecrypted data:\n%x\n", plainData, decryptedData)
 		}
 
-
 	}
 
 }

src/duplicacy_chunkdownloader.go

@@ -5,6 +5,7 @@
 package duplicacy
 
 import (
+	"io"
 	"sync/atomic"
 	"time"
 )
@@ -36,7 +37,7 @@ type ChunkDownloader struct {
 	showStatistics bool         // Show a stats log for each chunk if true
 	threads        int          // Number of threads
 
-    taskList [] ChunkDownloadTask                      // The list of chunks to be downloaded
+	taskList       []ChunkDownloadTask // The list of chunks to be downloaded
 	completedTasks map[int]bool        // Store downloaded chunks
 	lastChunkIndex int                 // a monotonically increasing number indicating the last chunk to be downloaded
 
@@ -53,7 +54,7 @@ type ChunkDownloader struct {
 }
 
 func CreateChunkDownloader(config *Config, storage Storage, snapshotCache *FileStorage, showStatistics bool, threads int) *ChunkDownloader {
-    downloader := &ChunkDownloader {
+	downloader := &ChunkDownloader{
 		config:         config,
 		storage:        storage,
 		snapshotCache:  snapshotCache,
@@ -77,20 +78,20 @@ func CreateChunkDownloader(config *Config, storage Storage, snapshotCache *FileS
 			defer CatchLogException()
 			for {
 				select {
-                case task := <- downloader.taskQueue:
+				case task := <-downloader.taskQueue:
 					downloader.Download(threadIndex, task)
-                case <- downloader.stopChannel:
+				case <-downloader.stopChannel:
 					return
 				}
 			}
-        } (i)
+		}(i)
 	}
 
 	return downloader
 }
 
 // AddFiles adds chunks needed by the specified files to the download list.
-func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files [] *Entry) {
+func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files []*Entry) {
 
 	downloader.taskList = nil
 	lastChunkIndex := -1
@@ -102,7 +103,7 @@ func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files [] *Entry)
 		}
 		for i := file.StartChunk; i <= file.EndChunk; i++ {
 			if lastChunkIndex != i {
-                task := ChunkDownloadTask {
+				task := ChunkDownloadTask{
 					chunkIndex:  len(downloader.taskList),
 					chunkHash:   snapshot.ChunkHashes[i],
 					chunkLength: snapshot.ChunkLengths[i],
@@ -111,13 +112,13 @@ func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files [] *Entry)
 				downloader.taskList = append(downloader.taskList, task)
 				downloader.totalChunkSize += int64(snapshot.ChunkLengths[i])
 			} else {
-                downloader.taskList[len(downloader.taskList) - 1].needed = true
+				downloader.taskList[len(downloader.taskList)-1].needed = true
 			}
 			lastChunkIndex = i
 		}
 		file.StartChunk = len(downloader.taskList) - (file.EndChunk - file.StartChunk) - 1
 		file.EndChunk = len(downloader.taskList) - 1
-        if file.EndChunk - file.StartChunk > maximumChunks {
+		if file.EndChunk-file.StartChunk > maximumChunks {
 			maximumChunks = file.EndChunk - file.StartChunk
 		}
 	}
@@ -125,7 +126,7 @@ func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files [] *Entry)
 
 // AddChunk adds a single chunk the download list.
 func (downloader *ChunkDownloader) AddChunk(chunkHash string) int {
-    task := ChunkDownloadTask {
+	task := ChunkDownloadTask{
 		chunkIndex:    len(downloader.taskList),
 		chunkHash:     chunkHash,
 		chunkLength:   0,
@@ -137,7 +138,7 @@ func (downloader *ChunkDownloader) AddChunk(chunkHash string) int {
 		downloader.taskQueue <- task
 		downloader.numberOfDownloadingChunks++
 		downloader.numberOfActiveChunks++
-        downloader.taskList[len(downloader.taskList) - 1].isDownloading = true
+		downloader.taskList[len(downloader.taskList)-1].isDownloading = true
 	}
 	return len(downloader.taskList) - 1
 }
@@ -163,7 +164,7 @@ func (downloader *ChunkDownloader) Prefetch(file *Entry) {
 				downloader.numberOfDownloadingChunks++
 				downloader.numberOfActiveChunks++
 			}
-        } else{
+		} else {
 			LOG_DEBUG("DOWNLOAD_PREFETCH", "%s chunk %s is not needed", file.Path,
 				downloader.config.GetChunkIDFromHash(task.chunkHash))
 		}
@@ -226,7 +227,7 @@ func (downloader *ChunkDownloader) WaitForChunk(chunkIndex int) (chunk *Chunk) {
 
 	// Now wait until the chunk to be downloaded appears in the completed tasks
 	for _, found := downloader.completedTasks[chunkIndex]; !found; _, found = downloader.completedTasks[chunkIndex] {
-        completion := <- downloader.completionChannel
+		completion := <-downloader.completionChannel
 		downloader.completedTasks[completion.chunkIndex] = true
 		downloader.taskList[completion.chunkIndex].chunk = completion.chunk
 		downloader.numberOfDownloadedChunks++
@@ -238,7 +239,7 @@ func (downloader *ChunkDownloader) WaitForChunk(chunkIndex int) (chunk *Chunk) {
 // Stop terminates all downloading goroutines
 func (downloader *ChunkDownloader) Stop() {
 	for downloader.numberOfDownloadingChunks > 0 {
-        completion := <- downloader.completionChannel
+		completion := <-downloader.completionChannel
 		downloader.completedTasks[completion.chunkIndex] = true
 		downloader.taskList[completion.chunkIndex].chunk = completion.chunk
 		downloader.numberOfDownloadedChunks++
@@ -286,7 +287,7 @@ func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadT
 				} else {
 					LOG_DEBUG("CHUNK_CACHE", "Chunk %s has been loaded from the snapshot cache", chunkID)
 
-                    downloader.completionChannel <- ChunkDownloadCompletion{ chunk: chunk, chunkIndex:task.chunkIndex }
+					downloader.completionChannel <- ChunkDownloadCompletion{chunk: chunk, chunkIndex: task.chunkIndex}
 					return false
 				}
 			}
@@ -297,6 +298,9 @@ func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadT
 	// will be set up before the encryption
 	chunk.Reset(false)
 
+	const MaxDownloadAttempts = 3
+	for downloadAttempt := 0; ; downloadAttempt++ {
+
 		// Find the chunk by ID first.
 		chunkPath, exist, _, err := downloader.storage.FindChunk(threadIndex, chunkID, false)
 		if err != nil {
@@ -306,13 +310,19 @@ func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadT
 
 		if !exist {
 			// No chunk is found.  Have to find it in the fossil pool again.
-        chunkPath, exist, _, err = downloader.storage.FindChunk(threadIndex, chunkID, true)
+			fossilPath, exist, _, err := downloader.storage.FindChunk(threadIndex, chunkID, true)
 			if err != nil {
 				LOG_ERROR("DOWNLOAD_CHUNK", "Failed to find the chunk %s: %v", chunkID, err)
 				return false
 			}
 
 			if !exist {
+				// Retry for the Hubic backend as it may return 404 even when the chunk exists
+				if _, ok := downloader.storage.(*HubicStorage); ok && downloadAttempt < MaxDownloadAttempts {
+					LOG_WARN("DOWNLOAD_RETRY", "Failed to find the chunk %s; retrying", chunkID)
+					continue
+				}
+
 				// A chunk is not found.  This is a serious error and hopefully it will never happen.
 				if err != nil {
 					LOG_FATAL("DOWNLOAD_CHUNK", "Chunk %s can't be found: %v", chunkID, err)
@@ -321,31 +331,63 @@ func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadT
 				}
 				return false
 			}
-        LOG_DEBUG("CHUNK_FOSSIL", "Chunk %s has been marked as a fossil", chunkID)
+
+			// We can't download the fossil directly.  We have to turn it back into a regular chunk and try
+			// downloading again.
+			err = downloader.storage.MoveFile(threadIndex, fossilPath, chunkPath)
+			if err != nil {
+				LOG_FATAL("DOWNLOAD_CHUNK", "Failed to resurrect chunk %s: %v", chunkID, err)
+				return false
+			}
+
+			LOG_WARN("DOWNLOAD_RESURRECT", "Fossil %s has been resurrected", chunkID)
+			continue
 		}
 
 		err = downloader.storage.DownloadFile(threadIndex, chunkPath, chunk)
 		if err != nil {
-        LOG_ERROR("UPLOAD_FATAL", "Failed to download the chunk %s: %v", chunkID, err)
+			_, isHubic := downloader.storage.(*HubicStorage)
+			// Retry on EOF or if it is a Hubic backend as it may return 404 even when the chunk exists
+			if (err == io.ErrUnexpectedEOF || isHubic) && downloadAttempt < MaxDownloadAttempts {
+				LOG_WARN("DOWNLOAD_RETRY", "Failed to download the chunk %s: %v; retrying", chunkID, err)
+				chunk.Reset(false)
+				continue
+			} else {
+				LOG_ERROR("DOWNLOAD_CHUNK", "Failed to download the chunk %s: %v", chunkID, err)
 				return false
 			}
-
+		}
 
 		err = chunk.Decrypt(downloader.config.ChunkKey, task.chunkHash)
 		if err != nil {
-        LOG_ERROR("UPLOAD_CHUNK", "Failed to decrypt the chunk %s: %v", chunkID, err)
+			if downloadAttempt < MaxDownloadAttempts {
+				LOG_WARN("DOWNLOAD_RETRY", "Failed to decrypt the chunk %s: %v; retrying", chunkID, err)
+				chunk.Reset(false)
+				continue
+			} else {
+				LOG_ERROR("DOWNLOAD_DECRYPT", "Failed to decrypt the chunk %s: %v", chunkID, err)
 				return false
 			}
+		}
 
 		actualChunkID := chunk.GetID()
 		if actualChunkID != chunkID {
-        LOG_FATAL("UPLOAD_CORRUPTED", "The chunk %s has a hash id of %s", chunkID, actualChunkID)
+			if downloadAttempt < MaxDownloadAttempts {
+				LOG_WARN("DOWNLOAD_RETRY", "The chunk %s has a hash id of %s; retrying", chunkID, actualChunkID)
+				chunk.Reset(false)
+				continue
+			} else {
+				LOG_FATAL("DOWNLOAD_CORRUPTED", "The chunk %s has a hash id of %s", chunkID, actualChunkID)
 				return false
 			}
+		}
+
+		break
+	}
 
 	if len(cachedPath) > 0 {
 		// Save a copy to the local snapshot cache
-        err = downloader.snapshotCache.UploadFile(threadIndex, cachedPath, chunk.GetBytes())
+		err := downloader.snapshotCache.UploadFile(threadIndex, cachedPath, chunk.GetBytes())
 		if err != nil {
 			LOG_WARN("DOWNLOAD_CACHE", "Failed to add the chunk %s to the snapshot cache: %v", chunkID, err)
 		}
@@ -362,16 +404,16 @@ func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadT
 		speed := downloadedChunkSize / (now - downloader.startTime)
 		remainingTime := int64(0)
 		if speed > 0 {
-            remainingTime = (downloader.totalChunkSize - downloadedChunkSize) / speed + 1
+			remainingTime = (downloader.totalChunkSize-downloadedChunkSize)/speed + 1
 		}
 		percentage := float32(downloadedChunkSize * 1000 / downloader.totalChunkSize)
 		LOG_INFO("DOWNLOAD_PROGRESS", "Downloaded chunk %d size %d, %sB/s %s %.1f%%",
-                    task.chunkIndex + 1, chunk.GetLength(),
-                    PrettySize(speed), PrettyTime(remainingTime), percentage / 10)
+			task.chunkIndex+1, chunk.GetLength(),
+			PrettySize(speed), PrettyTime(remainingTime), percentage/10)
 	} else {
 		LOG_DEBUG("CHUNK_DOWNLOAD", "Chunk %s has been downloaded", chunkID)
 	}
 
-    downloader.completionChannel <- ChunkDownloadCompletion{ chunk: chunk, chunkIndex:task.chunkIndex }
+	downloader.completionChannel <- ChunkDownloadCompletion{chunk: chunk, chunkIndex: task.chunkIndex}
 	return true
 }

src/duplicacy_chunkmaker.go

@@ -5,10 +5,10 @@
 package duplicacy
 
 import (
-    "io"
 	"crypto/sha256"
-    "encoding/hex"
 	"encoding/binary"
+	"encoding/hex"
+	"io"
 )
 
 // ChunkMaker breaks data into chunks using buzhash.  To save memory, the chunk maker only use a circular buffer
@@ -35,7 +35,7 @@ type ChunkMaker struct {
 // buzhash.
 func CreateChunkMaker(config *Config, hashOnly bool) *ChunkMaker {
 	size := 1
-    for size * 2 <= config.AverageChunkSize {
+	for size*2 <= config.AverageChunkSize {
 		size *= 2
 	}
 
@@ -44,7 +44,7 @@ func CreateChunkMaker(config *Config, hashOnly bool) *ChunkMaker {
 		return nil
 	}
 
-    maker := &ChunkMaker {
+	maker := &ChunkMaker{
 		hashMask:         uint64(config.AverageChunkSize - 1),
 		maximumChunkSize: config.MaximumChunkSize,
 		minimumChunkSize: config.MinimumChunkSize,
@@ -61,12 +61,12 @@ func CreateChunkMaker(config *Config, hashOnly bool) *ChunkMaker {
 
 	for i := 0; i < 64; i++ {
 		for j := 0; j < 4; j++ {
-            maker.randomTable[4 * i + j] = binary.LittleEndian.Uint64(randomData[8 * j : 8 * j + 8])
+			maker.randomTable[4*i+j] = binary.LittleEndian.Uint64(randomData[8*j : 8*j+8])
 		}
 		randomData = sha256.Sum256(randomData[:])
 	}
 
-    maker.buffer = make([]byte, 2 * config.MinimumChunkSize)
+	maker.buffer = make([]byte, 2*config.MinimumChunkSize)
 
 	return maker
 }
@@ -79,7 +79,7 @@ func rotateLeftByOne(value uint64) uint64 {
 	return (value << 1) | (value >> 63)
 }
 
-func (maker *ChunkMaker) buzhashSum(sum uint64, data [] byte) uint64 {
+func (maker *ChunkMaker) buzhashSum(sum uint64, data []byte) uint64 {
 	for i := 0; i < len(data); i++ {
 		sum = rotateLeftByOne(sum) ^ maker.randomTable[data[i]]
 	}
@@ -94,7 +94,7 @@ func (maker *ChunkMaker) buzhashUpdate(sum uint64, out byte, in byte, length int
 // 'nextReader' returns false, it will process remaining data in the buffer and then quit.  When a chunk is identified,
 // it will call 'endOfChunk' to return the chunk size and a boolean flag indicating if it is the last chunk.
 func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *Chunk, final bool),
-                                      nextReader func(size int64, hash string)(io.Reader, bool)) {
+	nextReader func(size int64, hash string) (io.Reader, bool)) {
 
 	maker.bufferStart = 0
 	maker.bufferSize = 0
@@ -121,13 +121,13 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 
 	// Move data from the buffer to the chunk.
 	fill := func(count int) {
-        if maker.bufferStart + count < maker.bufferCapacity {
-            chunk.Write(maker.buffer[maker.bufferStart : maker.bufferStart + count])
+		if maker.bufferStart+count < maker.bufferCapacity {
+			chunk.Write(maker.buffer[maker.bufferStart : maker.bufferStart+count])
 			maker.bufferStart += count
 			maker.bufferSize -= count
 		} else {
-            chunk.Write(maker.buffer[maker.bufferStart :])
-            chunk.Write(maker.buffer[: count - (maker.bufferCapacity - maker.bufferStart)])
+			chunk.Write(maker.buffer[maker.bufferStart:])
+			chunk.Write(maker.buffer[:count-(maker.bufferCapacity-maker.bufferStart)])
 			maker.bufferStart = count - (maker.bufferCapacity - maker.bufferStart)
 			maker.bufferSize -= count
 		}
@@ -148,7 +148,7 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 		for {
 			maker.bufferStart = 0
 			for maker.bufferStart < maker.minimumChunkSize && !isEOF {
-                count, err := reader.Read(maker.buffer[maker.bufferStart : maker.minimumChunkSize])
+				count, err := reader.Read(maker.buffer[maker.bufferStart:maker.minimumChunkSize])
 
 				if err != nil {
 					if err != io.EOF {
@@ -197,7 +197,7 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 				count = maker.bufferStart - start
 			}
 
-            count, err = reader.Read(maker.buffer[start : start + count])
+			count, err = reader.Read(maker.buffer[start : start+count])
 
 			if err != nil && err != io.EOF {
 				LOG_ERROR("CHUNK_MAKER", "Failed to read %d bytes: %s", count, err.Error())
@@ -205,7 +205,7 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 			}
 
 			maker.bufferSize += count
-            fileHasher.Write(maker.buffer[start : start + count])
+			fileHasher.Write(maker.buffer[start : start+count])
 			fileSize += int64(count)
 
 			// if EOF is seen, try to switch to next file and continue
@@ -229,18 +229,17 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 			return
 		}
 
-
 		// Minimum chunk size has been reached.  Calculate the buzhash for the minimum size chunk.
-        if (!minimumReached) {
+		if !minimumReached {
 
 			bytes := maker.minimumChunkSize
 
-            if maker.bufferStart + bytes < maker.bufferCapacity {
-                hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart : maker.bufferStart + bytes])
+			if maker.bufferStart+bytes < maker.bufferCapacity {
+				hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart:maker.bufferStart+bytes])
 			} else {
-                hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart :])
+				hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart:])
 				hashSum = maker.buzhashSum(hashSum,
-                                           maker.buffer[: bytes - (maker.bufferCapacity - maker.bufferStart)])
+					maker.buffer[:bytes-(maker.bufferCapacity-maker.bufferStart)])
 			}
 
 			if (hashSum & maker.hashMask) == 0 {
@@ -258,7 +257,7 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 		bytes := maker.bufferSize - maker.minimumChunkSize
 		isEOC := false
 		maxSize := maker.maximumChunkSize - chunk.GetLength()
-        for i := 0; i < maker.bufferSize - maker.minimumChunkSize; i++ {
+		for i := 0; i < maker.bufferSize-maker.minimumChunkSize; i++ {
 			out := maker.bufferStart + i
 			if out >= maker.bufferCapacity {
 				out -= maker.bufferCapacity
@@ -269,7 +268,7 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
 			}
 
 			hashSum = maker.buzhashUpdate(hashSum, maker.buffer[out], maker.buffer[in], maker.minimumChunkSize)
-            if (hashSum & maker.hashMask) == 0 || i == maxSize - maker.minimumChunkSize - 1 {
+			if (hashSum&maker.hashMask) == 0 || i == maxSize-maker.minimumChunkSize-1 {
 				// A chunk is completed.
 				bytes = i + 1 + maker.minimumChunkSize
 				isEOC = true

src/duplicacy_chunkmaker_test.go

@@ -5,12 +5,12 @@
 package duplicacy
 
 import (
-    "testing"
 	"bytes"
 	crypto_rand "crypto/rand"
-    "math/rand"
 	"io"
+	"math/rand"
 	"sort"
+	"testing"
 )
 
 func splitIntoChunks(content []byte, n, averageChunkSize, maxChunkSize, minChunkSize,
@@ -29,14 +29,14 @@ func splitIntoChunks(content []byte, n, averageChunkSize, maxChunkSize, minChunk
 
 	maker := CreateChunkMaker(config, false)
 
-    var chunks [] string
+	var chunks []string
 	totalChunkSize := 0
 	totalFileSize := int64(0)
 
 	//LOG_INFO("CHUNK_SPLIT", "bufferCapacity: %d", bufferCapacity)
 
-    buffers := make([] *bytes.Buffer, n)
-    sizes := make([] int, n)
+	buffers := make([]*bytes.Buffer, n)
+	sizes := make([]int, n)
 	sizes[0] = 0
 	for i := 1; i < n; i++ {
 		same := true
@@ -54,20 +54,20 @@ func splitIntoChunks(content []byte, n, averageChunkSize, maxChunkSize, minChunk
 
 	sort.Sort(sort.IntSlice(sizes))
 
-    for i := 0; i < n - 1; i++ {
-        buffers[i] = bytes.NewBuffer(content[sizes[i] : sizes[i + 1]])
+	for i := 0; i < n-1; i++ {
+		buffers[i] = bytes.NewBuffer(content[sizes[i]:sizes[i+1]])
 	}
-    buffers[n - 1] = bytes.NewBuffer(content[sizes[n - 1]:])
+	buffers[n-1] = bytes.NewBuffer(content[sizes[n-1]:])
 
 	i := 0
 
 	maker.ForEachChunk(buffers[0],
-        func (chunk *Chunk, final bool) {
+		func(chunk *Chunk, final bool) {
 			//LOG_INFO("CHUNK_SPLIT", "i: %d, chunk: %s, size: %d", i, chunk.GetHash(), size)
 			chunks = append(chunks, chunk.GetHash())
 			totalChunkSize += chunk.GetLength()
 		},
-        func (size int64, hash string) (io.Reader, bool) {
+		func(size int64, hash string) (io.Reader, bool) {
 			totalFileSize += size
 			i++
 			if i >= len(buffers) {
@@ -76,7 +76,7 @@ func splitIntoChunks(content []byte, n, averageChunkSize, maxChunkSize, minChunk
 			return buffers[i], true
 		})
 
-    if (totalFileSize != int64(totalChunkSize)) {
+	if totalFileSize != int64(totalChunkSize) {
 		LOG_ERROR("CHUNK_SPLIT", "total chunk size: %d, total file size: %d", totalChunkSize, totalFileSize)
 	}
 	return chunks, totalChunkSize
@@ -84,9 +84,8 @@ func splitIntoChunks(content []byte, n, averageChunkSize, maxChunkSize, minChunk
 
 func TestChunkMaker(t *testing.T) {
 
-
 	//sizes := [...] int { 64 }
-    sizes := [...] int { 64, 256, 1024, 1024 * 10 }
+	sizes := [...]int{64, 256, 1024, 1024 * 10}
 
 	for _, size := range sizes {
 
@@ -99,15 +98,15 @@ func TestChunkMaker(t *testing.T) {
 
 		chunkArray1, totalSize1 := splitIntoChunks(content, 10, 32, 64, 16, 32)
 
-        capacities := [...]int { 32, 33, 34, 61, 62, 63, 64, 65, 66, 126, 127, 128, 129, 130,
+		capacities := [...]int{32, 33, 34, 61, 62, 63, 64, 65, 66, 126, 127, 128, 129, 130,
 			255, 256, 257, 511, 512, 513, 1023, 1024, 1025,
-                                 32, 48, 64, 128, 256, 512, 1024, 2048, }
+			32, 48, 64, 128, 256, 512, 1024, 2048}
 
 		//capacities := [...]int { 32 }
 
 		for _, capacity := range capacities {
 
-            for _, n := range [...]int { 6, 7, 8, 9, 10 } {
+			for _, n := range [...]int{6, 7, 8, 9, 10} {
 				chunkArray2, totalSize2 := splitIntoChunks(content, n, 32, 64, 16, capacity)
 
 				if totalSize1 != totalSize2 {

src/duplicacy_chunkuploader.go

@@ -11,7 +11,7 @@ import (
 
 // ChunkUploadTask represents a chunk to be uploaded.
 type ChunkUploadTask struct {
-    chunk * Chunk
+	chunk      *Chunk
 	chunkIndex int
 }
 
@@ -36,7 +36,7 @@ type ChunkUploader struct {
 // CreateChunkUploader creates a chunk uploader.
 func CreateChunkUploader(config *Config, storage Storage, snapshotCache *FileStorage, threads int,
 	completionFunc func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int)) *ChunkUploader {
-    uploader := &ChunkUploader {
+	uploader := &ChunkUploader{
 		config:         config,
 		storage:        storage,
 		snapshotCache:  snapshotCache,
@@ -56,20 +56,20 @@ func (uploader *ChunkUploader) Start() {
 			defer CatchLogException()
 			for {
 				select {
-                case task := <- uploader.taskQueue:
+				case task := <-uploader.taskQueue:
 					uploader.Upload(threadIndex, task)
-                case <- uploader.stopChannel:
+				case <-uploader.stopChannel:
 					return
 				}
 			}
-        } (i)
+		}(i)
 	}
 }
 
 // StartChunk sends a chunk to be uploaded to  a waiting uploading goroutine.  It may block if all uploading goroutines are busy.
 func (uploader *ChunkUploader) StartChunk(chunk *Chunk, chunkIndex int) {
 	atomic.AddInt32(&uploader.numberOfUploadingTasks, 1)
-    uploader.taskQueue <- ChunkUploadTask {
+	uploader.taskQueue <- ChunkUploadTask{
 		chunk:      chunk,
 		chunkIndex: chunkIndex,
 	}
@@ -134,13 +134,17 @@ func (uploader *ChunkUploader) Upload(threadIndex int, task ChunkUploadTask) boo
 		return false
 	}
 
+	if !uploader.config.dryRun {
 		err = uploader.storage.UploadFile(threadIndex, chunkPath, chunk.GetBytes())
 		if err != nil {
 			LOG_ERROR("UPLOAD_CHUNK", "Failed to upload the chunk %s: %v", chunkID, err)
 			return false
 		}
-
 		LOG_DEBUG("CHUNK_UPLOAD", "Chunk %s has been uploaded", chunkID)
+	} else {
+		LOG_DEBUG("CHUNK_UPLOAD", "Uploading was skipped for chunk %s", chunkID)
+	}
+
 	uploader.completionFunc(chunk, task.chunkIndex, false, chunkSize, chunk.GetLength())
 	atomic.AddInt32(&uploader.numberOfUploadingTasks, -1)
 	return true

src/duplicacy_chunkuploader_test.go

@@ -6,10 +6,10 @@ package duplicacy
 
 import (
 	"os"
-    "time"
 	"path"
-    "testing"
 	"runtime/debug"
+	"testing"
+	"time"
 
 	crypto_rand "crypto/rand"
 	"math/rand"
@@ -32,7 +32,7 @@ func TestUploaderAndDownloader(t *testing.T) {
 				debug.PrintStack()
 			}
 		}
-    } ()
+	}()
 
 	testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
 	os.RemoveAll(testDir)
@@ -48,7 +48,7 @@ func TestUploaderAndDownloader(t *testing.T) {
 	storage.EnableTestMode()
 	storage.SetRateLimits(testRateLimit, testRateLimit)
 
-    for _, dir := range []string { "chunks", "snapshots" }  {
+	for _, dir := range []string{"chunks", "snapshots"} {
 		err = storage.CreateDirectory(0, dir)
 		if err != nil {
 			t.Errorf("Failed to create directory %s: %v", dir, err)
@@ -56,7 +56,6 @@ func TestUploaderAndDownloader(t *testing.T) {
 		}
 	}
 
-
 	numberOfChunks := 100
 	maxChunkSize := 64 * 1024
 
@@ -68,11 +67,11 @@ func TestUploaderAndDownloader(t *testing.T) {
 
 	config := CreateConfig()
 	config.MinimumChunkSize = 100
-    config.chunkPool = make(chan *Chunk, numberOfChunks * 2)
+	config.chunkPool = make(chan *Chunk, numberOfChunks*2)
 	totalFileSize := 0
 
 	for i := 0; i < numberOfChunks; i++ {
-        content := make([]byte, rand.Int() % maxChunkSize + 1)
+		content := make([]byte, rand.Int()%maxChunkSize+1)
 		_, err = crypto_rand.Read(content)
 		if err != nil {
 			t.Errorf("Error generating random content: %v", err)
@@ -102,7 +101,6 @@ func TestUploaderAndDownloader(t *testing.T) {
 
 	chunkUploader.Stop()
 
-
 	chunkDownloader := CreateChunkDownloader(config, storage, nil, true, testThreads)
 	chunkDownloader.totalChunkSize = int64(totalFileSize)
 
@@ -120,7 +118,7 @@ func TestUploaderAndDownloader(t *testing.T) {
 	chunkDownloader.Stop()
 
 	for _, file := range listChunks(storage) {
-        err = storage.DeleteFile(0, "chunks/" + file)
+		err = storage.DeleteFile(0, "chunks/"+file)
 		if err != nil {
 			t.Errorf("Failed to delete the file %s: %v", file, err)
 			return

src/duplicacy_config.go

@@ -5,18 +5,19 @@
 package duplicacy
 
 import (
-    "encoding/json"
 	"bytes"
-    "os"
+	"crypto/hmac"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
 	"fmt"
 	"hash"
+	"os"
 	"runtime"
 	"runtime/debug"
 	"sync/atomic"
-    "crypto/rand"
-    "crypto/hmac"
-    "crypto/sha256"
-    "encoding/hex"
 
 	blake2 "github.com/minio/blake2b-simd"
 )
@@ -28,6 +29,15 @@ var DEFAULT_KEY = []byte("duplicacy")
 // standard zlib levels of -1 to 9.
 var DEFAULT_COMPRESSION_LEVEL = 100
 
+// The new header of the config file (to differentiate from the old format where the salt and iterations are fixed)
+var CONFIG_HEADER = "duplicacy\001"
+
+// The length of the salt used in the new format
+var CONFIG_SALT_LENGTH = 32
+
+// The default iterations for key derivation
+var CONFIG_DEFAULT_ITERATIONS = 16384
+
 type Config struct {
 	CompressionLevel int `json:"compression-level"`
 	AverageChunkSize int `json:"average-chunk-size"`
@@ -36,6 +46,8 @@ type Config struct {
 
 	ChunkSeed []byte `json:"chunk-seed"`
 
+	FixedNesting bool `json:"fixed-nesting"`
+
 	// Use HMAC-SHA256(hashKey, plaintext) as the chunk hash.
 	// Use HMAC-SHA256(idKey, chunk hash) as the file name of the chunk
 	// For chunks, use HMAC-SHA256(chunkKey, chunk hash) as the encryption key
@@ -53,11 +65,12 @@ type Config struct {
 	// for encrypting a non-chunk file
 	FileKey []byte `json:"-"`
 
-    chunkPool chan *Chunk                  `json:"-"`
+	chunkPool      chan *Chunk
 	numberOfChunks int32
+	dryRun         bool
 }
 
-    // Create an alias to avoid recursive calls on Config.MarshalJSON
+// Create an alias to avoid recursive calls on Config.MarshalJSON
 type aliasedConfig Config
 
 type jsonableConfig struct {
@@ -69,9 +82,9 @@ type jsonableConfig struct {
 	FileKey   string `json:"file-key"`
 }
 
-func (config *Config) MarshalJSON() ([] byte, error) {
+func (config *Config) MarshalJSON() ([]byte, error) {
 
-    return json.Marshal(&jsonableConfig {
+	return json.Marshal(&jsonableConfig{
 		aliasedConfig: (*aliasedConfig)(config),
 		ChunkSeed:     hex.EncodeToString(config.ChunkSeed),
 		HashKey:       hex.EncodeToString(config.HashKey),
@@ -83,7 +96,7 @@ func (config *Config) MarshalJSON() ([] byte, error) {
 
 func (config *Config) UnmarshalJSON(description []byte) (err error) {
 
-    aliased := &jsonableConfig {
+	aliased := &jsonableConfig{
 		aliasedConfig: (*aliasedConfig)(config),
 	}
 
@@ -130,18 +143,19 @@ func (config *Config) Print() {
 }
 
 func CreateConfigFromParameters(compressionLevel int, averageChunkSize int, maximumChunkSize int, mininumChunkSize int,
-                  isEncrypted bool, copyFrom *Config) (config *Config) {
+	isEncrypted bool, copyFrom *Config, bitCopy bool) (config *Config) {
 
-    config = &Config {
+	config = &Config{
 		CompressionLevel: compressionLevel,
 		AverageChunkSize: averageChunkSize,
 		MaximumChunkSize: maximumChunkSize,
 		MinimumChunkSize: mininumChunkSize,
+		FixedNesting:     true,
 	}
 
 	if isEncrypted {
 		// Randomly generate keys
-        keys := make([]byte, 32 * 5)
+		keys := make([]byte, 32*5)
 		_, err := rand.Read(keys)
 		if err != nil {
 			LOG_ERROR("CONFIG_KEY", "Failed to generate random keys: %v", err)
@@ -168,28 +182,34 @@ func CreateConfigFromParameters(compressionLevel int, averageChunkSize int, maxi
 
 		config.ChunkSeed = copyFrom.ChunkSeed
 		config.HashKey = copyFrom.HashKey
+
+		if bitCopy {
+			config.IDKey = copyFrom.IDKey
+			config.ChunkKey = copyFrom.ChunkKey
+			config.FileKey = copyFrom.FileKey
+		}
 	}
 
-    config.chunkPool = make(chan *Chunk, runtime.NumCPU() * 16)
+	config.chunkPool = make(chan *Chunk, runtime.NumCPU()*16)
 
 	return config
 }
 
 func CreateConfig() (config *Config) {
-    return &Config {
+	return &Config{
 		HashKey:          DEFAULT_KEY,
 		IDKey:            DEFAULT_KEY,
 		CompressionLevel: DEFAULT_COMPRESSION_LEVEL,
-        chunkPool: make(chan *Chunk, runtime.NumCPU() * 16),
+		chunkPool:        make(chan *Chunk, runtime.NumCPU()*16),
 	}
 }
 
 func (config *Config) GetChunk() (chunk *Chunk) {
 	select {
-    case chunk = <- config.chunkPool :
+	case chunk = <-config.chunkPool:
 	default:
 		numberOfChunks := atomic.AddInt32(&config.numberOfChunks, 1)
-        if numberOfChunks >= int32(runtime.NumCPU() * 16) {
+		if numberOfChunks >= int32(runtime.NumCPU()*16) {
 			LOG_WARN("CONFIG_CHUNK", "%d chunks have been allocated", numberOfChunks)
 			if _, found := os.LookupEnv("DUPLICACY_CHUNK_DEBUG"); found {
 				debug.PrintStack()
@@ -200,7 +220,7 @@ func (config *Config) GetChunk() (chunk *Chunk) {
 	return chunk
 }
 
-func (config *Config) PutChunk(chunk *Chunk){
+func (config *Config) PutChunk(chunk *Chunk) {
 
 	if chunk == nil {
 		return
@@ -215,7 +235,7 @@ func (config *Config) PutChunk(chunk *Chunk){
 
 func (config *Config) NewKeyedHasher(key []byte) hash.Hash {
 	if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
-        hasher, err := blake2.New(&blake2.Config{ Size: 32, Key:key })
+		hasher, err := blake2.New(&blake2.Config{Size: 32, Key: key})
 		if err != nil {
 			LOG_ERROR("HASH_KEY", "Invalid hash key: %x", key)
 		}
@@ -258,9 +278,9 @@ func (hasher *DummyHasher) BlockSize() int {
 
 func (config *Config) NewFileHasher() hash.Hash {
 	if SkipFileHash {
-        return &DummyHasher {}
+		return &DummyHasher{}
 	} else if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
-        hasher, _ := blake2.New(&blake2.Config{ Size: 32 })
+		hasher, _ := blake2.New(&blake2.Config{Size: 32})
 		return hasher
 	} else {
 		return sha256.New()
@@ -315,10 +335,45 @@ func DownloadConfig(storage Storage, password string) (config *Config, isEncrypt
 		return nil, false, err
 	}
 
+	if len(configFile.GetBytes()) < len(ENCRYPTION_HEADER) {
+		return nil, false, fmt.Errorf("The storage has an invalid config file")
+	}
+
+	if string(configFile.GetBytes()[:len(ENCRYPTION_HEADER)-1]) == ENCRYPTION_HEADER[:len(ENCRYPTION_HEADER)-1] && len(password) == 0 {
+		return nil, true, fmt.Errorf("The storage is likely to have been initialized with a password before")
+	}
+
 	var masterKey []byte
 
 	if len(password) > 0 {
-        masterKey = GenerateKeyFromPassword(password)
+
+		if string(configFile.GetBytes()[:len(ENCRYPTION_HEADER)]) == ENCRYPTION_HEADER {
+			// This is the old config format with a static salt and a fixed number of iterations
+			masterKey = GenerateKeyFromPassword(password, DEFAULT_KEY, CONFIG_DEFAULT_ITERATIONS)
+			LOG_TRACE("CONFIG_FORMAT", "Using a static salt and %d iterations for key derivation", CONFIG_DEFAULT_ITERATIONS)
+		} else if string(configFile.GetBytes()[:len(CONFIG_HEADER)]) == CONFIG_HEADER {
+			// This is the new config format with a random salt and a configurable number of iterations
+			encryptedLength := len(configFile.GetBytes()) - CONFIG_SALT_LENGTH - 4
+
+			// Extract the salt and the number of iterations
+			saltStart := configFile.GetBytes()[len(CONFIG_HEADER):]
+			iterations := binary.LittleEndian.Uint32(saltStart[CONFIG_SALT_LENGTH : CONFIG_SALT_LENGTH+4])
+			LOG_TRACE("CONFIG_ITERATIONS", "Using %d iterations for key derivation", iterations)
+			masterKey = GenerateKeyFromPassword(password, saltStart[:CONFIG_SALT_LENGTH], int(iterations))
+
+			// Copy to a temporary buffer to replace the header and remove the salt and the number of riterations
+			var encrypted bytes.Buffer
+			encrypted.Write([]byte(ENCRYPTION_HEADER))
+			encrypted.Write(saltStart[CONFIG_SALT_LENGTH+4:])
+
+			configFile.Reset(false)
+			configFile.Write(encrypted.Bytes())
+			if len(configFile.GetBytes()) != encryptedLength {
+				LOG_ERROR("CONFIG_DOWNLOAD", "Encrypted config has %d bytes instead of expected %d bytes", len(configFile.GetBytes()), encryptedLength)
+			}
+		} else {
+			return nil, true, fmt.Errorf("The config file has an invalid header")
+		}
 
 		// Decrypt the config file.  masterKey == nil means no encryption.
 		err = configFile.Decrypt(masterKey, "")
@@ -330,23 +385,21 @@ func DownloadConfig(storage Storage, password string) (config *Config, isEncrypt
 	config = CreateConfig()
 
 	err = json.Unmarshal(configFile.GetBytes(), config)
-
 	if err != nil {
-        if bytes.Equal(configFile.GetBytes()[:9], []byte("duplicacy")) {
-            return nil, true, fmt.Errorf("The storage is likely to have been initialized with a password before")
-        } else {
 		return nil, false, fmt.Errorf("Failed to parse the config file: %v", err)
 	}
-    }
+
+	storage.SetNestingLevels(config)
 
 	return config, false, nil
 
 }
 
-func UploadConfig(storage Storage, config *Config, password string) (bool) {
+func UploadConfig(storage Storage, config *Config, password string, iterations int) bool {
 
 	// This is the key to encrypt the config file.
 	var masterKey []byte
+	salt := make([]byte, CONFIG_SALT_LENGTH)
 
 	if len(password) > 0 {
 
@@ -355,7 +408,13 @@ func UploadConfig(storage Storage, config *Config, password string) (bool) {
 			return false
 		}
 
-        masterKey = GenerateKeyFromPassword(password)
+		_, err := rand.Read(salt)
+		if err != nil {
+			LOG_ERROR("CONFIG_KEY", "Failed to generate random salt: %v", err)
+			return false
+		}
+
+		masterKey = GenerateKeyFromPassword(password, salt, iterations)
 	}
 
 	description, err := json.MarshalIndent(config, "", "    ")
@@ -372,11 +431,26 @@ func UploadConfig(storage Storage, config *Config, password string) (bool) {
 	if len(password) > 0 {
 		// Encrypt the config file with masterKey.  If masterKey is nil then no encryption is performed.
 		err = chunk.Encrypt(masterKey, "")
-
 		if err != nil {
 			LOG_ERROR("CONFIG_CREATE", "Failed to create the config file: %v", err)
 			return false
 		}
+
+		// The new encrypted format for config is CONFIG_HEADER + salt + #iterations + encrypted content
+		encryptedLength := len(chunk.GetBytes()) + CONFIG_SALT_LENGTH + 4
+
+		// Copy to a temporary buffer to replace the header and add the salt and the number of iterations
+		var encrypted bytes.Buffer
+		encrypted.Write([]byte(CONFIG_HEADER))
+		encrypted.Write(salt)
+		binary.Write(&encrypted, binary.LittleEndian, uint32(iterations))
+		encrypted.Write(chunk.GetBytes()[len(ENCRYPTION_HEADER):])
+
+		chunk.Reset(false)
+		chunk.Write(encrypted.Bytes())
+		if len(chunk.GetBytes()) != encryptedLength {
+			LOG_ERROR("CONFIG_CREATE", "Encrypted config has %d bytes instead of expected %d bytes", len(chunk.GetBytes()), encryptedLength)
+		}
 	}
 
 	err = storage.UploadFile(0, "config", chunk.GetBytes())
@@ -389,7 +463,7 @@ func UploadConfig(storage Storage, config *Config, password string) (bool) {
 		config.Print()
 	}
 
-    for _, subDir := range []string {"chunks", "snapshots"} {
+	for _, subDir := range []string{"chunks", "snapshots"} {
 		err = storage.CreateDirectory(0, subDir)
 		if err != nil {
 			LOG_ERROR("CONFIG_MKDIR", "Failed to create storage subdirectory: %v", err)
@@ -402,8 +476,8 @@ func UploadConfig(storage Storage, config *Config, password string) (bool) {
 // ConfigStorage makes the general storage space available for storing duplicacy format snapshots.  In essence,
 // it simply creates a file named 'config' that stores various parameters as well as a set of keys if encryption
 // is enabled.
-func ConfigStorage(storage Storage, compressionLevel int, averageChunkSize int, maximumChunkSize int,
-                   minimumChunkSize int, password string, copyFrom *Config) bool {
+func ConfigStorage(storage Storage, iterations int, compressionLevel int, averageChunkSize int, maximumChunkSize int,
+	minimumChunkSize int, password string, copyFrom *Config, bitCopy bool) bool {
 
 	exist, _, _, err := storage.GetFileInfo(0, "config")
 	if err != nil {
@@ -416,12 +490,11 @@ func ConfigStorage(storage Storage, compressionLevel int, averageChunkSize int,
 		return false
 	}
 
-
 	config := CreateConfigFromParameters(compressionLevel, averageChunkSize, maximumChunkSize, minimumChunkSize, len(password) > 0,
-                           copyFrom)
+		copyFrom, bitCopy)
 	if config == nil {
 		return false
 	}
 
-    return UploadConfig(storage, config, password)
+	return UploadConfig(storage, config, password, iterations)
 }

src/duplicacy_dropboxstorage.go

@@ -6,20 +6,21 @@ package duplicacy
 
 import (
 	"fmt"
-    "path"
 	"strings"
+
 	"github.com/gilbertchen/go-dropbox"
 )
 
 type DropboxStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	clients        []*dropbox.Files
+	minimumNesting int // The minimum level of directories to dive into before searching for the chunk file.
 	storageDir     string
 }
 
 // CreateDropboxStorage creates a dropbox storage object.
-func CreateDropboxStorage(accessToken string, storageDir string, threads int) (storage *DropboxStorage, err error) {
+func CreateDropboxStorage(accessToken string, storageDir string, minimumNesting int, threads int) (storage *DropboxStorage, err error) {
 
 	var clients []*dropbox.Files
 	for i := 0; i < threads; i++ {
@@ -31,13 +32,14 @@ func CreateDropboxStorage(accessToken string, storageDir string, threads int) (s
 		storageDir = "/" + storageDir
 	}
 
-    if len(storageDir) > 1 && storageDir[len(storageDir) - 1] == '/' {
-        storageDir = storageDir[:len(storageDir) - 1]
+	if len(storageDir) > 1 && storageDir[len(storageDir)-1] == '/' {
+		storageDir = storageDir[:len(storageDir)-1]
 	}
 
-    storage = &DropboxStorage {
+	storage = &DropboxStorage{
 		clients:        clients,
 		storageDir:     storageDir,
+		minimumNesting: minimumNesting,
 	}
 
 	err = storage.CreateDirectory(0, "")
@@ -45,6 +47,8 @@ func CreateDropboxStorage(accessToken string, storageDir string, threads int) (s
 		return nil, fmt.Errorf("Can't create storage directory: %v", err)
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{1}, 1)
 	return storage, nil
 }
 
@@ -55,13 +59,13 @@ func (storage *DropboxStorage) ListFiles(threadIndex int, dir string) (files []s
 		dir = "/" + dir
 	}
 
-    if len(dir) > 1 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	if len(dir) > 1 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
-    input := &dropbox.ListFolderInput {
-        Path : storage.storageDir + dir,
-        Recursive : false,
+	input := &dropbox.ListFolderInput{
+		Path:             storage.storageDir + dir,
+		Recursive:        false,
 		IncludeMediaInfo: false,
 		IncludeDeleted:   false,
 	}
@@ -85,7 +89,7 @@ func (storage *DropboxStorage) ListFiles(threadIndex int, dir string) (files []s
 
 		if output.HasMore {
 			output, err = storage.clients[threadIndex].ListFolderContinue(
-                              &dropbox.ListFolderContinueInput { Cursor: output.Cursor, })
+				&dropbox.ListFolderContinueInput{Cursor: output.Cursor})
 
 		} else {
 			break
@@ -102,7 +106,7 @@ func (storage *DropboxStorage) DeleteFile(threadIndex int, filePath string) (err
 		filePath = "/" + filePath
 	}
 
-    input := &dropbox.DeleteInput {
+	input := &dropbox.DeleteInput{
 		Path: storage.storageDir + filePath,
 	}
 	_, err = storage.clients[threadIndex].Delete(input)
@@ -123,7 +127,7 @@ func (storage *DropboxStorage) MoveFile(threadIndex int, from string, to string)
 	if to != "" && to[0] != '/' {
 		to = "/" + to
 	}
-    input := &dropbox.MoveInput {
+	input := &dropbox.MoveInput{
 		FromPath: storage.storageDir + from,
 		ToPath:   storage.storageDir + to,
 	}
@@ -137,12 +141,12 @@ func (storage *DropboxStorage) CreateDirectory(threadIndex int, dir string) (err
 		dir = "/" + dir
 	}
 
-    if len(dir) > 1 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	if len(dir) > 1 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
-    input := &dropbox.CreateFolderInput {
-        Path : storage.storageDir + dir,
+	input := &dropbox.CreateFolderInput{
+		Path: storage.storageDir + dir,
 	}
 
 	_, err = storage.clients[threadIndex].CreateFolder(input)
@@ -161,7 +165,7 @@ func (storage *DropboxStorage) GetFileInfo(threadIndex int, filePath string) (ex
 		filePath = "/" + filePath
 	}
 
-    input := &dropbox.GetMetadataInput {
+	input := &dropbox.GetMetadataInput{
 		Path:             storage.storageDir + filePath,
 		IncludeMediaInfo: false,
 	}
@@ -178,66 +182,6 @@ func (storage *DropboxStorage) GetFileInfo(threadIndex int, filePath string) (ex
 	return true, output.Tag == "folder", int64(output.Size), nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *DropboxStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-     dir := "/chunks"
-
-    suffix := ""
-    if isFossil {
-        suffix = ".fsl"
-    }
-
-    // The minimum level of directories to dive into before searching for the chunk file.
-    minimumLevel := 1
-
-    for level := 0; level * 2 < len(chunkID); level ++ {
-        if level >= minimumLevel {
-            filePath = path.Join(dir, chunkID[2 * level:]) + suffix
-            var size int64
-            exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-            if err != nil {
-                return "", false, 0, err
-            }
-            if exist {
-                return filePath, exist, size, nil
-            }
-        }
-
-        // Find the subdirectory the chunk file may reside.
-        subDir := path.Join(dir, chunkID[2 * level: 2 * level + 2])
-        exist, _, _, err = storage.GetFileInfo(threadIndex, subDir)
-        if err != nil {
-            return "", false, 0, err
-        }
-
-        if exist {
-            dir = subDir
-            continue
-        }
-
-        if level < minimumLevel {
-            // Create the subdirectory if it doesn't exist.
-            err = storage.CreateDirectory(threadIndex, subDir)
-            if err != nil {
-                return "", false, 0, err
-            }
-
-            dir = subDir
-            continue
-        }
-
-        // Teh chunk must be under this subdirectory but it doesn't exist.
-        return path.Join(dir, chunkID[2 * level:])[1:] + suffix, false, 0, nil
-
-    }
-
-    LOG_FATAL("CHUNK_FIND", "Chunk %s is still not found after having searched a maximum level of directories",
-              chunkID)
-    return "", false, 0, nil
-
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *DropboxStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
@@ -245,7 +189,7 @@ func (storage *DropboxStorage) DownloadFile(threadIndex int, filePath string, ch
 		filePath = "/" + filePath
 	}
 
-    input := &dropbox.DownloadInput {
+	input := &dropbox.DownloadInput{
 		Path: storage.storageDir + filePath,
 	}
 
@@ -256,7 +200,7 @@ func (storage *DropboxStorage) DownloadFile(threadIndex int, filePath string, ch
 
 	defer output.Body.Close()
 
-    _, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit / len(storage.clients))
+	_, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit/len(storage.clients))
 	return err
 
 }
@@ -267,12 +211,12 @@ func (storage *DropboxStorage) UploadFile(threadIndex int, filePath string, cont
 		filePath = "/" + filePath
 	}
 
-    input := &dropbox.UploadInput {
+	input := &dropbox.UploadInput{
 		Path:       storage.storageDir + filePath,
 		Mode:       dropbox.WriteModeOverwrite,
 		AutoRename: false,
 		Mute:       true,
-        Reader: CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.clients)),
+		Reader:     CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.clients)),
 	}
 
 	_, err = storage.clients[threadIndex].Upload(input)
@@ -281,16 +225,16 @@ func (storage *DropboxStorage) UploadFile(threadIndex int, filePath string, cont
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *DropboxStorage) IsCacheNeeded() (bool) { return true }
+func (storage *DropboxStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *DropboxStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *DropboxStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *DropboxStorage) IsStrongConsistent() (bool) { return false }
+func (storage *DropboxStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *DropboxStorage) IsFastListing() (bool) { return false }
+func (storage *DropboxStorage) IsFastListing() bool { return false }
 
 // Enable the test mode.
 func (storage *DropboxStorage) EnableTestMode() {}

src/duplicacy_entry.go

@@ -4,22 +4,20 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "path/filepath"
-    "io/ioutil"
-    "sort"
-    "regexp"
-    "strconv"
-    "time"
-    "encoding/json"
 	"encoding/base64"
-    "strings"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"regexp"
 	"runtime"
-
+	"sort"
+	"strconv"
+	"strings"
+	"time"
 )
 
-
 // This is the hidden directory in the repository for storing various files.
 var DUPLICACY_DIRECTORY = ".duplicacy"
 var DUPLICACY_FILE = ".duplicacy"
@@ -50,18 +48,18 @@ type Entry struct {
 // CreateEntry creates an entry from file properties.
 func CreateEntry(path string, size int64, time int64, mode uint32) *Entry {
 
-    if len(path) > 0 && path[len(path) - 1] != '/' && (mode & uint32(os.ModeDir)) != 0 {
+	if len(path) > 0 && path[len(path)-1] != '/' && (mode&uint32(os.ModeDir)) != 0 {
 		path += "/"
 	}
 
-    return &Entry {
-        Path : path,
-        Size : size,
-        Time : time,
-        Mode : mode,
+	return &Entry{
+		Path: path,
+		Size: size,
+		Time: time,
+		Mode: mode,
 
-        UID  : -1,
-        GID  : -1,
+		UID: -1,
+		GID: -1,
 	}
 
 }
@@ -72,15 +70,15 @@ func CreateEntryFromFileInfo(fileInfo os.FileInfo, directory string) *Entry {
 
 	mode := fileInfo.Mode()
 
-    if mode & os.ModeDir != 0 && mode & os.ModeSymlink != 0 {
+	if mode&os.ModeDir != 0 && mode&os.ModeSymlink != 0 {
 		mode ^= os.ModeDir
 	}
 
-    if path[len(path) - 1] != '/' && mode & os.ModeDir != 0 {
+	if path[len(path)-1] != '/' && mode&os.ModeDir != 0 {
 		path += "/"
 	}
 
-    entry := &Entry {
+	entry := &Entry{
 		Path: path,
 		Size: fileInfo.Size(),
 		Time: fileInfo.ModTime().Unix(),
@@ -95,14 +93,14 @@ func CreateEntryFromFileInfo(fileInfo os.FileInfo, directory string) *Entry {
 // CreateEntryFromJSON creates an entry from a json description.
 func (entry *Entry) UnmarshalJSON(description []byte) (err error) {
 
-    var object map[string]interface {}
+	var object map[string]interface{}
 
 	err = json.Unmarshal(description, &object)
 	if err != nil {
 		return err
 	}
 
-    var value interface {}
+	var value interface{}
 	var ok bool
 
 	if value, ok = object["name"]; ok {
@@ -171,7 +169,7 @@ func (entry *Entry) UnmarshalJSON(description []byte) (err error) {
 	}
 
 	if value, ok = object["attributes"]; ok {
-        if attributes, ok := value.(map[string]interface {}); !ok {
+		if attributes, ok := value.(map[string]interface{}); !ok {
 			return fmt.Errorf("Attributes are invalid for file '%s' in the snapshot", entry.Path)
 		} else {
 			entry.Attributes = make(map[string][]byte)
@@ -251,7 +249,7 @@ func (entry *Entry) convertToObject(encodeName bool) map[string]interface{} {
 }
 
 // MarshalJSON returns the json description of an entry.
-func (entry *Entry) MarshalJSON() ([] byte, error) {
+func (entry *Entry) MarshalJSON() ([]byte, error) {
 
 	object := entry.convertToObject(true)
 	description, err := json.Marshal(object)
@@ -259,15 +257,15 @@ func (entry *Entry) MarshalJSON() ([] byte, error) {
 }
 
 func (entry *Entry) IsFile() bool {
-    return entry.Mode & uint32(os.ModeType) == 0
+	return entry.Mode&uint32(os.ModeType) == 0
 }
 
 func (entry *Entry) IsDir() bool {
-    return entry.Mode & uint32(os.ModeDir) != 0
+	return entry.Mode&uint32(os.ModeDir) != 0
 }
 
 func (entry *Entry) IsLink() bool {
-    return entry.Mode & uint32(os.ModeSymlink) != 0
+	return entry.Mode&uint32(os.ModeSymlink) != 0
 }
 
 func (entry *Entry) GetPermissions() os.FileMode {
@@ -275,12 +273,12 @@ func (entry *Entry) GetPermissions() os.FileMode {
 }
 
 func (entry *Entry) IsSameAs(other *Entry) bool {
-    return entry.Size == other.Size && entry.Time <= other.Time + 1 && entry.Time >= other.Time - 1
+	return entry.Size == other.Size && entry.Time <= other.Time+1 && entry.Time >= other.Time-1
 }
 
 func (entry *Entry) IsSameAsFileInfo(other os.FileInfo) bool {
 	time := other.ModTime().Unix()
-    return entry.Size == other.Size() && entry.Time <= time + 1 && entry.Time >= time - 1
+	return entry.Size == other.Size() && entry.Time <= time+1 && entry.Time >= time-1
 }
 
 func (entry *Entry) String(maxSizeDigits int) string {
@@ -288,7 +286,7 @@ func (entry *Entry) String(maxSizeDigits int) string {
 	return fmt.Sprintf("%*d %s %64s %s", maxSizeDigits, entry.Size, modifiedTime, entry.Hash, entry.Path)
 }
 
-func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo) bool {
+func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo, setOwner bool) bool {
 
 	if fileInfo == nil {
 		stat, err := os.Stat(fullPath)
@@ -299,7 +297,7 @@ func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo) bool
 		}
 	}
 
-    if (*fileInfo).Mode() & os.ModePerm != entry.GetPermissions() {
+	if (*fileInfo).Mode()&os.ModePerm != entry.GetPermissions() {
 		err := os.Chmod(fullPath, entry.GetPermissions())
 		if err != nil {
 			LOG_ERROR("RESTORE_CHMOD", "Failed to set the file permissions: %v", err)
@@ -320,10 +318,13 @@ func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo) bool
 		entry.SetAttributesToFile(fullPath)
 	}
 
+	if setOwner {
 		return SetOwner(fullPath, entry, fileInfo)
+	} else {
+		return true
+	}
 }
 
-
 // Return -1 if 'left' should appear before 'right', 1 if opposite, and 0 if they are the same.
 // Files are always arranged before subdirectories under the same parent directory.
 func (left *Entry) Compare(right *Entry) int {
@@ -417,14 +418,14 @@ func (files FileInfoCompare) Less(i, j int) bool {
 	left := files[i]
 	right := files[j]
 
-    if left.IsDir() && left.Mode() & os.ModeSymlink == 0 {
-        if right.IsDir() && right.Mode() & os.ModeSymlink == 0 {
+	if left.IsDir() && left.Mode()&os.ModeSymlink == 0 {
+		if right.IsDir() && right.Mode()&os.ModeSymlink == 0 {
 			return left.Name() < right.Name()
 		} else {
 			return false
 		}
 	} else {
-        if right.IsDir() && right.Mode() & os.ModeSymlink == 0 {
+		if right.IsDir() && right.Mode()&os.ModeSymlink == 0 {
 			return true
 		} else {
 			return left.Name() < right.Name()
@@ -434,8 +435,8 @@ func (files FileInfoCompare) Less(i, j int) bool {
 
 // ListEntries returns a list of entries representing file and subdirectories under the directory 'path'.  Entry paths
 // are normalized as relative to 'top'.  'patterns' are used to exclude or include certain files.
-func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string, discardAttributes bool) (directoryList []*Entry,
-                                                                                   skippedFiles [] string, err error) {
+func ListEntries(top string, path string, fileList *[]*Entry, patterns []string, discardAttributes bool) (directoryList []*Entry,
+	skippedFiles []string, err error) {
 
 	LOG_DEBUG("LIST_ENTRIES", "Listing %s", path)
 
@@ -449,12 +450,12 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string
 	}
 
 	normalizedPath := path
-    if len(normalizedPath) > 0 && normalizedPath[len(normalizedPath) - 1] != '/' {
+	if len(normalizedPath) > 0 && normalizedPath[len(normalizedPath)-1] != '/' {
 		normalizedPath += "/"
 	}
 
 	normalizedTop := top
-    if normalizedTop != "" && normalizedTop[len(normalizedTop) - 1] != '/' {
+	if normalizedTop != "" && normalizedTop[len(normalizedTop)-1] != '/' {
 		normalizedTop += "/"
 	}
 
@@ -475,7 +476,7 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string
 			isRegular := false
 			isRegular, entry.Link, err = Readlink(filepath.Join(top, entry.Path))
 			if err != nil {
-                LOG_WARN("LIST_LINK", "Failed to read the symlink %s: %v", entry.Path, err )
+				LOG_WARN("LIST_LINK", "Failed to read the symlink %s: %v", entry.Path, err)
 				skippedFiles = append(skippedFiles, entry.Path)
 				continue
 			}
@@ -485,7 +486,7 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string
 			} else if path == "" && filepath.IsAbs(entry.Link) && !strings.HasPrefix(entry.Link, normalizedTop) {
 				stat, err := os.Stat(filepath.Join(top, entry.Path))
 				if err != nil {
-                    LOG_WARN("LIST_LINK", "Failed to read the symlink: %v", err )
+					LOG_WARN("LIST_LINK", "Failed to read the symlink: %v", err)
 					skippedFiles = append(skippedFiles, entry.Path)
 					continue
 				}
@@ -504,7 +505,7 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string
 			entry.ReadAttributes(top)
 		}
 
-        if f.Mode() & (os.ModeNamedPipe | os.ModeSocket | os.ModeDevice) != 0 {
+		if f.Mode()&(os.ModeNamedPipe|os.ModeSocket|os.ModeDevice) != 0 {
 			LOG_WARN("LIST_SKIP", "Skipped non-regular file %s", entry.Path)
 			skippedFiles = append(skippedFiles, entry.Path)
 			continue
@@ -526,7 +527,7 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string
 		}
 	}
 
-    for i, j := 0, len(directoryList) - 1; i < j; i, j = i + 1, j - 1 {
+	for i, j := 0, len(directoryList)-1; i < j; i, j = i+1, j-1 {
 		directoryList[i], directoryList[j] = directoryList[j], directoryList[i]
 	}
 
@@ -534,8 +535,8 @@ func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string
 }
 
 // Diff returns how many bytes remain unmodifiled between two files.
-func (entry *Entry) Diff(chunkHashes[]string, chunkLengths[]int,
-                         otherHashes[]string, otherLengths [] int) (modifiedLength int64) {
+func (entry *Entry) Diff(chunkHashes []string, chunkLengths []int,
+	otherHashes []string, otherLengths []int) (modifiedLength int64) {
 
 	var offset1, offset2 int64
 	i1 := entry.StartChunk
@@ -559,7 +560,7 @@ func (entry *Entry) Diff(chunkHashes[]string, chunkLengths[]int,
 			offset2 += int64(otherLengths[i2])
 			i2++
 		} else {
-            if chunkHashes[i1] == otherHashes[i2] && end - start == otherLengths[i2] {
+			if chunkHashes[i1] == otherHashes[i2] && end-start == otherLengths[i2] {
 			} else {
 				modifiedLength += int64(chunkLengths[i1])
 			}

src/duplicacy_entry_test.go

@@ -5,17 +5,17 @@
 package duplicacy
 
 import (
-    "testing"
 	"io/ioutil"
+	"math/rand"
 	"os"
 	"path/filepath"
-    "math/rand"
 	"sort"
+	"testing"
 )
 
 func TestEntrySort(t *testing.T) {
 
-    DATA := [...]string {
+	DATA := [...]string{
 		"ab",
 		"ab-",
 		"ab0",
@@ -44,15 +44,15 @@ func TestEntrySort(t *testing.T) {
 	var entry1, entry2 *Entry
 
 	for i, p1 := range DATA {
-        if p1[len(p1) - 1] == '/' {
-            entry1 = CreateEntry(p1, 0, 0, 0700 | uint32(os.ModeDir))
+		if p1[len(p1)-1] == '/' {
+			entry1 = CreateEntry(p1, 0, 0, 0700|uint32(os.ModeDir))
 		} else {
 			entry1 = CreateEntry(p1, 0, 0, 0700)
 		}
 		for j, p2 := range DATA {
 
-            if p2[len(p2) - 1] == '/' {
-                entry2 = CreateEntry(p2, 0, 0, 0700 | uint32(os.ModeDir))
+			if p2[len(p2)-1] == '/' {
+				entry2 = CreateEntry(p2, 0, 0, 0700|uint32(os.ModeDir))
 			} else {
 				entry2 = CreateEntry(p2, 0, 0, 0700)
 			}
@@ -88,7 +88,7 @@ func TestEntryList(t *testing.T) {
 	os.RemoveAll(testDir)
 	os.MkdirAll(testDir, 0700)
 
-    DATA := [...]string {
+	DATA := [...]string{
 		"ab",
 		"ab-",
 		"ab0",
@@ -107,19 +107,18 @@ func TestEntryList(t *testing.T) {
 		"ab3/c",
 	}
 
-
 	var entry1, entry2 *Entry
 
 	for i, p1 := range DATA {
-        if p1[len(p1) - 1] == '/' {
-            entry1 = CreateEntry(p1, 0, 0, 0700 | uint32(os.ModeDir))
+		if p1[len(p1)-1] == '/' {
+			entry1 = CreateEntry(p1, 0, 0, 0700|uint32(os.ModeDir))
 		} else {
 			entry1 = CreateEntry(p1, 0, 0, 0700)
 		}
 		for j, p2 := range DATA {
 
-            if p2[len(p2) - 1] == '/' {
-                entry2 = CreateEntry(p2, 0, 0, 0700 | uint32(os.ModeDir))
+			if p2[len(p2)-1] == '/' {
+				entry2 = CreateEntry(p2, 0, 0, 0700|uint32(os.ModeDir))
 			} else {
 				entry2 = CreateEntry(p2, 0, 0, 0700)
 			}
@@ -151,7 +150,7 @@ func TestEntryList(t *testing.T) {
 	for _, file := range DATA {
 
 		fullPath := filepath.Join(testDir, file)
-        if file[len(file) - 1] == '/' {
+		if file[len(file)-1] == '/' {
 			err := os.Mkdir(fullPath, 0700)
 			if err != nil {
 				t.Errorf("Mkdir(%s) returned an error: %s", fullPath, err)
@@ -171,8 +170,8 @@ func TestEntryList(t *testing.T) {
 	entries := make([]*Entry, 0, 4)
 
 	for len(directories) > 0 {
-        directory := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		directory := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 		entries = append(entries, directory)
 		subdirectories, _, err := ListEntries(testDir, directory.Path, &entries, nil, false)
 		if err != nil {
@@ -217,4 +216,3 @@ func TestEntryList(t *testing.T) {
 	}
 
 }
-

src/duplicacy_filereader.go

@@ -11,19 +11,19 @@ import (
 // FileReader wraps a number of files and turns them into a series of readers.
 type FileReader struct {
 	top   string
-    files [] *Entry
+	files []*Entry
 
 	CurrentFile  *os.File
 	CurrentIndex int
 	CurrentEntry *Entry
 
-    SkippedFiles [] string
+	SkippedFiles []string
 }
 
 // CreateFileReader creates a file reader.
-func CreateFileReader(top string, files[] *Entry) (*FileReader) {
+func CreateFileReader(top string, files []*Entry) *FileReader {
 
-    reader := &FileReader {
+	reader := &FileReader{
 		top:          top,
 		files:        files,
 		CurrentIndex: -1,
@@ -35,7 +35,7 @@ func CreateFileReader(top string, files[] *Entry) (*FileReader) {
 }
 
 // NextFile switchs to the next file in the file reader.
-func (reader *FileReader) NextFile() bool{
+func (reader *FileReader) NextFile() bool {
 
 	if reader.CurrentFile != nil {
 		reader.CurrentFile.Close()
@@ -68,7 +68,3 @@ func (reader *FileReader) NextFile() bool{
 	reader.CurrentFile = nil
 	return false
 }
-
-
-
-

src/duplicacy_filestorage.go

@@ -5,27 +5,27 @@
 package duplicacy
 
 import (
-    "os"
 	"fmt"
-    "path"
 	"io"
 	"io/ioutil"
-    "time"
 	"math/rand"
+	"os"
+	"path"
+	"strings"
+	"time"
 )
 
 // FileStorage is a local on-disk file storage implementing the Storage interface.
 type FileStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    minimumLevel int       // The minimum level of directories to dive into before searching for the chunk file.
 	isCacheNeeded   bool // Network storages require caching
 	storageDir      string
 	numberOfThreads int
 }
 
 // CreateFileStorage creates a file storage.
-func CreateFileStorage(storageDir string, minimumLevel int, isCacheNeeded bool, threads int) (storage *FileStorage, err error) {
+func CreateFileStorage(storageDir string, isCacheNeeded bool, threads int) (storage *FileStorage, err error) {
 
 	var stat os.FileInfo
 
@@ -45,13 +45,12 @@ func CreateFileStorage(storageDir string, minimumLevel int, isCacheNeeded bool,
 		}
 	}
 
-    for storageDir[len(storageDir) - 1] ==  '/' {
-        storageDir = storageDir[:len(storageDir) - 1]
+	for storageDir[len(storageDir)-1] == '/' {
+		storageDir = storageDir[:len(storageDir)-1]
 	}
 
-    storage = &FileStorage {
-        storageDir : storageDir,
-        minimumLevel: minimumLevel,
+	storage = &FileStorage{
+		storageDir:      storageDir,
 		isCacheNeeded:   isCacheNeeded,
 		numberOfThreads: threads,
 	}
@@ -59,6 +58,8 @@ func CreateFileStorage(storageDir string, minimumLevel int, isCacheNeeded bool,
 	// Random number fo generating the temporary chunk file suffix.
 	rand.Seed(time.Now().UnixNano())
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{2, 3}, 2)
 	return storage, nil
 }
 
@@ -77,7 +78,7 @@ func (storage *FileStorage) ListFiles(threadIndex int, dir string) (files []stri
 
 	for _, f := range list {
 		name := f.Name()
-        if f.IsDir() && name[len(name) - 1] != '/' {
+		if f.IsDir() && name[len(name)-1] != '/' {
 			name += "/"
 		}
 		files = append(files, name)
@@ -126,67 +127,6 @@ func (storage *FileStorage) GetFileInfo(threadIndex int, filePath string) (exist
 	return true, stat.IsDir(), stat.Size(), nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with the
-// suffix '.fsl'.
-func (storage *FileStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-     dir := path.Join(storage.storageDir, "chunks")
-
-    suffix := ""
-    if isFossil {
-        suffix = ".fsl"
-    }
-
-    for level := 0; level * 2 < len(chunkID); level ++ {
-        if level >= storage.minimumLevel {
-            filePath = path.Join(dir, chunkID[2 * level:]) + suffix
-            // Use Lstat() instead of Stat() since 1) Stat() doesn't work for deduplicated disks on Windows and 2) there isn't
-            // really a need to follow the link if filePath is a link.
-            stat, err := os.Lstat(filePath)
-            if err != nil {
-                LOG_DEBUG("FS_FIND", "File %s can't be found: %v", filePath, err)
-            } else if stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], false, 0, fmt.Errorf("The path %s is a directory", filePath)
-            } else {
-                return filePath[len(storage.storageDir) + 1:], true, stat.Size(), nil
-            }
-        }
-
-        // Find the subdirectory the chunk file may reside.
-        subDir := path.Join(dir, chunkID[2 * level: 2 * level + 2])
-        stat, err := os.Stat(subDir)
-        if err == nil && stat.IsDir() {
-            dir = subDir
-            continue
-        }
-
-        if level < storage.minimumLevel {
-            // Create the subdirectory if it doesn't exist.
-
-            if err == nil && !stat.IsDir() {
-                return "", false, 0, fmt.Errorf("The path %s is not a directory", subDir)
-            }
-
-            err = os.Mkdir(subDir, 0744)
-            if err != nil {
-                // The directory may have been created by other threads so check it again.
-                stat, _ := os.Stat(subDir)
-                if stat == nil || !stat.IsDir() {
-                    return "", false, 0, err
-                }
-            }
-            dir = subDir
-            continue
-        }
-
-        // The chunk must be under this subdirectory but it doesn't exist.
-        return path.Join(dir, chunkID[2 * level:])[len(storage.storageDir) + 1:] + suffix, false, 0, nil
-
-    }
-
-    return "", false, 0, fmt.Errorf("The maximum level of directories searched")
-
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *FileStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
@@ -197,7 +137,7 @@ func (storage *FileStorage) DownloadFile(threadIndex int, filePath string, chunk
 	}
 
 	defer file.Close()
-    if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit / storage.numberOfThreads); err != nil {
+	if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit/storage.numberOfThreads); err != nil {
 		return err
 	}
 
@@ -210,6 +150,26 @@ func (storage *FileStorage) UploadFile(threadIndex int, filePath string, content
 
 	fullPath := path.Join(storage.storageDir, filePath)
 
+	if len(strings.Split(filePath, "/")) > 2 {
+		dir := path.Dir(fullPath)
+		// Use Lstat() instead of Stat() since 1) Stat() doesn't work for deduplicated disks on Windows and 2) there isn't
+		// really a need to follow the link if filePath is a link.
+		stat, err := os.Lstat(dir)
+		if err != nil {
+			if !os.IsNotExist(err) {
+				return err
+			}
+			err = os.MkdirAll(dir, 0744)
+			if err != nil {
+				return err
+			}
+		} else {
+			if !stat.IsDir() {
+				fmt.Errorf("The path %s is not a directory", dir)
+			}
+		}
+	}
+
 	letters := "abcdefghijklmnopqrstuvwxyz"
 	suffix := make([]byte, 8)
 	for i := range suffix {
@@ -218,12 +178,12 @@ func (storage *FileStorage) UploadFile(threadIndex int, filePath string, content
 
 	temporaryFile := fullPath + "." + string(suffix) + ".tmp"
 
-    file, err := os.OpenFile(temporaryFile, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0644)
+	file, err := os.OpenFile(temporaryFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
 	if err != nil {
 		return err
 	}
 
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
 	_, err = io.Copy(file, reader)
 	if err != nil {
 		file.Close()
@@ -248,16 +208,16 @@ func (storage *FileStorage) UploadFile(threadIndex int, filePath string, content
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *FileStorage) IsCacheNeeded () (bool) { return storage.isCacheNeeded }
+func (storage *FileStorage) IsCacheNeeded() bool { return storage.isCacheNeeded }
 
 // If the 'MoveFile' method is implemented.
-func (storage *FileStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *FileStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *FileStorage) IsStrongConsistent() (bool) { return true }
+func (storage *FileStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *FileStorage) IsFastListing() (bool) { return false }
+func (storage *FileStorage) IsFastListing() bool { return false }
 
 // Enable the test mode.
 func (storage *FileStorage) EnableTestMode() {}

src/duplicacy_gcdstorage.go

@@ -5,18 +5,18 @@
 package duplicacy
 
 import (
-    "io"
+	"encoding/json"
 	"fmt"
-    "net"
-    "path"
-    "time"
-    "sync"
-    "strings"
-    "net/http"
-    "net/url"
+	"io"
 	"io/ioutil"
 	"math/rand"
-    "encoding/json"
+	"net"
+	"net/http"
+	"net/url"
+	"path"
+	"strings"
+	"sync"
+	"time"
 
 	"golang.org/x/net/context"
 	"golang.org/x/oauth2"
@@ -24,18 +24,24 @@ import (
 	"google.golang.org/api/googleapi"
 )
 
+var (
+	GCDFileMimeType = "application/octet-stream"
+	GCDDirectoryMimeType = "application/vnd.google-apps.folder"
+)
+
 type GCDStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	service     *drive.Service
-    idCache map[string]string
-    idCacheLock *sync.Mutex
-    backoffs []int
+	idCache     map[string]string  // only directories are saved in this cache
+	idCacheLock sync.Mutex
+	backoffs    []int // desired backoff time in seconds for each thread
+	attempts    []int // number of failed attempts since last success for each thread
 
+	createDirectoryLock sync.Mutex
 	isConnected         bool
 	numberOfThreads     int
 	TestMode            bool
-
 }
 
 type GCDConfig struct {
@@ -47,10 +53,17 @@ type GCDConfig struct {
 
 func (storage *GCDStorage) shouldRetry(threadIndex int, err error) (bool, error) {
 
+	const MAX_ATTEMPTS = 15
+
+	maximumBackoff := 64
+	if maximumBackoff < storage.numberOfThreads {
+		maximumBackoff = storage.numberOfThreads
+	}
 	retry := false
 	message := ""
 	if err == nil {
 		storage.backoffs[threadIndex] = 1
+		storage.attempts[threadIndex] = 0
 		return false, nil
 	} else if e, ok := err.(*googleapi.Error); ok {
 		if 500 <= e.Code && e.Code < 600 {
@@ -63,7 +76,7 @@ func (storage *GCDStorage) shouldRetry(threadIndex int, err error) (bool, error)
 			retry = true
 		} else if e.Code == 403 {
 			// User Rate Limit Exceeded
-            message = "User Rate Limit Exceeded"
+			message = e.Message
 			retry = true
 		} else if e.Code == 401 {
 			// Only retry on authorization error when storage has been connected before
@@ -84,21 +97,39 @@ func (storage *GCDStorage) shouldRetry(threadIndex int, err error) (bool, error)
 		retry = err.Temporary()
 	}
 
-    if !retry || storage.backoffs[threadIndex] >= 256 {
-        storage.backoffs[threadIndex] = 1
+	if !retry {
 		return false, err
 	}
 
-    delay := float32(storage.backoffs[threadIndex]) * rand.Float32()
-    LOG_DEBUG("GCD_RETRY", "%s; retrying after %.2f seconds", message, delay)
-    time.Sleep(time.Duration(float32(storage.backoffs[threadIndex]) * float32(time.Second)))
+	if storage.attempts[threadIndex] >= MAX_ATTEMPTS {
+		LOG_INFO("GCD_RETRY", "[%d] Maximum number of retries reached (backoff: %d, attempts: %d)",
+			threadIndex, storage.backoffs[threadIndex], storage.attempts[threadIndex])
+		storage.backoffs[threadIndex] = 1
+		storage.attempts[threadIndex] = 0
+		return false, err
+	}
+
+	if storage.backoffs[threadIndex] < maximumBackoff {
 		storage.backoffs[threadIndex] *= 2
+	}
+	if storage.backoffs[threadIndex] > maximumBackoff {
+		storage.backoffs[threadIndex] = maximumBackoff
+	}
+	storage.attempts[threadIndex] += 1
+	delay := float64(storage.backoffs[threadIndex]) * rand.Float64() * 2
+	LOG_DEBUG("GCD_RETRY", "[%d] %s; retrying after %.2f seconds (backoff: %d, attempts: %d)",
+		threadIndex, message, delay, storage.backoffs[threadIndex], storage.attempts[threadIndex])
+	time.Sleep(time.Duration(delay * float64(time.Second)))
+
 	return true, nil
 }
 
-func (storage *GCDStorage) convertFilePath(filePath string) (string) {
+// convertFilePath converts the path for a fossil in the form of 'chunks/id.fsl' to 'fossils/id'.  This is because
+// GCD doesn't support file renaming. Instead, it only allows one file to be moved from one directory to another.
+// By adding a layer of path conversion we're pretending that we can rename between 'chunks/id' and 'chunks/id.fsl'
+func (storage *GCDStorage) convertFilePath(filePath string) string {
 	if strings.HasPrefix(filePath, "chunks/") && strings.HasSuffix(filePath, ".fsl") {
-        return "fossils/" + filePath[len("chunks/"):len(filePath) - len(".fsl")]
+		return "fossils/" + filePath[len("chunks/"):len(filePath)-len(".fsl")]
 	}
 	return filePath
 }
@@ -129,21 +160,21 @@ func (storage *GCDStorage) deletePathID(path string) {
 	storage.idCacheLock.Unlock()
 }
 
-func (storage *GCDStorage) listFiles(threadIndex int, parentID string, listFiles bool) ([]*drive.File, error) {
+func (storage *GCDStorage) listFiles(threadIndex int, parentID string, listFiles bool, listDirectories bool) ([]*drive.File, error) {
 
 	if parentID == "" {
 		return nil, fmt.Errorf("No parent ID provided")
 	}
 
-    files := []*drive.File {}
+	files := []*drive.File{}
 
 	startToken := ""
 
-    query := "'" + parentID + "' in parents and "
-    if listFiles {
-        query += "mimeType != 'application/vnd.google-apps.folder'"
-    } else {
-        query += "mimeType = 'application/vnd.google-apps.folder'"
+	query := "'" + parentID + "' in parents and trashed = false "
+	if listFiles && !listDirectories {
+		query += "and mimeType != 'application/vnd.google-apps.folder'"
+	} else if !listFiles && !listDirectories {
+		query += "and mimeType = 'application/vnd.google-apps.folder'"
 	}
 
 	maxCount := int64(1000)
@@ -174,7 +205,6 @@ func (storage *GCDStorage) listFiles(threadIndex int, parentID string, listFiles
 		}
 	}
 
-
 	return files, nil
 }
 
@@ -184,7 +214,7 @@ func (storage *GCDStorage) listByName(threadIndex int, parentID string, name str
 	var err error
 
 	for {
-        query := "name = '" + name + "' and '" + parentID + "' in parents"
+		query := "name = '" + name + "' and '" + parentID + "' in parents and trashed = false "
 		fileList, err = storage.service.Files.List().Q(query).Fields("files(name, mimeType, id, size)").Do()
 
 		if retry, e := storage.shouldRetry(threadIndex, err); e == nil && !retry {
@@ -202,10 +232,17 @@ func (storage *GCDStorage) listByName(threadIndex int, parentID string, name str
 
 	file := fileList.Files[0]
 
-    return file.Id, file.MimeType == "application/vnd.google-apps.folder", file.Size, nil
+	return file.Id, file.MimeType == GCDDirectoryMimeType, file.Size, nil
 }
 
-func (storage *GCDStorage) getIDFromPath(threadIndex int, path string) (string, error) {
+// getIDFromPath returns the id of the given path.  If 'createDirectories' is true, create the given path and all its
+// parent directories if they don't exist.  Note that if 'createDirectories' is false, it may return an empty 'fileID'
+// if the file doesn't exist.
+func (storage *GCDStorage) getIDFromPath(threadIndex int, filePath string, createDirectories bool) (string, error) {
+
+	if fileID, ok := storage.findPathID(filePath); ok {
+		return fileID, nil
+	}
 
 	fileID := "root"
 
@@ -213,22 +250,18 @@ func (storage *GCDStorage) getIDFromPath(threadIndex int, path string) (string,
 		fileID = rootID
 	}
 
-    names := strings.Split(path, "/")
+	names := strings.Split(filePath, "/")
 	current := ""
 	for i, name := range names {
-
-        if len(current) == 0 {
-            current = name
-        } else {
-            current = current + "/" + name
-        }
-
+		// Find the intermediate directory in the cache first.
+		current = path.Join(current, name)
 		currentID, ok := storage.findPathID(current)
 		if ok {
 			fileID = currentID
 			continue
 		}
 
+		// Check if the directory exists.
 		var err error
 		var isDir bool
 		fileID, isDir, _, err = storage.listByName(threadIndex, fileID, name)
@@ -236,10 +269,30 @@ func (storage *GCDStorage) getIDFromPath(threadIndex int, path string) (string,
 			return "", err
 		}
 		if fileID == "" {
-            return "", fmt.Errorf("Path %s doesn't exist", path)
+			if !createDirectories {
+				return "", nil
+			}
+
+			// Only one thread can create the directory at a time -- GCD allows multiple directories
+			// to have the same name but different ids.
+			storage.createDirectoryLock.Lock()
+			err = storage.CreateDirectory(threadIndex, current)
+			storage.createDirectoryLock.Unlock()
+
+			if err != nil {
+				return "", fmt.Errorf("Failed to create directory '%s': %v", current, err)
+			}
+			currentID, ok = storage.findPathID(current)
+			if !ok {
+				return "", fmt.Errorf("Directory '%s' created by id not found", current)
+			}
+			fileID = currentID
+			continue
+		} else if isDir {
+			storage.savePathID(current, fileID)
 		}
 		if i != len(names) - 1 && !isDir {
-            return "", fmt.Errorf("Invalid path %s", path)
+			return "", fmt.Errorf("Path '%s' is not a directory", current)
 		}
 	}
 	return fileID, nil
@@ -253,40 +306,47 @@ func CreateGCDStorage(tokenFile string, storagePath string, threads int) (storag
 		return nil, err
 	}
 
-    gcdConfig := &GCDConfig {}
+	gcdConfig := &GCDConfig{}
 	if err := json.Unmarshal(description, gcdConfig); err != nil {
 		return nil, err
 	}
 
-    config := oauth2.Config{
+	oauth2Config := oauth2.Config{
 		ClientID:     gcdConfig.ClientID,
 		ClientSecret: gcdConfig.ClientSecret,
 		Endpoint:     gcdConfig.Endpoint,
 	}
 
-    authClient := config.Client(context.Background(), &gcdConfig.Token)
+	authClient := oauth2Config.Client(context.Background(), &gcdConfig.Token)
 
 	service, err := drive.New(authClient)
 	if err != nil {
 		return nil, err
 	}
 
-    storage = &GCDStorage {
+	storage = &GCDStorage{
 		service:         service,
 		numberOfThreads: threads,
 		idCache:         make(map[string]string),
-        idCacheLock: &sync.Mutex{},
 		backoffs:        make([]int, threads),
+		attempts:        make([]int, threads),
 	}
 
-    storagePathID, err := storage.getIDFromPath(0, storagePath)
+	for i := range storage.backoffs {
+		storage.backoffs[i] = 1
+		storage.attempts[i] = 0
+	}
+
+	storagePathID, err := storage.getIDFromPath(0, storagePath, true)
 	if err != nil {
 		return nil, err
 	}
 
+	// Reset the id cache and start with 'storagePathID' as the root
+	storage.idCache = make(map[string]string)
 	storage.idCache[""] = storagePathID
 
-    for _, dir := range []string { "chunks", "snapshots", "fossils" } {
+	for _, dir := range []string{"chunks", "snapshots", "fossils"} {
 		dirID, isDir, _, err := storage.listByName(0, storagePathID, dir)
 		if err != nil {
 			return nil, err
@@ -297,7 +357,7 @@ func CreateGCDStorage(tokenFile string, storagePath string, threads int) (storag
 				return nil, err
 			}
 		} else if !isDir {
-            return nil, fmt.Errorf("%s/%s is not a directory", storagePath + "/" + dir)
+			return nil, fmt.Errorf("%s/%s is not a directory", storagePath, dir)
 		} else {
 			storage.idCache[dir] = dirID
 		}
@@ -305,19 +365,20 @@ func CreateGCDStorage(tokenFile string, storagePath string, threads int) (storag
 
 	storage.isConnected = true
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
-
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *GCDStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	if dir == "snapshots" {
 
-        files, err := storage.listFiles(threadIndex, storage.getPathID(dir), false)
+		files, err := storage.listFiles(threadIndex, storage.getPathID(dir), false, true)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -330,12 +391,15 @@ func (storage *GCDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 		}
 		return subDirs, nil, nil
 	} else if strings.HasPrefix(dir, "snapshots/") {
-        pathID, err := storage.getIDFromPath(threadIndex, dir)
+		pathID, err := storage.getIDFromPath(threadIndex, dir, false)
 		if err != nil {
 			return nil, nil, err
 		}
+		if pathID == "" {
+			return nil, nil, fmt.Errorf("Path '%s' does not exist", dir)
+		}
 
-        entries, err := storage.listFiles(threadIndex, pathID, true)
+		entries, err := storage.listFiles(threadIndex, pathID, true, false)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -343,7 +407,6 @@ func (storage *GCDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 		files := []string{}
 
 		for _, entry := range entries {
-            storage.savePathID(dir + "/" + entry.Name, entry.Id)
 			files = append(files, entry.Name)
 		}
 		return files, nil, nil
@@ -351,20 +414,33 @@ func (storage *GCDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 		files := []string{}
 		sizes := []int64{}
 
-        for _, parent := range []string { "chunks", "fossils" } {
-            entries, err := storage.listFiles(threadIndex, storage.getPathID(parent), true)
+		parents := []string{"chunks", "fossils"}
+		for i := 0; i < len(parents); i++ {
+			parent := parents[i]
+			pathID, ok := storage.findPathID(parent)
+			if !ok {
+				continue
+			}
+			entries, err := storage.listFiles(threadIndex, pathID, true, true)
 			if err != nil {
 				return nil, nil, err
 			}
-
 			for _, entry := range entries {
+				if entry.MimeType != GCDDirectoryMimeType {
 					name := entry.Name
-                if parent == "fossils" {
-                    name += ".fsl"
+					if strings.HasPrefix(parent, "fossils") {
+						name = parent + "/" + name + ".fsl"
+						name = name[len("fossils/"):]
+					} else {
+						name = parent + "/" + name
+						name = name[len("chunks/"):]
 					}
-                storage.savePathID(parent + "/" + entry.Name, entry.Id)
 					files = append(files, name)
 					sizes = append(sizes, entry.Size)
+				} else {
+					parents = append(parents, parent+ "/" + entry.Name)
+					storage.savePathID(parent + "/" + entry.Name, entry.Id)
+				}
 			}
 		}
 		return files, sizes, nil
@@ -375,14 +451,11 @@ func (storage *GCDStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *GCDStorage) DeleteFile(threadIndex int, filePath string) (err error) {
 	filePath = storage.convertFilePath(filePath)
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
-        fileID, err = storage.getIDFromPath(threadIndex, filePath)
+	fileID, err := storage.getIDFromPath(threadIndex, filePath, false)
 	if err != nil {
 		LOG_TRACE("GCD_STORAGE", "Ignored file deletion error: %v", err)
 		return nil
 	}
-    }
 
 	for {
 		err = storage.service.Files.Delete(fileID).Fields("id").Do()
@@ -407,16 +480,27 @@ func (storage *GCDStorage) MoveFile(threadIndex int, from string, to string) (er
 	from = storage.convertFilePath(from)
 	to = storage.convertFilePath(to)
 
-    fileID, ok := storage.findPathID(from)
-    if !ok {
-        return fmt.Errorf("Attempting to rename file %s with unknown id", to)
+	fileID, err := storage.getIDFromPath(threadIndex, from, false)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of '%s': %v", from, err)
+	}
+	if fileID == "" {
+		return fmt.Errorf("The file '%s' to be moved does not exist", from)
 	}
 
-    fromParentID := storage.getPathID("chunks")
-    toParentID := storage.getPathID("fossils")
+	fromParent := path.Dir(from)
+	fromParentID, err := storage.getIDFromPath(threadIndex, fromParent, false)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of the parent directory '%s': %v", fromParent, err)
+	}
+	if fromParentID == "" {
+		return fmt.Errorf("The parent directory '%s' does not exist", fromParent)
+	}
 
-    if strings.HasPrefix(from, "fossils") {
-        fromParentID, toParentID = toParentID, fromParentID
+	toParent := path.Dir(to)
+	toParentID, err := storage.getIDFromPath(threadIndex, toParent, true)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of the parent directory '%s': %v", toParent, err)
 	}
 
 	for {
@@ -430,16 +514,14 @@ func (storage *GCDStorage) MoveFile(threadIndex int, from string, to string) (er
 		}
 	}
 
-    storage.savePathID(to, storage.getPathID(from))
-    storage.deletePathID(from)
 	return nil
 }
 
-// CreateDirectory creates a new directory.
+// createDirectory creates a new directory.
 func (storage *GCDStorage) CreateDirectory(threadIndex int, dir string) (err error) {
 
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	exist, isDir, _, err := storage.GetFileInfo(threadIndex, dir)
@@ -454,30 +536,42 @@ func (storage *GCDStorage) CreateDirectory(threadIndex int, dir string) (err err
 		return nil
 	}
 
-    parentID := storage.getPathID("")
-    name := dir
-
-    if strings.HasPrefix(dir, "snapshots/") {
-        parentID = storage.getPathID("snapshots")
-        name = dir[len("snapshots/"):]
+	parentDir := path.Dir(dir)
+	if parentDir == "." {
+		parentDir = ""
 	}
-
-    file := &drive.File {
-        Name: name,
-        MimeType: "application/vnd.google-apps.folder",
-        Parents: []string { parentID },
+	parentID := storage.getPathID(parentDir)
+	if parentID == "" {
+		return fmt.Errorf("Parent directory '%s' does not exist", parentDir)
 	}
+	name := path.Base(dir)
+
+	var file *drive.File
 
 	for {
+		file = &drive.File{
+			Name:     name,
+			MimeType: GCDDirectoryMimeType,
+			Parents:  []string{parentID},
+		}
+
 		file, err = storage.service.Files.Create(file).Fields("id").Do()
 		if retry, err := storage.shouldRetry(threadIndex, err); err == nil && !retry {
 			break
-        } else if retry {
+		} else {
+
+			// Check if the directory has already been created by other thread
+			if _, ok := storage.findPathID(dir); ok {
+				return nil
+			}
+
+			if retry {
 				continue
 			} else {
 				return err
 			}
 		}
+	}
 
 	storage.savePathID(dir, file.Id)
 	return nil
@@ -485,72 +579,46 @@ func (storage *GCDStorage) CreateDirectory(threadIndex int, dir string) (err err
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *GCDStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
+	}
+	filePath = storage.convertFilePath(filePath)
+
+	fileID, ok := storage.findPathID(filePath)
+	if ok {
+		// Only directories are saved in the case so this must be a directory
+		return true, true, 0, nil
 	}
 
-    // GetFileInfo is never called on a fossil
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
 	dir := path.Dir(filePath)
 	if dir == "." {
 		dir = ""
 	}
-        dirID, err := storage.getIDFromPath(threadIndex, dir)
+	dirID, err := storage.getIDFromPath(threadIndex, dir, false)
 	if err != nil {
 		return false, false, 0, err
 	}
+	if dirID == "" {
+		return false, false, 0, nil
+	}
 
 	fileID, isDir, size, err = storage.listByName(threadIndex, dirID, path.Base(filePath))
-        if fileID != "" {
+	if fileID != "" && isDir {
 		storage.savePathID(filePath, fileID)
 	}
 	return fileID != "", isDir, size, err
-    }
 
-    for {
-        file, err := storage.service.Files.Get(fileID).Fields("id, mimeType").Do()
-        if retry, err := storage.shouldRetry(threadIndex, err); err == nil && !retry {
-            return true, file.MimeType == "application/vnd.google-apps.folder", file.Size, nil
-        } else if retry {
-            continue
-        } else {
-            return false, false, 0, err
-        }
-    }
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *GCDStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    parentID := ""
-    filePath = "chunks/" + chunkID
-    realPath := storage.convertFilePath(filePath)
-    if isFossil {
-        parentID = storage.getPathID("fossils")
-        filePath += ".fsl"
-    } else {
-        parentID = storage.getPathID("chunks")
-    }
-
-    fileID := ""
-    fileID, _, size, err = storage.listByName(threadIndex, parentID, chunkID)
-    if fileID != "" {
-        storage.savePathID(realPath, fileID)
-    }
-    return filePath, fileID != "", size, err
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *GCDStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	// We never download the fossil so there is no need to convert the path
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
-        fileID, err = storage.getIDFromPath(threadIndex, filePath)
+	fileID, err := storage.getIDFromPath(threadIndex, filePath, false)
 	if err != nil {
 		return err
 	}
-        storage.savePathID(filePath, fileID)
+	if fileID == "" {
+		return fmt.Errorf("%s does not exist", filePath)
 	}
 
 	var response *http.Response
@@ -568,7 +636,7 @@ func (storage *GCDStorage) DownloadFile(threadIndex int, filePath string, chunk
 
 	defer response.Body.Close()
 
-    _, err = RateLimitedCopy(chunk, response.Body, storage.DownloadRateLimit / storage.numberOfThreads)
+	_, err = RateLimitedCopy(chunk, response.Body, storage.DownloadRateLimit/storage.numberOfThreads)
 	return err
 }
 
@@ -582,23 +650,19 @@ func (storage *GCDStorage) UploadFile(threadIndex int, filePath string, content
 		parent = ""
 	}
 
-    parentID, ok := storage.findPathID(parent)
-    if !ok {
-        parentID, err = storage.getIDFromPath(threadIndex, parent)
+	parentID, err := storage.getIDFromPath(threadIndex, parent, true)
 	if err != nil {
 		return err
 	}
-        storage.savePathID(parent, parentID)
-    }
 
-    file := &drive.File {
+	file := &drive.File{
 		Name:     path.Base(filePath),
-        MimeType: "application/octet-stream",
-        Parents: []string { parentID },
+		MimeType: GCDFileMimeType,
+		Parents:  []string{parentID},
 	}
 
 	for {
-        reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
+		reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
 		_, err = storage.service.Files.Create(file).Media(reader).Fields("id").Do()
 		if retry, err := storage.shouldRetry(threadIndex, err); err == nil && !retry {
 			break
@@ -614,16 +678,16 @@ func (storage *GCDStorage) UploadFile(threadIndex int, filePath string, content
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *GCDStorage) IsCacheNeeded() (bool) { return true }
+func (storage *GCDStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *GCDStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *GCDStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *GCDStorage) IsStrongConsistent() (bool) { return false }
+func (storage *GCDStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *GCDStorage) IsFastListing() (bool) { return true }
+func (storage *GCDStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *GCDStorage) EnableTestMode() { storage.TestMode = true }

src/duplicacy_gcsstorage.go

@@ -5,33 +5,32 @@
 package duplicacy
 
 import (
-    "io"
-	"fmt"
-	"net"
-	"time"
-    "net/url"
-    "math/rand"
-    "io/ioutil"
 	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"net"
+	"net/url"
+	"time"
 
+	gcs "cloud.google.com/go/storage"
 	"golang.org/x/net/context"
 	"golang.org/x/oauth2"
 	"golang.org/x/oauth2/google"
-	gcs "cloud.google.com/go/storage"
+	"google.golang.org/api/googleapi"
 	"google.golang.org/api/iterator"
 	"google.golang.org/api/option"
-    "google.golang.org/api/googleapi"
 )
 
 type GCSStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	bucket     *gcs.BucketHandle
 	storageDir string
 
 	numberOfThreads int
 	TestMode        bool
-
 }
 
 type GCSConfig struct {
@@ -51,7 +50,7 @@ func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, th
 		return nil, err
 	}
 
-    var object map[string]interface {}
+	var object map[string]interface{}
 
 	err = json.Unmarshal(description, &object)
 	if err != nil {
@@ -74,7 +73,7 @@ func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, th
 		}
 		tokenSource = config.TokenSource(ctx)
 	} else {
-		gcsConfig := &GCSConfig {}
+		gcsConfig := &GCSConfig{}
 		if err := json.Unmarshal(description, gcsConfig); err != nil {
 			return nil, err
 		}
@@ -92,18 +91,19 @@ func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, th
 
 	bucket := client.Bucket(bucketName)
 
-    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
 		storageDir += "/"
 	}
 
-    storage = &GCSStorage {
+	storage = &GCSStorage{
 		bucket:          bucket,
 		storageDir:      storageDir,
 		numberOfThreads: threads,
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
-
 }
 
 func (storage *GCSStorage) shouldRetry(backoff *int, err error) (bool, error) {
@@ -149,14 +149,13 @@ func (storage *GCSStorage) shouldRetry(backoff *int, err error) (bool, error) {
 	return true, nil
 }
 
-
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *GCSStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
-	query := gcs.Query {
+	query := gcs.Query{
 		Prefix: storage.storageDir + dir + "/",
 	}
 	dirOnly := false
@@ -240,19 +239,6 @@ func (storage *GCSStorage) GetFileInfo(threadIndex int, filePath string) (exist
 	return true, false, attributes.Size, nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *GCSStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-	exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-	return filePath, exist, size, err
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *GCSStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	readCloser, err := storage.bucket.Object(storage.storageDir + filePath).NewReader(context.Background())
@@ -260,7 +246,7 @@ func (storage *GCSStorage) DownloadFile(threadIndex int, filePath string, chunk
 		return err
 	}
 	defer readCloser.Close()
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThreads)
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThreads)
 	return err
 }
 
@@ -271,7 +257,7 @@ func (storage *GCSStorage) UploadFile(threadIndex int, filePath string, content
 	for {
 		writeCloser := storage.bucket.Object(storage.storageDir + filePath).NewWriter(context.Background())
 		defer writeCloser.Close()
-		reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
+		reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
 		_, err = io.Copy(writeCloser, reader)
 
 		if retry, e := storage.shouldRetry(&backoff, err); e == nil && !retry {
@@ -288,16 +274,16 @@ func (storage *GCSStorage) UploadFile(threadIndex int, filePath string, content
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *GCSStorage) IsCacheNeeded() (bool) { return true }
+func (storage *GCSStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *GCSStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *GCSStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *GCSStorage) IsStrongConsistent() (bool) { return true }
+func (storage *GCSStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *GCSStorage) IsFastListing() (bool) { return true }
+func (storage *GCSStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *GCSStorage) EnableTestMode() { storage.TestMode = true }

src/duplicacy_hubicclient.go

@@ -5,18 +5,18 @@
 package duplicacy
 
 import (
-    "fmt"
-    "net"
-    "time"
-    "sync"
 	"bytes"
-    "strings"
-    "io/ioutil"
 	"encoding/json"
+	"fmt"
 	"io"
+	"io/ioutil"
+	"math/rand"
+	"net"
 	"net/http"
 	net_url "net/url"
-    "math/rand"
+	"strings"
+	"sync"
+	"time"
 
 	"golang.org/x/oauth2"
 )
@@ -65,14 +65,14 @@ func NewHubicClient(tokenFile string) (*HubicClient, error) {
 	}
 
 	client := &HubicClient{
-        HTTPClient: &http.Client {
-            Transport: &http.Transport {
+		HTTPClient: &http.Client{
+			Transport: &http.Transport{
 				Dial: (&net.Dialer{
 					Timeout:   30 * time.Second,
 					KeepAlive: 30 * time.Second,
 				}).Dial,
 				TLSHandshakeTimeout:   60 * time.Second,
-                ResponseHeaderTimeout: 30 * time.Second,
+				ResponseHeaderTimeout: 300 * time.Second,
 				ExpectContinueTimeout: 10 * time.Second,
 			},
 		},
@@ -82,7 +82,7 @@ func NewHubicClient(tokenFile string) (*HubicClient, error) {
 		CredentialLock: &sync.Mutex{},
 	}
 
-    err = client.RefreshToken()
+	err = client.RefreshToken(false)
 	if err != nil {
 		return nil, err
 	}
@@ -100,7 +100,7 @@ func (client *HubicClient) call(url string, method string, input interface{}, ex
 	var response *http.Response
 
 	backoff := 1
-    for i := 0; i < 8; i++ {
+	for i := 0; i < 11; i++ {
 
 		LOG_DEBUG("HUBIC_CALL", "%s %s", method, url)
 
@@ -137,7 +137,7 @@ func (client *HubicClient) call(url string, method string, input interface{}, ex
 
 		if url == HubicCredentialURL {
 			client.TokenLock.Lock()
-            request.Header.Set("Authorization", "Bearer " + client.Token.AccessToken)
+			request.Header.Set("Authorization", "Bearer "+client.Token.AccessToken)
 			client.TokenLock.Unlock()
 		} else if url != HubicRefreshTokenURL {
 			client.CredentialLock.Lock()
@@ -151,6 +151,13 @@ func (client *HubicClient) call(url string, method string, input interface{}, ex
 
 		response, err = client.HTTPClient.Do(request)
 		if err != nil {
+			if url != HubicCredentialURL {
+				retryAfter := time.Duration((0.5 + rand.Float32()) * 1000.0 * float32(backoff))
+				LOG_INFO("HUBIC_CALL", "%s %s returned an error: %v; retry after %d milliseconds", method, url, err, retryAfter)
+				time.Sleep(retryAfter * time.Millisecond)
+				backoff *= 2
+				continue
+			}
 			return nil, 0, "", err
 		}
 
@@ -172,14 +179,14 @@ func (client *HubicClient) call(url string, method string, input interface{}, ex
 		if response.StatusCode == 401 {
 
 			if url == HubicRefreshTokenURL {
-                return nil, 0, "", HubicError { Status: response.StatusCode, Message: "Authorization error when refreshing token"}
+				return nil, 0, "", HubicError{Status: response.StatusCode, Message: "Authorization error when refreshing token"}
 			}
 
 			if url == HubicCredentialURL {
-                return nil, 0, "", HubicError { Status: response.StatusCode, Message: "Authorization error when retrieving credentials"}
+				return nil, 0, "", HubicError{Status: response.StatusCode, Message: "Authorization error when retrieving credentials"}
 			}
 
-            err = client.RefreshToken()
+			err = client.RefreshToken(true)
 			if err != nil {
 				return nil, 0, "", err
 			}
@@ -190,24 +197,30 @@ func (client *HubicClient) call(url string, method string, input interface{}, ex
 			}
 			continue
 		} else if response.StatusCode >= 500 && response.StatusCode < 600 {
-            retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
+			retryAfter := time.Duration((0.5 + rand.Float32()) * 1000.0 * float32(backoff))
+			LOG_INFO("HUBIC_RETRY", "Response status: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
+			time.Sleep(retryAfter * time.Millisecond)
+			backoff *= 2
+			continue
+		} else if response.StatusCode == 408 {
+			retryAfter := time.Duration((0.5 + rand.Float32()) * 1000.0 * float32(backoff))
 			LOG_INFO("HUBIC_RETRY", "Response status: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
 			time.Sleep(retryAfter * time.Millisecond)
 			backoff *= 2
 			continue
 		} else {
-            return nil, 0, "", HubicError { Status: response.StatusCode, Message: "Hubic API error"}
+			return nil, 0, "", HubicError{Status: response.StatusCode, Message: "Hubic API error"}
 		}
 	}
 
 	return nil, 0, "", fmt.Errorf("Maximum number of retries reached")
 }
 
-func (client *HubicClient) RefreshToken() (err error) {
+func (client *HubicClient) RefreshToken(force bool) (err error) {
 	client.TokenLock.Lock()
 	defer client.TokenLock.Unlock()
 
-    if client.Token.Valid() {
+	if !force && client.Token.Valid() {
 		return nil
 	}
 
@@ -264,7 +277,7 @@ type HubicEntry struct {
 
 func (client *HubicClient) ListEntries(path string) ([]HubicEntry, error) {
 
-    if len(path) > 0 && path[len(path) - 1] != '/' {
+	if len(path) > 0 && path[len(path)-1] != '/' {
 		path += "/"
 	}
 
@@ -308,8 +321,8 @@ func (client *HubicClient) ListEntries(path string) ([]HubicEntry, error) {
 				marker = entry.Name
 			} else {
 				marker = entry.Subdir
-                for len(entry.Subdir) > 0 && entry.Subdir[len(entry.Subdir) - 1] == '/' {
-                    entry.Subdir = entry.Subdir[:len(entry.Subdir) - 1]
+				for len(entry.Subdir) > 0 && entry.Subdir[len(entry.Subdir)-1] == '/' {
+					entry.Subdir = entry.Subdir[:len(entry.Subdir)-1]
 				}
 				entry.Name = entry.Subdir
 				entry.Type = "application/directory"
@@ -329,8 +342,8 @@ func (client *HubicClient) ListEntries(path string) ([]HubicEntry, error) {
 
 func (client *HubicClient) GetFileInfo(path string) (bool, bool, int64, error) {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
 	}
 
 	client.CredentialLock.Lock()
@@ -353,8 +366,8 @@ func (client *HubicClient) GetFileInfo(path string) (bool, bool, int64, error) {
 
 func (client *HubicClient) DownloadFile(path string) (io.ReadCloser, int64, error) {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
 	}
 
 	client.CredentialLock.Lock()
@@ -367,8 +380,8 @@ func (client *HubicClient) DownloadFile(path string) (io.ReadCloser, int64, erro
 
 func (client *HubicClient) UploadFile(path string, content []byte, rateLimit int) (err error) {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
 	}
 
 	client.CredentialLock.Lock()
@@ -390,8 +403,8 @@ func (client *HubicClient) UploadFile(path string, content []byte, rateLimit int
 
 func (client *HubicClient) DeleteFile(path string) error {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
 	}
 
 	client.CredentialLock.Lock()
@@ -410,12 +423,12 @@ func (client *HubicClient) DeleteFile(path string) error {
 
 func (client *HubicClient) MoveFile(from string, to string) error {
 
-    for len(from) > 0 && from[len(from) - 1] == '/' {
-        from = from[:len(from) - 1]
+	for len(from) > 0 && from[len(from)-1] == '/' {
+		from = from[:len(from)-1]
 	}
 
-    for len(to) > 0 && to[len(to) - 1] == '/' {
-        to = to[:len(to) - 1]
+	for len(to) > 0 && to[len(to)-1] == '/' {
+		to = to[:len(to)-1]
 	}
 
 	client.CredentialLock.Lock()
@@ -436,10 +449,10 @@ func (client *HubicClient) MoveFile(from string, to string) error {
 	return client.DeleteFile(from)
 }
 
-func (client *HubicClient) CreateDirectory(path string) (error) {
+func (client *HubicClient) CreateDirectory(path string) error {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
 	}
 
 	client.CredentialLock.Lock()

src/duplicacy_hubicclient_test.go

@@ -5,11 +5,11 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "testing"
 	"crypto/sha256"
 	"encoding/hex"
+	"fmt"
+	"io"
+	"testing"
 
 	crypto_rand "crypto/rand"
 	"math/rand"
@@ -73,7 +73,7 @@ func TestHubicClient(t *testing.T) {
 	maxFileSize := 64 * 1024
 
 	for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
+		content := make([]byte, rand.Int()%maxFileSize+1)
 		_, err = crypto_rand.Read(content)
 		if err != nil {
 			t.Errorf("Error generating random content: %v", err)
@@ -86,7 +86,7 @@ func TestHubicClient(t *testing.T) {
 
 		fmt.Printf("file: %s\n", filename)
 
-        err = hubicClient.UploadFile("test/test1/" + filename, content, 100)
+		err = hubicClient.UploadFile("test/test1/"+filename, content, 100)
 		if err != nil {
 			/*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
 				t.Errorf("Failed to upload the file %s: %v", filename, err)
@@ -104,9 +104,9 @@ func TestHubicClient(t *testing.T) {
 	for _, entry := range entries {
 
 		exists, isDir, size, err := hubicClient.GetFileInfo("test/test1/" + entry.Name)
-        fmt.Printf("%s exists: %t, isDir: %t, size: %d, err: %v\n", "test/test1/" + entry.Name, exists, isDir, size, err)
+		fmt.Printf("%s exists: %t, isDir: %t, size: %d, err: %v\n", "test/test1/"+entry.Name, exists, isDir, size, err)
 
-        err = hubicClient.MoveFile("test/test1/" + entry.Name, "test/test2/" + entry.Name)
+		err = hubicClient.MoveFile("test/test1/"+entry.Name, "test/test2/"+entry.Name)
 		if err != nil {
 			t.Errorf("Failed to move %s: %v", entry.Name, err)
 			return

src/duplicacy_hubicstorage.go

@@ -10,7 +10,7 @@ import (
 )
 
 type HubicStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	client          *HubicClient
 	storageDir      string
@@ -20,8 +20,8 @@ type HubicStorage struct {
 // CreateHubicStorage creates an Hubic storage object.
 func CreateHubicStorage(tokenFile string, storagePath string, threads int) (storage *HubicStorage, err error) {
 
-    for len(storagePath) > 0 && storagePath[len(storagePath) - 1] == '/' {
-        storagePath = storagePath[:len(storagePath) - 1]
+	for len(storagePath) > 0 && storagePath[len(storagePath)-1] == '/' {
+		storagePath = storagePath[:len(storagePath)-1]
 	}
 
 	client, err := NewHubicClient(tokenFile)
@@ -42,13 +42,13 @@ func CreateHubicStorage(tokenFile string, storagePath string, threads int) (stor
 		return nil, fmt.Errorf("Path '%s' is not a directory", storagePath)
 	}
 
-    storage = &HubicStorage {
+	storage = &HubicStorage{
 		client:          client,
 		storageDir:      storagePath,
 		numberOfThreads: threads,
 	}
 
-    for _, path := range []string { "chunks", "snapshots" } {
+	for _, path := range []string{"chunks", "snapshots"} {
 		dir := storagePath + "/" + path
 		exists, isDir, _, err := client.GetFileInfo(dir)
 		if err != nil {
@@ -64,14 +64,15 @@ func CreateHubicStorage(tokenFile string, storagePath string, threads int) (stor
 		}
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
-
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *HubicStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	if dir == "snapshots" {
@@ -83,7 +84,7 @@ func (storage *HubicStorage) ListFiles(threadIndex int, dir string) ([]string, [
 		subDirs := []string{}
 		for _, entry := range entries {
 			if entry.Type == "application/directory" {
-                subDirs = append(subDirs, entry.Name + "/")
+				subDirs = append(subDirs, entry.Name+"/")
 			}
 		}
 		return subDirs, nil, nil
@@ -105,18 +106,20 @@ func (storage *HubicStorage) ListFiles(threadIndex int, dir string) ([]string, [
 	} else {
 		files := []string{}
 		sizes := []int64{}
-        entries, err := storage.client.ListEntries(storage.storageDir + "/chunks")
+		entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
 		if err != nil {
 			return nil, nil, err
 		}
 
 		for _, entry := range entries {
 			if entry.Type == "application/directory" {
-                continue
-            }
+				files = append(files,  entry.Name + "/")
+				sizes = append(sizes, 0)
+			} else {
 				files = append(files, entry.Name)
 				sizes = append(sizes, entry.Size)
 			}
+		}
 		return files, sizes, nil
 	}
 
@@ -142,8 +145,8 @@ func (storage *HubicStorage) MoveFile(threadIndex int, from string, to string) (
 
 // CreateDirectory creates a new directory.
 func (storage *HubicStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	return storage.client.CreateDirectory(storage.storageDir + "/" + dir)
@@ -152,24 +155,12 @@ func (storage *HubicStorage) CreateDirectory(threadIndex int, dir string) (err e
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *HubicStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
 	}
 	return storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *HubicStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
-    return filePath, exist, size, err
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *HubicStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	readCloser, _, err := storage.client.DownloadFile(storage.storageDir + "/" + filePath)
@@ -179,27 +170,27 @@ func (storage *HubicStorage) DownloadFile(threadIndex int, filePath string, chun
 
 	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThreads)
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThreads)
 	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *HubicStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    return storage.client.UploadFile(storage.storageDir + "/" + filePath, content, storage.UploadRateLimit / storage.numberOfThreads)
+	return storage.client.UploadFile(storage.storageDir+"/"+filePath, content, storage.UploadRateLimit/storage.numberOfThreads)
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *HubicStorage) IsCacheNeeded() (bool) { return true }
+func (storage *HubicStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *HubicStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *HubicStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *HubicStorage) IsStrongConsistent() (bool) { return false }
+func (storage *HubicStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *HubicStorage) IsFastListing() (bool) { return true }
+func (storage *HubicStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *HubicStorage) EnableTestMode() {

src/duplicacy_keyring_windows.go

@@ -5,10 +5,10 @@
 package duplicacy
 
 import (
+	"encoding/json"
+	"io/ioutil"
 	"syscall"
 	"unsafe"
-    "io/ioutil"
-    "encoding/json"
 )
 
 var keyringFile string
@@ -33,11 +33,11 @@ func SetKeyringFile(path string) {
 
 func keyringEncrypt(value []byte) ([]byte, error) {
 
-    dataIn := DATA_BLOB {
+	dataIn := DATA_BLOB{
 		pbData: &value[0],
 		cbData: uint32(len(value)),
 	}
-    dataOut := DATA_BLOB {}
+	dataOut := DATA_BLOB{}
 
 	r, _, err := procEncryptData.Call(uintptr(unsafe.Pointer(&dataIn)),
 		0, 0, 0, 0, 0, uintptr(unsafe.Pointer(&dataOut)))
@@ -57,11 +57,11 @@ func keyringEncrypt(value []byte) ([]byte, error) {
 
 func keyringDecrypt(value []byte) ([]byte, error) {
 
-    dataIn := DATA_BLOB {
+	dataIn := DATA_BLOB{
 		pbData: &value[0],
 		cbData: uint32(len(value)),
 	}
-    dataOut := DATA_BLOB {}
+	dataOut := DATA_BLOB{}
 
 	r, _, err := procDecryptData.Call(uintptr(unsafe.Pointer(&dataIn)),
 		0, 0, 0, 0, 0, uintptr(unsafe.Pointer(&dataOut)))

src/duplicacy_log.go

@@ -5,12 +5,12 @@
 package duplicacy
 
 import (
-    "os"
 	"fmt"
-    "time"
+	"os"
+	"runtime/debug"
 	"sync"
 	"testing"
-    "runtime/debug"
+	"time"
 )
 
 const (

src/duplicacy_oneclient.go

@@ -5,16 +5,16 @@
 package duplicacy
 
 import (
-    "fmt"
-    "time"
-    "sync"
 	"bytes"
-    "strings"
-    "io/ioutil"
 	"encoding/json"
+	"fmt"
 	"io"
-    "net/http"
+	"io/ioutil"
 	"math/rand"
+	"net/http"
+	"strings"
+	"sync"
+	"time"
 
 	"golang.org/x/oauth2"
 )
@@ -65,6 +65,8 @@ func NewOneDriveClient(tokenFile string) (*OneDriveClient, error) {
 		TokenLock:  &sync.Mutex{},
 	}
 
+	client.RefreshToken(false)
+
 	return client, nil
 }
 
@@ -108,7 +110,7 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
 
 		if url != OneDriveRefreshTokenURL {
 			client.TokenLock.Lock()
-            request.Header.Set("Authorization", "Bearer " + client.Token.AccessToken)
+			request.Header.Set("Authorization", "Bearer "+client.Token.AccessToken)
 			client.TokenLock.Unlock()
 		}
 		if contentType != "" {
@@ -121,7 +123,7 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
 				if strings.Contains(err.Error(), "TLS handshake timeout") {
 					// Give a long timeout regardless of backoff when a TLS timeout happens, hoping that
 					// idle connections are not to be reused on reconnect.
-                    retryAfter := time.Duration(rand.Float32() * 60000 + 180000)
+					retryAfter := time.Duration(rand.Float32()*60000 + 180000)
 					LOG_INFO("ONEDRIVE_RETRY", "TLS handshake timeout; retry after %d milliseconds", retryAfter)
 					time.Sleep(retryAfter * time.Millisecond)
 				} else {
@@ -144,17 +146,17 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
 
 		defer response.Body.Close()
 
-        errorResponse := &OneDriveErrorResponse {
-            Error:  OneDriveError { Status: response.StatusCode },
+		errorResponse := &OneDriveErrorResponse{
+			Error: OneDriveError{Status: response.StatusCode},
 		}
 
 		if response.StatusCode == 401 {
 
 			if url == OneDriveRefreshTokenURL {
-                return nil, 0, OneDriveError { Status: response.StatusCode, Message: "Authorization error when refreshing token"}
+				return nil, 0, OneDriveError{Status: response.StatusCode, Message: "Authorization error when refreshing token"}
 			}
 
-            err = client.RefreshToken()
+			err = client.RefreshToken(true)
 			if err != nil {
 				return nil, 0, err
 			}
@@ -167,7 +169,7 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
 			continue
 		} else {
 			if err := json.NewDecoder(response.Body).Decode(errorResponse); err != nil {
-                return nil, 0, OneDriveError { Status: response.StatusCode, Message: fmt.Sprintf("Unexpected response"), }
+				return nil, 0, OneDriveError{Status: response.StatusCode, Message: fmt.Sprintf("Unexpected response")}
 			}
 
 			errorResponse.Error.Status = response.StatusCode
@@ -178,11 +180,11 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
 	return nil, 0, fmt.Errorf("Maximum number of retries reached")
 }
 
-func (client *OneDriveClient) RefreshToken() (err error) {
+func (client *OneDriveClient) RefreshToken(force bool) (err error) {
 	client.TokenLock.Lock()
 	defer client.TokenLock.Unlock()
 
-    if client.Token.Valid() {
+	if !force && client.Token.Valid() {
 		return nil
 	}
 
@@ -213,7 +215,7 @@ func (client *OneDriveClient) RefreshToken() (err error) {
 type OneDriveEntry struct {
 	ID     string
 	Name   string
-    Folder   map[string] interface {}
+	Folder map[string]interface{}
 	Size   int64
 }
 
@@ -245,7 +247,7 @@ func (client *OneDriveClient) ListEntries(path string) ([]OneDriveEntry, error)
 
 		defer readCloser.Close()
 
-        output := &OneDriveListEntriesOutput {}
+		output := &OneDriveListEntriesOutput{}
 
 		if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
 			return nil, err
@@ -340,7 +342,7 @@ func (client *OneDriveClient) MoveFile(path string, parent string) error {
 	return nil
 }
 
-func (client *OneDriveClient) CreateDirectory(path string, name string) (error) {
+func (client *OneDriveClient) CreateDirectory(path string, name string) error {
 
 	url := OneDriveAPIURL + "/root/children"
 

src/duplicacy_oneclient_test.go

@@ -5,11 +5,11 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "testing"
 	"crypto/sha256"
 	"encoding/hex"
+	"fmt"
+	"io"
+	"testing"
 
 	crypto_rand "crypto/rand"
 	"math/rand"
@@ -30,7 +30,6 @@ func TestOneDriveClient(t *testing.T) {
 		fmt.Printf("name: %s, isDir: %t\n", file.Name, len(file.Folder) != 0)
 	}
 
-
 	testID, _, _, err := oneDriveClient.GetFileInfo("test")
 	if err != nil {
 		t.Errorf("Failed to list the test directory: %v", err)
@@ -74,7 +73,7 @@ func TestOneDriveClient(t *testing.T) {
 	maxFileSize := 64 * 1024
 
 	for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
+		content := make([]byte, rand.Int()%maxFileSize+1)
 		_, err = crypto_rand.Read(content)
 		if err != nil {
 			t.Errorf("Error generating random content: %v", err)
@@ -87,7 +86,7 @@ func TestOneDriveClient(t *testing.T) {
 
 		fmt.Printf("file: %s\n", filename)
 
-        err = oneDriveClient.UploadFile("test/test1/" + filename, content, 100)
+		err = oneDriveClient.UploadFile("test/test1/"+filename, content, 100)
 		if err != nil {
 			/*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
 				t.Errorf("Failed to upload the file %s: %v", filename, err)
@@ -103,7 +102,7 @@ func TestOneDriveClient(t *testing.T) {
 	}
 
 	for _, entry := range entries {
-        err = oneDriveClient.MoveFile("test/test1/" + entry.Name, "test/test2")
+		err = oneDriveClient.MoveFile("test/test1/"+entry.Name, "test/test2")
 		if err != nil {
 			t.Errorf("Failed to move %s: %v", entry.Name, err)
 			return

src/duplicacy_onestorage.go

@@ -11,7 +11,7 @@ import (
 )
 
 type OneDriveStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	client         *OneDriveClient
 	storageDir     string
@@ -21,8 +21,8 @@ type OneDriveStorage struct {
 // CreateOneDriveStorage creates an OneDrive storage object.
 func CreateOneDriveStorage(tokenFile string, storagePath string, threads int) (storage *OneDriveStorage, err error) {
 
-    for len(storagePath) > 0 && storagePath[len(storagePath) - 1] == '/' {
-        storagePath = storagePath[:len(storagePath) - 1]
+	for len(storagePath) > 0 && storagePath[len(storagePath)-1] == '/' {
+		storagePath = storagePath[:len(storagePath)-1]
 	}
 
 	client, err := NewOneDriveClient(tokenFile)
@@ -43,13 +43,13 @@ func CreateOneDriveStorage(tokenFile string, storagePath string, threads int) (s
 		return nil, fmt.Errorf("Path '%s' is not a directory", storagePath)
 	}
 
-    storage = &OneDriveStorage {
+	storage = &OneDriveStorage{
 		client:         client,
 		storageDir:     storagePath,
 		numberOfThread: threads,
 	}
 
-    for _, path := range []string { "chunks", "fossils", "snapshots" } {
+	for _, path := range []string{"chunks", "fossils", "snapshots"} {
 		dir := storagePath + "/" + path
 		dirID, isDir, _, err := client.GetFileInfo(dir)
 		if err != nil {
@@ -65,14 +65,23 @@ func CreateOneDriveStorage(tokenFile string, storagePath string, threads int) (s
 		}
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
 
 }
 
+func (storage *OneDriveStorage) convertFilePath(filePath string) string {
+	if strings.HasPrefix(filePath, "chunks/") && strings.HasSuffix(filePath, ".fsl") {
+		return "fossils/" + filePath[len("chunks/"):len(filePath)-len(".fsl")]
+	}
+	return filePath
+}
+
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *OneDriveStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	if dir == "snapshots" {
@@ -84,7 +93,7 @@ func (storage *OneDriveStorage) ListFiles(threadIndex int, dir string) ([]string
 		subDirs := []string{}
 		for _, entry := range entries {
 			if len(entry.Folder) > 0 {
-                subDirs = append(subDirs, entry.Name + "/")
+				subDirs = append(subDirs, entry.Name+"/")
 			}
 		}
 		return subDirs, nil, nil
@@ -105,19 +114,29 @@ func (storage *OneDriveStorage) ListFiles(threadIndex int, dir string) ([]string
 	} else {
 		files := []string{}
 		sizes := []int64{}
-        for _, parent := range []string {"chunks", "fossils" } {
+		parents := []string{"chunks", "fossils"}
+		for i := 0; i < len(parents); i++ {
+			parent := parents[i]
 			entries, err := storage.client.ListEntries(storage.storageDir + "/" + parent)
 			if err != nil {
 				return nil, nil, err
 			}
 
 			for _, entry := range entries {
+				if len(entry.Folder) == 0 {
 					name := entry.Name
-                if parent == "fossils" {
-                    name += ".fsl"
+					if strings.HasPrefix(parent, "fossils") {
+						name = parent + "/" + name + ".fsl"
+						name = name[len("fossils/"):]
+					} else {
+						name = parent + "/" + name
+						name = name[len("chunks/"):]
 					}
 					files = append(files, name)
 					sizes = append(sizes, entry.Size)
+				} else {
+					parents = append(parents, parent+"/"+entry.Name)
+				}
 			}
 		}
 		return files, sizes, nil
@@ -127,9 +146,7 @@ func (storage *OneDriveStorage) ListFiles(threadIndex int, dir string) ([]string
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *OneDriveStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    if strings.HasSuffix(filePath, ".fsl") && strings.HasPrefix(filePath, "chunks/") {
-        filePath = "fossils/" + filePath[len("chunks/"):len(filePath) - len(".fsl")]
-    }
+	filePath = storage.convertFilePath(filePath)
 
 	err = storage.client.DeleteFile(storage.storageDir + "/" + filePath)
 	if e, ok := err.(OneDriveError); ok && e.Status == 404 {
@@ -141,14 +158,11 @@ func (storage *OneDriveStorage) DeleteFile(threadIndex int, filePath string) (er
 
 // MoveFile renames the file.
 func (storage *OneDriveStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    fromPath := storage.storageDir + "/" + from
-    toParent := storage.storageDir + "/fossils"
-    if strings.HasSuffix(from, ".fsl") {
-        fromPath = storage.storageDir + "/fossils/" + from[len("chunks/"):len(from) - len(".fsl")]
-        toParent = storage.storageDir + "/chunks"
-    }
 
-    err = storage.client.MoveFile(fromPath, toParent)
+	fromPath := storage.storageDir + "/" + storage.convertFilePath(from)
+	toPath := storage.storageDir + "/" + storage.convertFilePath(to)
+
+	err = storage.client.MoveFile(fromPath, path.Dir(toPath))
 	if err != nil {
 		if e, ok := err.(OneDriveError); ok && e.Status == 409 {
 			LOG_DEBUG("ONEDRIVE_MOVE", "Ignore 409 conflict error")
@@ -161,8 +175,8 @@ func (storage *OneDriveStorage) MoveFile(threadIndex int, from string, to string
 
 // CreateDirectory creates a new directory.
 func (storage *OneDriveStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
 	}
 
 	parent := path.Dir(dir)
@@ -170,34 +184,23 @@ func (storage *OneDriveStorage) CreateDirectory(threadIndex int, dir string) (er
 	if parent == "." {
 		return storage.client.CreateDirectory(storage.storageDir, dir)
 	} else {
-        return storage.client.CreateDirectory(storage.storageDir + "/" + parent, path.Base(dir))
+		return storage.client.CreateDirectory(storage.storageDir+"/"+parent, path.Base(dir))
 	}
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *OneDriveStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
 	}
+
+	filePath = storage.convertFilePath(filePath)
+
 	fileID, isDir, size, err := storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
 	return fileID != "", isDir, size, err
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *OneDriveStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    realPath := storage.storageDir + "/" + filePath
-    if isFossil {
-        filePath += ".fsl"
-        realPath = storage.storageDir + "/fossils/" + chunkID
-    }
-
-    fileID, _, size, err := storage.client.GetFileInfo(realPath)
-    return filePath, fileID != "", size, err
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *OneDriveStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	readCloser, _, err := storage.client.DownloadFile(storage.storageDir + "/" + filePath)
@@ -207,13 +210,13 @@ func (storage *OneDriveStorage) DownloadFile(threadIndex int, filePath string, c
 
 	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThread)
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThread)
 	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *OneDriveStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    err = storage.client.UploadFile(storage.storageDir + "/" + filePath, content, storage.UploadRateLimit / storage.numberOfThread)
+	err = storage.client.UploadFile(storage.storageDir+"/"+filePath, content, storage.UploadRateLimit/storage.numberOfThread)
 
 	if e, ok := err.(OneDriveError); ok && e.Status == 409 {
 		LOG_TRACE("ONEDRIVE_UPLOAD", "File %s already exists", filePath)
@@ -225,16 +228,16 @@ func (storage *OneDriveStorage) UploadFile(threadIndex int, filePath string, con
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *OneDriveStorage) IsCacheNeeded() (bool) { return true }
+func (storage *OneDriveStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *OneDriveStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *OneDriveStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *OneDriveStorage) IsStrongConsistent() (bool) { return false }
+func (storage *OneDriveStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *OneDriveStorage) IsFastListing() (bool) { return true }
+func (storage *OneDriveStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *OneDriveStorage) EnableTestMode() {

src/duplicacy_preference.go

@@ -6,10 +6,11 @@ package duplicacy
 
 import (
 	"encoding/json"
-    "path"
 	"io/ioutil"
-    "reflect"
 	"os"
+	"path"
+	"reflect"
+	"strings"
 )
 
 // Preference stores options for each storage.
@@ -25,7 +26,7 @@ type Preference struct {
 }
 
 var preferencePath string
-var Preferences [] Preference
+var Preferences []Preference
 
 func LoadPreferences(repository string) bool {
 
@@ -43,7 +44,7 @@ func LoadPreferences(repository string) bool {
 			LOG_ERROR("DOT_DUPLICACY_PATH", "Failed to locate the preference path: %v", err)
 			return false
 		}
-        realPreferencePath := string(content)
+		realPreferencePath := strings.TrimSpace(string(content))
 		stat, err := os.Stat(realPreferencePath)
 		if err != nil {
 			LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", content, err)
@@ -73,6 +74,13 @@ func LoadPreferences(repository string) bool {
 		return false
 	}
 
+	for _, preference := range Preferences {
+		if strings.ToLower(preference.Name) == "ssh" {
+			LOG_ERROR("PREFERENCE_INVALID", "'%s' is an invalid storage name", preference.Name)
+			return false
+		}
+	}
+
 	return true
 }
 
@@ -90,7 +98,7 @@ func SetDuplicacyPreferencePath(p string) {
 	preferencePath = p
 }
 
-func SavePreferences() (bool) {
+func SavePreferences() bool {
 	description, err := json.MarshalIndent(Preferences, "", "    ")
 	if err != nil {
 		LOG_ERROR("PREFERENCE_MARSHAL", "Failed to marshal the repository preferences: %v", err)
@@ -107,7 +115,7 @@ func SavePreferences() (bool) {
 	return true
 }
 
-func FindPreference(name string) (*Preference) {
+func FindPreference(name string) *Preference {
 	for i, preference := range Preferences {
 		if preference.Name == name || preference.StorageURL == name {
 			return &Preferences[i]

src/duplicacy_s3cstorage.go

@@ -6,13 +6,14 @@ package duplicacy
 
 import (
 	"time"
+
 	"github.com/gilbertchen/goamz/aws"
 	"github.com/gilbertchen/goamz/s3"
 )
 
 // S3CStorage is a storage backend for s3 compatible storages that require V2 Signing.
 type S3CStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	buckets    []*s3.Bucket
 	storageDir string
@@ -30,10 +31,10 @@ func CreateS3CStorage(regionName string, endpoint string, bucketName string, sto
 		}
 		region = aws.Regions[regionName]
 	} else {
-        region = aws.Region { Name: regionName, S3Endpoint:"https://" + endpoint }
+		region = aws.Region{Name: regionName, S3Endpoint: "https://" + endpoint}
 	}
 
-    auth := aws.Auth{ AccessKey: accessKey, SecretKey: secretKey }
+	auth := aws.Auth{AccessKey: accessKey, SecretKey: secretKey}
 
 	var buckets []*s3.Bucket
 	for i := 0; i < threads; i++ {
@@ -48,27 +49,29 @@ func CreateS3CStorage(regionName string, endpoint string, bucketName string, sto
 		buckets = append(buckets, bucket)
 	}
 
-    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
 		storageDir += "/"
 	}
 
-    storage = &S3CStorage {
+	storage = &S3CStorage{
 		buckets:    buckets,
 		storageDir: storageDir,
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *S3CStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
-    if len(dir) > 0 && dir[len(dir) - 1] != '/' {
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
 		dir += "/"
 	}
 
 	dirLength := len(storage.storageDir + dir)
 	if dir == "snapshots/" {
-        results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "/", "", 100)
+		results, err := storage.buckets[threadIndex].List(storage.storageDir+dir, "/", "", 100)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -80,7 +83,7 @@ func (storage *S3CStorage) ListFiles(threadIndex int, dir string) (files []strin
 	} else if dir == "chunks/" {
 		marker := ""
 		for {
-            results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "", marker, 1000)
+			results, err := storage.buckets[threadIndex].List(storage.storageDir+dir, "", marker, 1000)
 			if err != nil {
 				return nil, nil, err
 			}
@@ -94,13 +97,13 @@ func (storage *S3CStorage) ListFiles(threadIndex int, dir string) (files []strin
 				break
 			}
 
-            marker = results.Contents[len(results.Contents) - 1].Key
+			marker = results.Contents[len(results.Contents)-1].Key
 		}
 		return files, sizes, nil
 
 	} else {
 
-        results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "", "", 1000)
+		results, err := storage.buckets[threadIndex].List(storage.storageDir+dir, "", "", 1000)
 		if err != nil {
 			return nil, nil, err
 		}
@@ -120,8 +123,8 @@ func (storage *S3CStorage) DeleteFile(threadIndex int, filePath string) (err err
 // MoveFile renames the file.
 func (storage *S3CStorage) MoveFile(threadIndex int, from string, to string) (err error) {
 
-    options := s3.CopyOptions { ContentType: "application/duplicacy" }
-    _, err = storage.buckets[threadIndex].PutCopy(storage.storageDir + to, s3.Private, options, storage.buckets[threadIndex].Name + "/" + storage.storageDir + from)
+	options := s3.CopyOptions{ContentType: "application/duplicacy"}
+	_, err = storage.buckets[threadIndex].PutCopy(storage.storageDir+to, s3.Private, options, storage.buckets[threadIndex].Name+"/"+storage.storageDir+from)
 	if err != nil {
 		return nil
 	}
@@ -137,7 +140,7 @@ func (storage *S3CStorage) CreateDirectory(threadIndex int, dir string) (err err
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *S3CStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    response, err := storage.buckets[threadIndex].Head(storage.storageDir + filePath, nil)
+	response, err := storage.buckets[threadIndex].Head(storage.storageDir+filePath, nil)
 	if err != nil {
 		if e, ok := err.(*s3.Error); ok && (e.StatusCode == 403 || e.StatusCode == 404) {
 			return false, false, 0, nil
@@ -153,25 +156,6 @@ func (storage *S3CStorage) GetFileInfo(threadIndex int, filePath string) (exist
 	}
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *S3CStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-    if err != nil {
-        return "", false, 0, err
-    } else {
-        return filePath, exist, size, err
-    }
-
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *S3CStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
@@ -182,7 +166,7 @@ func (storage *S3CStorage) DownloadFile(threadIndex int, filePath string, chunk
 
 	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.buckets))
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/len(storage.buckets))
 	return err
 
 }
@@ -190,23 +174,23 @@ func (storage *S3CStorage) DownloadFile(threadIndex int, filePath string, chunk
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *S3CStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
 
-    options := s3.Options { }
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.buckets))
-    return storage.buckets[threadIndex].PutReader(storage.storageDir + filePath, reader, int64(len(content)), "application/duplicacy", s3.Private, options)
+	options := s3.Options{}
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.buckets))
+	return storage.buckets[threadIndex].PutReader(storage.storageDir+filePath, reader, int64(len(content)), "application/duplicacy", s3.Private, options)
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *S3CStorage) IsCacheNeeded () (bool) { return true }
+func (storage *S3CStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *S3CStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *S3CStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *S3CStorage) IsStrongConsistent() (bool) { return false }
+func (storage *S3CStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *S3CStorage) IsFastListing() (bool) { return true }
+func (storage *S3CStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *S3CStorage) EnableTestMode() {}

src/duplicacy_s3storage.go

@@ -5,8 +5,8 @@
 package duplicacy
 
 import (
-    "strings"
 	"reflect"
+	"strings"
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/aws/awserr"
@@ -16,7 +16,7 @@ import (
 )
 
 type S3Storage struct {
-    RateLimitedStorage
+	StorageBase
 
 	client          *s3.S3
 	bucket          string
@@ -34,7 +34,7 @@ func CreateS3Storage(regionName string, endpoint string, bucketName string, stor
 	auth := credentials.NewStaticCredentials(accessKey, secretKey, token)
 
 	if regionName == "" && endpoint == "" {
-        defaultRegionConfig := &aws.Config {
+		defaultRegionConfig := &aws.Config{
 			Region:      aws.String("us-east-1"),
 			Credentials: auth,
 		}
@@ -53,7 +53,7 @@ func CreateS3Storage(regionName string, endpoint string, bucketName string, stor
 		}
 	}
 
-    config := &aws.Config {
+	s3Config := &aws.Config{
 		Region:           aws.String(regionName),
 		Credentials:      auth,
 		Endpoint:         aws.String(endpoint),
@@ -61,29 +61,31 @@ func CreateS3Storage(regionName string, endpoint string, bucketName string, stor
 		DisableSSL:       aws.Bool(!isSSLSupported),
 	}
 
-    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
 		storageDir += "/"
 	}
 
-    storage = &S3Storage {
-        client: s3.New(session.New(config)),
+	storage = &S3Storage{
+		client:          s3.New(session.New(s3Config)),
 		bucket:          bucketName,
 		storageDir:      storageDir,
 		numberOfThreads: threads,
 	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
 	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
-    if len(dir) > 0 && dir[len(dir) - 1] != '/' {
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
 		dir += "/"
 	}
 
 	if dir == "snapshots/" {
 		dir = storage.storageDir + dir
-        input := s3.ListObjectsInput {
+		input := s3.ListObjectsInput{
 			Bucket:    aws.String(storage.bucket),
 			Prefix:    aws.String(dir),
 			Delimiter: aws.String("/"),
@@ -103,7 +105,7 @@ func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string
 		dir = storage.storageDir + dir
 		marker := ""
 		for {
-            input := s3.ListObjectsInput {
+			input := s3.ListObjectsInput{
 				Bucket:  aws.String(storage.bucket),
 				Prefix:  aws.String(dir),
 				MaxKeys: aws.Int64(1000),
@@ -124,7 +126,7 @@ func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string
 				break
 			}
 
-            marker = *output.Contents[len(output.Contents) - 1].Key            
+			marker = *output.Contents[len(output.Contents)-1].Key
 		}
 		return files, sizes, nil
 	}
@@ -133,7 +135,7 @@ func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *S3Storage) DeleteFile(threadIndex int, filePath string) (err error) {
-    input := &s3.DeleteObjectInput {
+	input := &s3.DeleteObjectInput{
 		Bucket: aws.String(storage.bucket),
 		Key:    aws.String(storage.storageDir + filePath),
 	}
@@ -144,7 +146,7 @@ func (storage *S3Storage) DeleteFile(threadIndex int, filePath string) (err erro
 // MoveFile renames the file.
 func (storage *S3Storage) MoveFile(threadIndex int, from string, to string) (err error) {
 
-    input := &s3.CopyObjectInput {
+	input := &s3.CopyObjectInput{
 		Bucket:     aws.String(storage.bucket),
 		CopySource: aws.String(storage.bucket + "/" + storage.storageDir + from),
 		Key:        aws.String(storage.storageDir + to),
@@ -167,7 +169,7 @@ func (storage *S3Storage) CreateDirectory(threadIndex int, dir string) (err erro
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *S3Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    input := &s3.HeadObjectInput {
+	input := &s3.HeadObjectInput{
 		Bucket: aws.String(storage.bucket),
 		Key:    aws.String(storage.storageDir + filePath),
 	}
@@ -188,29 +190,10 @@ func (storage *S3Storage) GetFileInfo(threadIndex int, filePath string) (exist b
 	}
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *S3Storage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-    if err != nil {
-        return "", false, 0, err
-    } else {
-        return filePath, exist, size, err
-    }
-
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *S3Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
-   input := &s3.GetObjectInput {
+	input := &s3.GetObjectInput{
 		Bucket: aws.String(storage.bucket),
 		Key:    aws.String(storage.storageDir + filePath),
 	}
@@ -222,7 +205,7 @@ func (storage *S3Storage) DownloadFile(threadIndex int, filePath string, chunk *
 
 	defer output.Body.Close()
 
-    _, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit / len(storage.bucket))
+	_, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit/len(storage.bucket))
 	return err
 
 }
@@ -233,11 +216,11 @@ func (storage *S3Storage) UploadFile(threadIndex int, filePath string, content [
 	attempts := 0
 
 	for {
-        input := &s3.PutObjectInput {
+		input := &s3.PutObjectInput{
 			Bucket:      aws.String(storage.bucket),
 			Key:         aws.String(storage.storageDir + filePath),
 			ACL:         aws.String(s3.ObjectCannedACLPrivate),
-            Body: CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.bucket)),
+			Body:        CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.bucket)),
 			ContentType: aws.String("application/duplicacy"),
 		}
 
@@ -255,16 +238,16 @@ func (storage *S3Storage) UploadFile(threadIndex int, filePath string, content [
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *S3Storage) IsCacheNeeded () (bool) { return true }
+func (storage *S3Storage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *S3Storage) IsMoveFileImplemented() (bool) { return true }
+func (storage *S3Storage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *S3Storage) IsStrongConsistent() (bool) { return false }
+func (storage *S3Storage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *S3Storage) IsFastListing() (bool) { return true }
+func (storage *S3Storage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *S3Storage) EnableTestMode() {}

src/duplicacy_sftpstorage.go

@@ -7,56 +7,57 @@ package duplicacy
 import (
 	"fmt"
 	"io"
-    "os"
-    "net"
-    "path"
-    "time"
-    "runtime"
 	"math/rand"
+	"net"
+	"os"
+	"path"
+	"runtime"
+	"strings"
+	"time"
 
-    "golang.org/x/crypto/ssh"
 	"github.com/pkg/sftp"
+	"golang.org/x/crypto/ssh"
 )
 
 type SFTPStorage struct {
-    RateLimitedStorage
+	StorageBase
 
 	client          *sftp.Client
+	minimumNesting  int // The minimum level of directories to dive into before searching for the chunk file.
 	storageDir      string
 	numberOfThreads int
 }
 
 func CreateSFTPStorageWithPassword(server string, port int, username string, storageDir string,
-                                   password string, threads int) (storage *SFTPStorage, err error) {
-
-    authMethods := [] ssh.AuthMethod { ssh.Password(password) }
+	minimumNesting int, password string, threads int) (storage *SFTPStorage, err error) {
 
+	authMethods := []ssh.AuthMethod{ssh.Password(password)}
 
 	hostKeyCallback := func(hostname string, remote net.Addr,
 		key ssh.PublicKey) error {
 		return nil
 	}
 
-    return CreateSFTPStorage(server, port, username, storageDir, authMethods, hostKeyCallback, threads)
+	return CreateSFTPStorage(server, port, username, storageDir, minimumNesting, authMethods, hostKeyCallback, threads)
 }
 
-func CreateSFTPStorage(server string, port int, username string, storageDir string,
-                       authMethods [] ssh.AuthMethod,
+func CreateSFTPStorage(server string, port int, username string, storageDir string, minimumNesting int,
+	authMethods []ssh.AuthMethod,
 	hostKeyCallback func(hostname string, remote net.Addr,
 		key ssh.PublicKey) error, threads int) (storage *SFTPStorage, err error) {
 
-    config := &ssh.ClientConfig{
+	sftpConfig := &ssh.ClientConfig{
 		User:            username,
 		Auth:            authMethods,
 		HostKeyCallback: hostKeyCallback,
 	}
 
 	if server == "sftp.hidrive.strato.com" {
-        config.Ciphers = []string {"aes128-cbc", "aes128-ctr", "aes256-ctr"}
+		sftpConfig.Ciphers = []string{"aes128-ctr", "aes256-ctr"}
 	}
 
 	serverAddress := fmt.Sprintf("%s:%d", server, port)
-    connection, err := ssh.Dial("tcp", serverAddress, config)
+	connection, err := ssh.Dial("tcp", serverAddress, sftpConfig)
 	if err != nil {
 		return nil, err
 	}
@@ -67,8 +68,8 @@ func CreateSFTPStorage(server string, port int, username string, storageDir stri
 		return nil, err
 	}
 
-    for storageDir[len(storageDir) - 1] ==  '/' {
-        storageDir = storageDir[:len(storageDir) - 1]
+	for storageDir[len(storageDir)-1] == '/' {
+		storageDir = storageDir[:len(storageDir)-1]
 	}
 
 	fileInfo, err := client.Stat(storageDir)
@@ -80,9 +81,10 @@ func CreateSFTPStorage(server string, port int, username string, storageDir stri
 		return nil, fmt.Errorf("The storage path %s is not a directory", storageDir)
 	}
 
-    storage = &SFTPStorage {
+	storage = &SFTPStorage{
 		client:          client,
 		storageDir:      storageDir,
+		minimumNesting:  minimumNesting,
 		numberOfThreads: threads,
 	}
 
@@ -91,6 +93,8 @@ func CreateSFTPStorage(server string, port int, username string, storageDir stri
 
 	runtime.SetFinalizer(storage, CloseSFTPStorage)
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{2, 3}, 2)
 	return storage, nil
 }
 
@@ -108,7 +112,7 @@ func (storage *SFTPStorage) ListFiles(threadIndex int, dirPath string) (files []
 
 	for _, entry := range entries {
 		name := entry.Name()
-        if entry.IsDir() && name[len(name) - 1] != '/' {
+		if entry.IsDir() && name[len(name)-1] != '/' {
 			name += "/"
 		}
 
@@ -175,67 +179,6 @@ func (storage *SFTPStorage) GetFileInfo(threadIndex int, filePath string) (exist
 	return true, fileInfo.IsDir(), fileInfo.Size(), nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *SFTPStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-     dir := path.Join(storage.storageDir, "chunks")
-
-    suffix := ""
-    if isFossil {
-        suffix = ".fsl"
-    }
-
-    // The minimum level of directories to dive into before searching for the chunk file.
-    minimumLevel := 2
-
-    for level := 0; level * 2 < len(chunkID); level ++ {
-        if level >= minimumLevel {
-            filePath = path.Join(dir, chunkID[2 * level:]) + suffix
-            if stat, err := storage.client.Stat(filePath); err == nil && !stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], true, stat.Size(), nil
-            } else if err == nil && stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], true, 0, fmt.Errorf("The path %s is a directory", filePath)
-            }
-        }
-
-        // Find the subdirectory the chunk file may reside.
-        subDir := path.Join(dir, chunkID[2 * level: 2 * level + 2])
-        stat, err := storage.client.Stat(subDir)
-        if err == nil && stat.IsDir() {
-            dir = subDir
-            continue
-        }
-
-        if level < minimumLevel {
-            // Create the subdirectory if is doesn't exist.
-
-            if err == nil && !stat.IsDir() {
-                return "", false, 0, fmt.Errorf("The path %s is not a directory", subDir)
-            }
-
-            err = storage.client.Mkdir(subDir)
-            if err != nil {
-                // The directory may have been created by other threads so check it again.
-                stat, _ := storage.client.Stat(subDir)
-                if stat == nil || !stat.IsDir() {
-                    return "", false, 0, fmt.Errorf("Failed to create the directory %s: %v", subDir, err)
-                }
-            }
-
-            dir = subDir
-            continue
-        }
-
-        // Teh chunk must be under this subdirectory but it doesn't exist.
-        return path.Join(dir, chunkID[2 * level:])[len(storage.storageDir) + 1:] + suffix, false, 0, nil
-
-    }
-
-    LOG_FATAL("CHUNK_FIND", "Chunk %s is still not found after having searched a maximum level of directories",
-              chunkID)
-    return "", false, 0, nil
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *SFTPStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	file, err := storage.client.Open(path.Join(storage.storageDir, filePath))
@@ -245,7 +188,7 @@ func (storage *SFTPStorage) DownloadFile(threadIndex int, filePath string, chunk
 	}
 
 	defer file.Close()
-    if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit / storage.numberOfThreads); err != nil {
+	if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit/storage.numberOfThreads); err != nil {
 		return err
 	}
 
@@ -257,6 +200,30 @@ func (storage *SFTPStorage) UploadFile(threadIndex int, filePath string, content
 
 	fullPath := path.Join(storage.storageDir, filePath)
 
+	dirs := strings.Split(filePath, "/")
+	if len(dirs) > 1 {
+		fullDir := path.Dir(fullPath)
+		_, err := storage.client.Stat(fullDir)
+		if err != nil {
+			// The error may be caused by a non-existent fullDir, or a broken connection.  In either case,
+			// we just assume it is the former because there isn't a way to tell which is the case.
+			for i, _ := range dirs[1 : len(dirs)-1] {
+				subDir := path.Join(storage.storageDir, path.Join(dirs[0:i+2]...))
+				// We don't check the error; just keep going blindly but always store the last err
+				err = storage.client.Mkdir(subDir)
+			}
+
+			// If there is an error creating the dirs, we check fullDir one more time, because another thread
+			// may happen to create the same fullDir ahead of this thread
+			if err != nil {
+				_, err := storage.client.Stat(fullDir)
+				if err != nil {
+					return err
+				}
+			}
+		}
+	}
+
 	letters := "abcdefghijklmnopqrstuvwxyz"
 	suffix := make([]byte, 8)
 	for i := range suffix {
@@ -265,12 +232,12 @@ func (storage *SFTPStorage) UploadFile(threadIndex int, filePath string, content
 
 	temporaryFile := fullPath + "." + string(suffix) + ".tmp"
 
-    file, err := storage.client.OpenFile(temporaryFile, os.O_WRONLY | os.O_CREATE | os.O_TRUNC)
+	file, err := storage.client.OpenFile(temporaryFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
 	if err != nil {
 		return err
 	}
 
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
 	_, err = io.Copy(file, reader)
 	if err != nil {
 		file.Close()
@@ -294,16 +261,23 @@ func (storage *SFTPStorage) UploadFile(threadIndex int, filePath string, content
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *SFTPStorage) IsCacheNeeded () (bool) { return true }
+func (storage *SFTPStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *SFTPStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *SFTPStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *SFTPStorage) IsStrongConsistent() (bool) { return true }
+func (storage *SFTPStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *SFTPStorage) IsFastListing() (bool) { return false }
+func (storage *SFTPStorage) IsFastListing() bool {
+	for _, level := range storage.readLevels {
+		if level > 1 {
+			return false
+		}
+	}
+	return true
+}
 
 // Enable the test mode.
 func (storage *SFTPStorage) EnableTestMode() {}

src/duplicacy_shadowcopy_windows.go

@@ -5,11 +5,11 @@
 package duplicacy
 
 import (
-    "syscall"
-    "unsafe"
-    "time"
 	"os"
 	"runtime"
+	"syscall"
+	"time"
+	"unsafe"
 
 	ole "github.com/gilbertchen/go-ole"
 )
@@ -28,7 +28,7 @@ type IVSSAsyncVtbl struct {
 	queryStatus uintptr
 }
 
-func (async *IVSSAsync) VTable() * IVSSAsyncVtbl {
+func (async *IVSSAsync) VTable() *IVSSAsyncVtbl {
 	return (*IVSSAsyncVtbl)(unsafe.Pointer(async.RawVTable))
 }
 
@@ -55,7 +55,7 @@ func (async *IVSSAsync) Wait(seconds int) bool {
 		if status == VSS_S_ASYNC_FINISHED {
 			return true
 		}
-        if time.Now().Unix() - startTime > int64(seconds) {
+		if time.Now().Unix()-startTime > int64(seconds) {
 			LOG_WARN("IVSSASYNC_TIMEOUT", "IVssAsync is pending for more than %d seconds\n", seconds)
 			return false
 		}
@@ -77,7 +77,6 @@ func getIVSSAsync(unknown *ole.IUnknown, iid *ole.GUID) (async *IVSSAsync) {
 	return
 }
 
-
 //665c1d5f-c218-414d-a05d-7fef5f9d5c86
 var IID_IVSS = &ole.GUID{0x665c1d5f, 0xc218, 0x414d, [8]byte{0xa0, 0x5d, 0x7f, 0xef, 0x5f, 0x9d, 0x5c, 0x86}}
 
@@ -137,7 +136,7 @@ type IVSSVtbl struct {
 	queryRevertStatus             uintptr
 }
 
-func (vss *IVSS) VTable() * IVSSVtbl {
+func (vss *IVSS) VTable() *IVSSVtbl {
 	return (*IVSSVtbl)(unsafe.Pointer(vss.RawVTable))
 }
 
@@ -238,7 +237,7 @@ type SnapshotProperties struct {
 	Status               int
 }
 
-func (vss *IVSS) GetSnapshotProperties(snapshotSetID ole.GUID, properties *SnapshotProperties) (int) {
+func (vss *IVSS) GetSnapshotProperties(snapshotSetID ole.GUID, properties *SnapshotProperties) int {
 	var ret uintptr
 	if runtime.GOARCH == "386" {
 		address := uint(uintptr(unsafe.Pointer(&snapshotSetID)))
@@ -292,8 +291,7 @@ func (vss *IVSS) DeleteSnapshots(snapshotID ole.GUID) (int, int, ole.GUID) {
 	return int(ret), int(deleted), deletedGUID
 }
 
-
-func uint16ArrayToString(p *uint16) (string) {
+func uint16ArrayToString(p *uint16) string {
 	if p == nil {
 		return ""
 	}
@@ -418,7 +416,7 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 		return top
 	}
 
-    if !async.Wait(20) {
+	if !async.Wait(60) {
 		LOG_ERROR("VSS_GATHER", "Shadow copy creation failed: GatherWriterMetadata didn't finish properly")
 		return top
 	}
@@ -458,7 +456,7 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 		return top
 	}
 
-    if !async.Wait(20) {
+	if !async.Wait(60) {
 		LOG_ERROR("VSS_PREPARE", "Shadow copy creation failed: PrepareForBackup didn't finish properly")
 		return top
 	}
@@ -475,15 +473,13 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 		return top
 	}
 
-    if !async.Wait(60) {
+	if !async.Wait(180) {
 		LOG_ERROR("VSS_SNAPSHOT", "Shadow copy creation failed: DoSnapshotSet didn't finish properly")
 		return top
 	}
 	async.Release()
 
-
-    properties := SnapshotProperties {
-    }
+	properties := SnapshotProperties{}
 
 	ret = vssBackupComponent.GetSnapshotProperties(snapshotID, &properties)
 	if ret != 0 {
@@ -512,7 +508,7 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 	preferencePath := GetDuplicacyPreferencePath()
 	shadowLink = preferencePath + "\\shadow"
 	os.Remove(shadowLink)
-    err = os.Symlink(snapshotPath + "\\", shadowLink)
+	err = os.Symlink(snapshotPath+"\\", shadowLink)
 	if err != nil {
 		LOG_ERROR("VSS_SYMLINK", "Failed to create a symbolic link to the shadow copy just created: %v", err)
 		return top
@@ -521,5 +517,3 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
 	return shadowLink + "\\" + top[2:]
 
 }
-
-

src/duplicacy_snapshot.go

@@ -5,15 +5,15 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "time"
-    "path"
-    "strings"
-    "strconv"
-    "io/ioutil"
-    "encoding/json"
 	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+	"strconv"
+	"strings"
+	"time"
 )
 
 // Snapshot represents a backup of the repository.
@@ -47,10 +47,10 @@ type Snapshot struct {
 }
 
 // CreateEmptySnapshot creates an empty snapshot.
-func CreateEmptySnapshot (id string) (snapshto *Snapshot) {
+func CreateEmptySnapshot(id string) (snapshto *Snapshot) {
 	return &Snapshot{
-        ID : id,
-        Revision : 0,
+		ID:        id,
+		Revision:  0,
 		StartTime: time.Now().Unix(),
 	}
 }
@@ -60,8 +60,8 @@ func CreateEmptySnapshot (id string) (snapshto *Snapshot) {
 func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, skippedDirectories []string,
 	skippedFiles []string, err error) {
 
-    snapshot = &Snapshot {
-        ID : id,
+	snapshot = &Snapshot{
+		ID:        id,
 		Revision:  0,
 		StartTime: time.Now().Unix(),
 	}
@@ -76,17 +76,30 @@ func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, ski
 				continue
 			}
 
-            if pattern[0] != '+' && pattern[0] != '-' {
+			if pattern[0] == '#' {
+				continue
+			}
+
+			if IsUnspecifiedFilter(pattern) {
 				pattern = "+" + pattern
 			}
 
-            if pattern == "+" || pattern == "-" {
+			if IsEmptyFilter(pattern) {
 				continue
 			}
 
+			if strings.HasPrefix(pattern, "i:") || strings.HasPrefix(pattern, "e:") {
+				valid, err := IsValidRegex(pattern[2:])
+				if !valid || err != nil {
+					LOG_ERROR("SNAPSHOT_FILTER", "Invalid regular expression encountered for filter: \"%s\", error: %v", pattern, err)
+				}
+			}
+
 			patterns = append(patterns, pattern)
 		}
 
+		LOG_DEBUG("REGEX_DEBUG", "There are %d compiled regular expressions stored", len(RegexMap))
+
 		LOG_INFO("SNAPSHOT_FILTER", "Loaded %d include/exclude pattern(s)", len(patterns))
 
 		if IsTracing() {
@@ -109,8 +122,8 @@ func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, ski
 
 	for len(directories) > 0 {
 
-        directory := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		directory := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 		snapshot.Files = append(snapshot.Files, directory)
 		subdirectories, skipped, err := ListEntries(top, directory.Path, &snapshot.Files, patterns, snapshot.discardAttributes)
 		if err != nil {
@@ -139,9 +152,9 @@ func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, ski
 
 // This is the struct used to save/load incomplete snapshots
 type IncompleteSnapshot struct {
-    Files [] *Entry
+	Files        []*Entry
 	ChunkHashes  []string
-    ChunkLengths [] int
+	ChunkLengths []int
 }
 
 // LoadIncompleteSnapshot loads the incomplete snapshot if it exists
@@ -171,7 +184,7 @@ func LoadIncompleteSnapshot() (snapshot *Snapshot) {
 		chunkHashes = append(chunkHashes, string(hash))
 	}
 
-    snapshot = &Snapshot {
+	snapshot = &Snapshot{
 		Files:        incompleteSnapshot.Files,
 		ChunkHashes:  chunkHashes,
 		ChunkLengths: incompleteSnapshot.ChunkLengths,
@@ -197,7 +210,7 @@ func SaveIncompleteSnapshot(snapshot *Snapshot) {
 		chunkHashes = append(chunkHashes, hex.EncodeToString([]byte(chunkHash)))
 	}
 
-    incompleteSnapshot := IncompleteSnapshot {
+	incompleteSnapshot := IncompleteSnapshot{
 		Files:        files,
 		ChunkHashes:  chunkHashes,
 		ChunkLengths: snapshot.ChunkLengths,
@@ -234,14 +247,14 @@ func RemoveIncompleteSnapshot() {
 // CreateSnapshotFromDescription creates a snapshot from json decription.
 func CreateSnapshotFromDescription(description []byte) (snapshot *Snapshot, err error) {
 
-    var root map[string] interface{}
+	var root map[string]interface{}
 
 	err = json.Unmarshal(description, &root)
 	if err != nil {
 		return nil, err
 	}
 
-    snapshot = &Snapshot {}
+	snapshot = &Snapshot{}
 
 	if value, ok := root["id"]; !ok {
 		return nil, fmt.Errorf("No id is specified in the snapshot")
@@ -295,7 +308,7 @@ func CreateSnapshotFromDescription(description []byte) (snapshot *Snapshot, err
 		}
 	}
 
-    for _, sequenceType := range []string { "files", "chunks", "lengths" } {
+	for _, sequenceType := range []string{"files", "chunks", "lengths"} {
 		if value, ok := root[sequenceType]; !ok {
 			return nil, fmt.Errorf("No %s are specified in the snapshot", sequenceType)
 		} else if _, ok = value.([]interface{}); !ok {
@@ -323,7 +336,7 @@ func CreateSnapshotFromDescription(description []byte) (snapshot *Snapshot, err
 // LoadChunks construct 'ChunkHashes' from the json description.
 func (snapshot *Snapshot) LoadChunks(description []byte) (err error) {
 
-    var root [] interface {}
+	var root []interface{}
 	err = json.Unmarshal(description, &root)
 	if err != nil {
 		return err
@@ -350,7 +363,7 @@ func (snapshot *Snapshot) LoadLengths(description []byte) (err error) {
 }
 
 // MarshalJSON creates a json representation of the snapshot.
-func (snapshot *Snapshot) MarshalJSON() ([] byte, error) {
+func (snapshot *Snapshot) MarshalJSON() ([]byte, error) {
 
 	object := make(map[string]interface{})
 
@@ -373,7 +386,7 @@ func (snapshot *Snapshot) MarshalJSON() ([] byte, error) {
 }
 
 // MarshalSequence creates a json represetion for the specified chunk sequence.
-func (snapshot *Snapshot) MarshalSequence(sequenceType string) ([] byte, error) {
+func (snapshot *Snapshot) MarshalSequence(sequenceType string) ([]byte, error) {
 
 	if sequenceType == "files" {
 		return json.Marshal(snapshot.Files)
@@ -385,7 +398,7 @@ func (snapshot *Snapshot) MarshalSequence(sequenceType string) ([] byte, error)
 }
 
 // SetSequence assign a chunk sequence to the specified field.
-func (snapshot *Snapshot) SetSequence(sequenceType string, sequence [] string) {
+func (snapshot *Snapshot) SetSequence(sequenceType string, sequence []string) {
 	if sequenceType == "files" {
 		snapshot.FileSequence = sequence
 	} else if sequenceType == "chunks" {
@@ -396,7 +409,7 @@ func (snapshot *Snapshot) SetSequence(sequenceType string, sequence [] string) {
 }
 
 // encodeSequence turns a sequence of binary hashes into a sequence of hex hashes.
-func encodeSequence(sequence[] string) ([] string) {
+func encodeSequence(sequence []string) []string {
 
 	sequenceInHex := make([]string, len(sequence))
 
@@ -406,5 +419,3 @@ func encodeSequence(sequence[] string) ([] string) {
 
 	return sequenceInHex
 }
-
-

src/duplicacy_snapshotmanager.go

@@ -5,19 +5,21 @@
 package duplicacy
 
 import (
-    "io"
-    "os"
-    "fmt"
-    "sort"
 	"bytes"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math"
+	"os"
+	"path"
 	"regexp"
+	"sort"
 	"strconv"
 	"strings"
+	"text/tabwriter"
 	"time"
-    "path"
-    "io/ioutil"
-    "encoding/json"
-    "encoding/hex"
 
 	"github.com/aryann/difflib"
 )
@@ -33,7 +35,7 @@ type FossilCollection struct {
 	EndTime int64 `json:"end_time"`
 
 	// The lastest revision for each snapshot id when the fossil collection was created.
-    LastRevisions map[string] int       `json:"last_revisions"`
+	LastRevisions map[string]int `json:"last_revisions"`
 
 	// Fossils (i.e., chunks not referenced by any snapshots)
 	Fossils []string `json:"fossils"`
@@ -43,23 +45,23 @@ type FossilCollection struct {
 }
 
 // CreateFossilCollection creates an empty fossil collection
-func CreateFossilCollection(allSnapshots map[string][] *Snapshot) *FossilCollection{
+func CreateFossilCollection(allSnapshots map[string][]*Snapshot) *FossilCollection {
 
-    lastRevisions := make(map[string] int)
+	lastRevisions := make(map[string]int)
 	for id, snapshots := range allSnapshots {
-        lastRevisions[id] = snapshots[len(snapshots) - 1].Revision
+		lastRevisions[id] = snapshots[len(snapshots)-1].Revision
 	}
 
-    return &FossilCollection {
-        LastRevisions : lastRevisions,
+	return &FossilCollection{
+		LastRevisions: lastRevisions,
 	}
 }
 
 // IsDeletable determines if the previously collected fossils are safe to be permanently removed.  If so, it will
 // also returns a number of snapshots that were created during or after these fossils were being collected.
 // Therefore, some fossils may be referenced by these new snapshots and they must be resurrected.
-func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignoredIDs [] string,
-                        allSnapshots map[string][] *Snapshot) (isDeletable bool, newSnapshots []*Snapshot) {
+func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignoredIDs []string,
+	allSnapshots map[string][]*Snapshot) (isDeletable bool, newSnapshots []*Snapshot) {
 
 	hasNewSnapshot := make(map[string]bool)
 	lastSnapshotTime := make(map[string]int64)
@@ -109,7 +111,7 @@ func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignored
 
 			// If this snapshot ends before this fossil collection, then it is still possible that another snapshot
 			// might be in progress (although very unlikely).  So we only deem it deletable if that is not the case.
-            if snapshot.EndTime > collection.EndTime + int64(extraTime){
+			if snapshot.EndTime > collection.EndTime+int64(extraTime) {
 				hasNewSnapshot[hostID] = true
 				newSnapshots = append(newSnapshots, snapshot)
 				break
@@ -124,7 +126,7 @@ func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignored
 			LOG_TRACE("SNAPSHOT_NO_NEW", "No new snapshot from %s since the fossil collection step", snapshotID)
 		}
 
-        lastSnapshot := allSnapshots[snapshotID][len(allSnapshots[snapshotID]) - 1]
+		lastSnapshot := allSnapshots[snapshotID][len(allSnapshots[snapshotID])-1]
 		if lastSnapshot.EndTime > lastSnapshotTime[hostID] {
 			lastSnapshotTime[hostID] = lastSnapshot.EndTime
 		}
@@ -138,7 +140,7 @@ func (collection *FossilCollection) IsDeletable(isStrongConsistent bool, ignored
 			// snapshot id during the last 7 days.  A snapshot created at the roughly same time as this fossil
 			// collection would have finsihed already, while a snapshot currently being created does not affect
 			// this fossil collection.
-            if lastSnapshotTime[hostID] > 0 && lastSnapshotTime[hostID] < time.Now().Unix() - maxSnapshotRunningTime * secondsInDay {
+			if lastSnapshotTime[hostID] > 0 && lastSnapshotTime[hostID] < time.Now().Unix()-maxSnapshotRunningTime*secondsInDay {
 				LOG_INFO("SNAPSHOT_INACTIVE", "Ignore snapshot %s whose last revision was created %d days ago",
 					hostID, maxSnapshotRunningTime)
 				continue
@@ -172,13 +174,12 @@ type SnapshotManager struct {
 	snapshotCache *FileStorage
 
 	chunkDownloader *ChunkDownloader
-
 }
 
 // CreateSnapshotManager creates a snapshot manager
 func CreateSnapshotManager(config *Config, storage Storage) *SnapshotManager {
 
-    manager := &SnapshotManager {
+	manager := &SnapshotManager{
 		config:    config,
 		storage:   storage,
 		fileChunk: CreateChunk(config, true),
@@ -229,14 +230,14 @@ func (manager *SnapshotManager) DownloadSnapshot(snapshotID string, revision int
 // sequenceReader loads the chunks pointed to by 'sequence' one by one as needed.  This avoid loading all chunks into
 // the memory before passing them to the json unmarshaller.
 type sequenceReader struct {
-    sequence [] string
+	sequence   []string
 	buffer     *bytes.Buffer
 	index      int
-    refillFunc func(hash string) ([]byte)
+	refillFunc func(hash string) []byte
 }
 
 // Read reads a new chunk using the refill function when there is no more data in the buffer
-func (reader *sequenceReader)Read(data []byte) (n int, err error) {
+func (reader *sequenceReader) Read(data []byte) (n int, err error) {
 	if len(reader.buffer.Bytes()) == 0 {
 		if reader.index < len(reader.sequence) {
 			newData := reader.refillFunc(reader.sequence[reader.index])
@@ -268,21 +269,21 @@ func (manager *SnapshotManager) DownloadSequence(sequence []string) (content []b
 	return content
 }
 
-func (manager *SnapshotManager) DownloadSnapshotFileSequence(snapshot *Snapshot, patterns []string) bool {
+func (manager *SnapshotManager) DownloadSnapshotFileSequence(snapshot *Snapshot, patterns []string, attributesNeeded bool) bool {
 
 	manager.CreateChunkDownloader()
 
-    reader := sequenceReader {
+	reader := sequenceReader{
 		sequence: snapshot.FileSequence,
 		buffer:   new(bytes.Buffer),
-        refillFunc: func (chunkHash string) ([]byte) {
+		refillFunc: func(chunkHash string) []byte {
 			i := manager.chunkDownloader.AddChunk(chunkHash)
 			chunk := manager.chunkDownloader.WaitForChunk(i)
 			return chunk.GetBytes()
 		},
 	}
 
-    files := make([] *Entry, 0)
+	files := make([]*Entry, 0)
 	decoder := json.NewDecoder(&reader)
 
 	// read open bracket
@@ -303,7 +304,8 @@ func (manager *SnapshotManager) DownloadSnapshotFileSequence(snapshot *Snapshot,
 			return false
 		}
 
-        if len(patterns) != 0 && !MatchPath(entry.Path, patterns) {
+		// If we don't need the attributes or the file isn't included we clear the attributes to save memory
+		if !attributesNeeded || (len(patterns) != 0 && !MatchPath(entry.Path, patterns)) {
 			entry.Attributes = nil
 		}
 
@@ -313,7 +315,6 @@ func (manager *SnapshotManager) DownloadSnapshotFileSequence(snapshot *Snapshot,
 	return true
 }
 
-
 // DownloadSnapshotSequence downloads the content represented by a sequence of chunks, and then unmarshal the content
 // using the specified 'loadFunction'.  It purpose is to decode the chunk sequences representing chunk hashes or chunk lengths
 // in a snapshot.
@@ -329,7 +330,6 @@ func (manager *SnapshotManager) DownloadSnapshotSequence(snapshot *Snapshot, seq
 
 	content := manager.DownloadSequence(sequence)
 
-
 	if len(content) == 0 {
 		LOG_ERROR("SNAPSHOT_PARSE", "Failed to load %s specified in the snapshot %s at revision %d",
 			sequenceType, snapshot.ID, snapshot.Revision)
@@ -348,9 +348,9 @@ func (manager *SnapshotManager) DownloadSnapshotSequence(snapshot *Snapshot, seq
 // DownloadSnapshotContents loads all chunk sequences in a snapshot.  A snapshot, when just created, only contains
 // some metadata and theree sequence representing files, chunk hashes, and chunk lengths.  This function must be called
 // for the actual content of the snapshot to be usable.
-func (manager *SnapshotManager) DownloadSnapshotContents(snapshot *Snapshot, patterns []string) bool {
+func (manager *SnapshotManager) DownloadSnapshotContents(snapshot *Snapshot, patterns []string, attributesNeeded bool) bool {
 
-    manager.DownloadSnapshotFileSequence(snapshot, patterns)
+	manager.DownloadSnapshotFileSequence(snapshot, patterns, attributesNeeded)
 	manager.DownloadSnapshotSequence(snapshot, "chunks")
 	manager.DownloadSnapshotSequence(snapshot, "lengths")
 
@@ -365,7 +365,7 @@ func (manager *SnapshotManager) DownloadSnapshotContents(snapshot *Snapshot, pat
 }
 
 // CleanSnapshotCache removes all files not referenced by the specified 'snapshot' in the snapshot cache.
-func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, allSnapshots map[string] []*Snapshot) bool {
+func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, allSnapshots map[string][]*Snapshot) bool {
 
 	if allSnapshots == nil {
 		// If the 'fossils' directory exists then don't clean the cache as all snapshots will be needed later
@@ -407,7 +407,7 @@ func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, all
 
 	allCachedSnapshots, _ := manager.ListAllFiles(manager.snapshotCache, "snapshots/")
 	for _, snapshotFile := range allCachedSnapshots {
-        if snapshotFile[len(snapshotFile) - 1] == '/' {
+		if snapshotFile[len(snapshotFile)-1] == '/' {
 			continue
 		}
 
@@ -458,7 +458,7 @@ func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, all
 
 	allFiles, _ := manager.ListAllFiles(manager.snapshotCache, "chunks/")
 	for _, file := range allFiles {
-        if file[len(file) - 1] != '/' {
+		if file[len(file)-1] != '/' {
 			chunkID := strings.Replace(file, "/", "", -1)
 			if _, found := chunks[chunkID]; !found {
 				LOG_DEBUG("SNAPSHOT_CLEAN", "Delete chunk %s from the snapshot cache", chunkID)
@@ -476,7 +476,7 @@ func (manager *SnapshotManager) CleanSnapshotCache(latestSnapshot *Snapshot, all
 }
 
 // ListSnapshotIDs returns all snapshot ids.
-func (manager *SnapshotManager) ListSnapshotIDs() (snapshotIDs [] string, err error) {
+func (manager *SnapshotManager) ListSnapshotIDs() (snapshotIDs []string, err error) {
 
 	LOG_TRACE("SNAPSHOT_LIST_IDS", "Listing all snapshot ids")
 
@@ -486,8 +486,8 @@ func (manager *SnapshotManager) ListSnapshotIDs() (snapshotIDs [] string, err er
 	}
 
 	for _, dir := range dirs {
-        if len(dir) > 0 && dir[len(dir) - 1] == '/' {
-            snapshotIDs = append(snapshotIDs, dir[:len(dir) - 1])
+		if len(dir) > 0 && dir[len(dir)-1] == '/' {
+			snapshotIDs = append(snapshotIDs, dir[:len(dir)-1])
 		}
 	}
 
@@ -495,7 +495,7 @@ func (manager *SnapshotManager) ListSnapshotIDs() (snapshotIDs [] string, err er
 }
 
 // ListSnapshotRevisions returns the list of all revisions given a snapshot id.
-func (manager *SnapshotManager) ListSnapshotRevisions(snapshotID string) (revisions [] int, err error) {
+func (manager *SnapshotManager) ListSnapshotRevisions(snapshotID string) (revisions []int, err error) {
 
 	LOG_TRACE("SNAPSHOT_LIST_REVISIONS", "Listing revisions for snapshot %s", snapshotID)
 
@@ -517,7 +517,7 @@ func (manager *SnapshotManager) ListSnapshotRevisions(snapshotID string) (revisi
 	}
 
 	for _, file := range files {
-        if len(file) > 0 && file[len(file) - 1] != '/' {
+		if len(file) > 0 && file[len(file)-1] != '/' {
 			revision, err := strconv.Atoi(file)
 			if err == nil {
 				revisions = append(revisions, revision)
@@ -554,7 +554,7 @@ func (manager *SnapshotManager) downloadLatestSnapshot(snapshotID string) (remot
 	}
 
 	if remote != nil {
-        manager.DownloadSnapshotContents(remote, nil)
+		manager.DownloadSnapshotContents(remote, nil, false)
 	}
 
 	return remote
@@ -569,8 +569,8 @@ func (manager *SnapshotManager) ListAllFiles(storage Storage, top string) (allFi
 
 	for len(directories) > 0 {
 
-        dir := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 
 		LOG_TRACE("LIST_FILES", "Listing %s", dir)
 
@@ -586,36 +586,20 @@ func (manager *SnapshotManager) ListAllFiles(storage Storage, top string) (allFi
 		}
 
 		for i, file := range files {
-            if len(file) > 0 && file[len(file) - 1] == '/' {
-                directories = append(directories, dir + file)
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
 			} else {
 				allFiles = append(allFiles, (dir + file)[len(top):])
 				allSizes = append(allSizes, sizes[i])
 			}
 		}
-
-        if top == "chunks/" {
-            // We're listing all chunks so this is the perfect place to detect if a directory contains too many
-            // chunks. Create sub-directories if necessary
-            if len(files) > 1024 && !storage.IsFastListing() {
-                for i := 0; i < 256; i++ {
-                    subdir := dir + fmt.Sprintf("%02x\n", i)
-                    manager.storage.CreateDirectory(0, subdir)
-                }
-            }
-        } else {
-            // Remove chunk sub-directories that are empty
-            if len(files) == 0 && strings.HasPrefix(dir, "chunks/") && dir != "chunks/" {
-                storage.DeleteFile(0, dir)
-            }
-        }
 	}
 
 	return allFiles, allSizes
 }
 
 // GetSnapshotChunks returns all chunks referenced by a given snapshot.
-func (manager *SnapshotManager) GetSnapshotChunks(snapshot *Snapshot) (chunks [] string) {
+func (manager *SnapshotManager) GetSnapshotChunks(snapshot *Snapshot) (chunks []string) {
 
 	for _, chunkHash := range snapshot.FileSequence {
 		chunks = append(chunks, manager.config.GetChunkIDFromHash(chunkHash))
@@ -654,7 +638,7 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 	LOG_DEBUG("LIST_PARAMETERS", "id: %s, revisions: %v, tag: %s, showFiles: %t, showChunks: %t",
 		snapshotID, revisionsToList, tag, showFiles, showChunks)
 
-    var snapshotIDs [] string
+	var snapshotIDs []string
 	var err error
 
 	if snapshotID == "" {
@@ -664,7 +648,7 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 			return 0
 		}
 	} else {
-        snapshotIDs = []string { snapshotID }
+		snapshotIDs = []string{snapshotID}
 	}
 
 	numberOfSnapshots := 0
@@ -696,7 +680,7 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 			}
 
 			if showFiles {
-                manager.DownloadSnapshotFileSequence(snapshot, nil)
+				manager.DownloadSnapshotFileSequence(snapshot, nil, false)
 			}
 
 			if showFiles {
@@ -711,7 +695,7 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 						totalFiles++
 						totalFileSize += file.Size
 						if file.Size > maxSize {
-                            maxSize = maxSize * 10 + 9
+							maxSize = maxSize*10 + 9
 							maxSizeDigits += 1
 						}
 						if file.EndChunk > lastChunk {
@@ -728,7 +712,7 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 
 				metaChunks := len(snapshot.FileSequence) + len(snapshot.ChunkSequence) + len(snapshot.LengthSequence)
 				LOG_INFO("SNAPSHOT_STATS", "Files: %d, total size: %d, file chunks: %d, metadata chunks: %d",
-                         totalFiles, totalFileSize, lastChunk + 1, metaChunks)
+					totalFiles, totalFileSize, lastChunk+1, metaChunks)
 			}
 
 			if showChunks {
@@ -746,13 +730,13 @@ func (manager *SnapshotManager) ListSnapshots(snapshotID string, revisionsToList
 }
 
 // ListSnapshots shows the information about a snapshot.
-func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToCheck []int, tag string, showStatistics bool,
+func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToCheck []int, tag string, showStatistics bool, showTabular bool,
 	checkFiles bool, searchFossils bool, resurrect bool) bool {
 
 	LOG_DEBUG("LIST_PARAMETERS", "id: %s, revisions: %v, tag: %s, showStatistics: %t, checkFiles: %t, searchFossils: %t, resurrect: %t",
 		snapshotID, revisionsToCheck, tag, showStatistics, checkFiles, searchFossils, resurrect)
 
-    snapshotMap := make(map[string] [] *Snapshot)
+	snapshotMap := make(map[string][]*Snapshot)
 	var err error
 
 	// Stores the chunk file size for each chunk
@@ -768,7 +752,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 	allChunks, allSizes := manager.ListAllFiles(manager.storage, "chunks/")
 
 	for i, chunk := range allChunks {
-        if len(chunk) == 0 || chunk[len(chunk) - 1] == '/'{
+		if len(chunk) == 0 || chunk[len(chunk)-1] == '/' {
 			continue
 		}
 
@@ -795,7 +779,6 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 		snapshotMap[snapshotID] = nil
 	}
 
-
 	snapshotIDIndex := 0
 	for snapshotID, _ = range snapshotMap {
 
@@ -817,7 +800,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 			}
 
 			if checkFiles {
-                manager.DownloadSnapshotContents(snapshot, nil)
+				manager.DownloadSnapshotContents(snapshot, nil, false)
 				manager.VerifySnapshot(snapshot)
 				continue
 			}
@@ -835,7 +818,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 				if !found {
 					if !searchFossils {
 						missingChunks += 1
-                        LOG_WARN("SNAPHOST_VALIDATE",
+						LOG_WARN("SNAPSHOT_VALIDATE",
 							"Chunk %s referenced by snapshot %s at revision %d does not exist",
 							chunkID, snapshotID, revision)
 						continue
@@ -843,14 +826,14 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 
 					chunkPath, exist, size, err := manager.storage.FindChunk(0, chunkID, true)
 					if err != nil {
-                        LOG_ERROR("SNAPHOST_VALIDATE", "Failed to check the existence of chunk %s: %v",
+						LOG_ERROR("SNAPSHOT_VALIDATE", "Failed to check the existence of chunk %s: %v",
 							chunkID, err)
 						return false
 					}
 
 					if !exist {
 						missingChunks += 1
-                        LOG_WARN("SNAPHOST_VALIDATE",
+						LOG_WARN("SNAPSHOT_VALIDATE",
 							"Chunk %s referenced by snapshot %s at revision %d does not exist",
 							chunkID, snapshotID, revision)
 						continue
@@ -859,7 +842,7 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 					if resurrect {
 						manager.resurrectChunk(chunkPath, chunkID)
 					} else {
-                        LOG_WARN("SNAPHOST_FOSSIL", "Chunk %s referenced by snapshot %s at revision %d " +
+						LOG_WARN("SNAPSHOT_FOSSIL", "Chunk %s referenced by snapshot %s at revision %d "+
 							"has been marked as a fossil", chunkID, snapshotID, revision)
 					}
 
@@ -894,8 +877,18 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 		snapshotIDIndex += 1
 	}
 
+	if showTabular {
+		manager.ShowStatisticsTabular(snapshotMap, chunkSizeMap, chunkUniqueMap, chunkSnapshotMap)
+	} else if showStatistics {
+		manager.ShowStatistics(snapshotMap, chunkSizeMap, chunkUniqueMap, chunkSnapshotMap)
+	}
 
-    if showStatistics {
+	return true
+}
+
+// Print snapshot and revision statistics
+func (manager *SnapshotManager) ShowStatistics(snapshotMap map[string][]*Snapshot, chunkSizeMap map[string]int64, chunkUniqueMap map[string]bool,
+	chunkSnapshotMap map[string]int) {
 	for snapshotID, snapshotList := range snapshotMap {
 
 		snapshotChunks := make(map[string]bool)
@@ -940,10 +933,89 @@ func (manager *SnapshotManager) CheckSnapshots(snapshotID string, revisionsToChe
 		LOG_INFO("SNAPSHOT_CHECK", "Snapshot %s all revisions: %s total chunk bytes, %s unique chunk bytes",
 			snapshotID, PrettyNumber(totalChunkSize), PrettyNumber(uniqueChunkSize))
 	}
+}
+
+// Print snapshot and revision statistics in tabular format
+func (manager *SnapshotManager) ShowStatisticsTabular(snapshotMap map[string][]*Snapshot, chunkSizeMap map[string]int64, chunkUniqueMap map[string]bool,
+	chunkSnapshotMap map[string]int) {
+	tableBuffer := new(bytes.Buffer)
+	tableWriter := tabwriter.NewWriter(tableBuffer, 0, 0, 1, ' ', tabwriter.AlignRight|tabwriter.Debug)
+
+	for snapshotID, snapshotList := range snapshotMap {
+		fmt.Fprintln(tableWriter, "")
+		fmt.Fprintln(tableWriter, " snap \trev \t \tfiles \tbytes \tchunks \tbytes \tuniq \tbytes \tnew \tbytes \t")
+		snapshotChunks := make(map[string]bool)
+
+		earliestSeenChunks := make(map[string]int)
+
+		for _, snapshot := range snapshotList {
+			for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+				if earliestSeenChunks[chunkID] == 0 {
+					earliestSeenChunks[chunkID] = math.MaxInt32
+				}
+				earliestSeenChunks[chunkID] = MinInt(earliestSeenChunks[chunkID], snapshot.Revision)
+			}
 		}
 
-    return true
+		for _, snapshot := range snapshotList {
 
+			chunks := make(map[string]bool)
+			for _, chunkID := range manager.GetSnapshotChunks(snapshot) {
+				chunks[chunkID] = true
+				snapshotChunks[chunkID] = true
+			}
+
+			var totalChunkSize int64
+			var uniqueChunkSize int64
+			var totalChunkCount int64
+			var uniqueChunkCount int64
+			var newChunkCount int64
+			var newChunkSize int64
+
+			for chunkID, _ := range chunks {
+				chunkSize := chunkSizeMap[chunkID]
+				totalChunkSize += chunkSize
+				totalChunkCount += 1
+				if earliestSeenChunks[chunkID] == snapshot.Revision {
+					newChunkCount += 1
+					newChunkSize += chunkSize
+				}
+				if chunkUniqueMap[chunkID] {
+					uniqueChunkSize += chunkSize
+					uniqueChunkCount += 1
+				}
+			}
+
+			files := " \t "
+			if snapshot.FileSize != 0 && snapshot.NumberOfFiles != 0 {
+				files = fmt.Sprintf("%d \t%s", snapshot.NumberOfFiles, PrettyNumber(snapshot.FileSize))
+			}
+			creationTime := time.Unix(snapshot.StartTime, 0).Format("2006-01-02 15:04")
+			fmt.Fprintln(tableWriter, fmt.Sprintf(
+				"%s \t%d \t@ %s %5s \t%s \t%d \t%s \t%d \t%s \t%d \t%s \t",
+				snapshotID, snapshot.Revision, creationTime, snapshot.Options, files, totalChunkCount, PrettyNumber(totalChunkSize), uniqueChunkCount, PrettyNumber(uniqueChunkSize), newChunkCount, PrettyNumber(newChunkSize)))
+		}
+
+		var totalChunkSize int64
+		var uniqueChunkSize int64
+		var totalChunkCount int64
+		var uniqueChunkCount int64
+		for chunkID, _ := range snapshotChunks {
+			chunkSize := chunkSizeMap[chunkID]
+			totalChunkSize += chunkSize
+			totalChunkCount += 1
+
+			if chunkSnapshotMap[chunkID] != -1 {
+				uniqueChunkSize += chunkSize
+				uniqueChunkCount += 1
+			}
+		}
+		fmt.Fprintln(tableWriter, fmt.Sprintf(
+			"%s \tall \t \t \t \t%d \t%s \t%d \t%s \t \t \t",
+			snapshotID, totalChunkCount, PrettyNumber(totalChunkSize), uniqueChunkCount, PrettyNumber(uniqueChunkSize)))
+	}
+	tableWriter.Flush()
+	LOG_INFO("SNAPSHOT_CHECK", tableBuffer.String())
 }
 
 // ConvertSequence converts a sequence of chunk hashes into a sequence of chunk ids.
@@ -1006,7 +1078,7 @@ func (manager *SnapshotManager) VerifySnapshot(snapshot *Snapshot) bool {
 		return false
 	}
 
-    files := make([]*Entry, 0, len(snapshot.Files) / 2)
+	files := make([]*Entry, 0, len(snapshot.Files)/2)
 	for _, file := range snapshot.Files {
 		if file.IsFile() && file.Size != 0 {
 			files = append(files, file)
@@ -1016,7 +1088,7 @@ func (manager *SnapshotManager) VerifySnapshot(snapshot *Snapshot) bool {
 	sort.Sort(ByChunk(files))
 	corruptedFiles := 0
 	for _, file := range files {
-        if !manager.RetrieveFile(snapshot, file, func([]byte) {} ) {
+		if !manager.RetrieveFile(snapshot, file, func([]byte) {}) {
 			corruptedFiles++
 		}
 		LOG_TRACE("SNAPSHOT_VERIFY", "%s", file.Path)
@@ -1034,7 +1106,7 @@ func (manager *SnapshotManager) VerifySnapshot(snapshot *Snapshot) bool {
 }
 
 // RetrieveFile retrieve the file in the specifed snapshot.
-func (manager *SnapshotManager) RetrieveFile(snapshot *Snapshot, file *Entry, output func([]byte)()) bool {
+func (manager *SnapshotManager) RetrieveFile(snapshot *Snapshot, file *Entry, output func([]byte)) bool {
 
 	if file.Size == 0 {
 		return true
@@ -1042,6 +1114,14 @@ func (manager *SnapshotManager) RetrieveFile(snapshot *Snapshot, file *Entry, ou
 
 	manager.CreateChunkDownloader()
 
+	// Temporarily disable the snapshot cache of the download so that downloaded file chunks won't be saved
+	// to the cache.
+	snapshotCache := manager.chunkDownloader.snapshotCache
+	manager.chunkDownloader.snapshotCache = nil
+	defer func() {
+		manager.chunkDownloader.snapshotCache = snapshotCache
+	}()
+
 	fileHasher := manager.config.NewFileHasher()
 	alternateHash := false
 	if strings.HasPrefix(file.Hash, "#") {
@@ -1088,7 +1168,7 @@ func (manager *SnapshotManager) RetrieveFile(snapshot *Snapshot, file *Entry, ou
 }
 
 // FindFile returns the file entry that has the given file name.
-func (manager *SnapshotManager) FindFile(snapshot *Snapshot, filePath string, suppressError bool) (*Entry) {
+func (manager *SnapshotManager) FindFile(snapshot *Snapshot, filePath string, suppressError bool) *Entry {
 	for _, entry := range snapshot.Files {
 		if entry.Path == filePath {
 			return entry
@@ -1129,7 +1209,8 @@ func (manager *SnapshotManager) PrintFile(snapshotID string, revision int, path
 		patterns = []string{path}
 	}
 
-    if !manager.DownloadSnapshotContents(snapshot, patterns) {
+	// If no path is specified, we're printing the snapshot so we need all attributes
+	if !manager.DownloadSnapshotContents(snapshot, patterns, path == "") {
 		return false
 	}
 
@@ -1139,14 +1220,14 @@ func (manager *SnapshotManager) PrintFile(snapshotID string, revision int, path
 	}
 
 	file := manager.FindFile(snapshot, path, false)
-    var content [] byte
+	var content []byte
 	if !manager.RetrieveFile(snapshot, file, func(chunk []byte) { content = append(content, chunk...) }) {
 		LOG_ERROR("SNAPSHOT_RETRIEVE", "File %s is corrupted in snapshot %s at revision %d",
 			path, snapshot.ID, snapshot.Revision)
 		return false
 	}
 
-    fmt.Printf("%s\n", string(content))
+	fmt.Printf("%s", string(content))
 
 	return true
 }
@@ -1187,12 +1268,11 @@ func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []
 		leftSnapshot = manager.DownloadSnapshot(snapshotID, revisions[0])
 	}
 
-
 	if len(filePath) > 0 {
 
-        manager.DownloadSnapshotContents(leftSnapshot, nil)
+		manager.DownloadSnapshotContents(leftSnapshot, nil, false)
 		if rightSnapshot != nil && rightSnapshot.Revision != 0 {
-            manager.DownloadSnapshotContents(rightSnapshot, nil)
+			manager.DownloadSnapshotContents(rightSnapshot, nil, false)
 		}
 
 		var leftFile []byte
@@ -1227,7 +1307,7 @@ func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []
 
 		after := 10
 		before := 10
-        var buffer [] string
+		var buffer []string
 		on := false
 		distance := 0
 
@@ -1268,9 +1348,9 @@ func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []
 	}
 
 	// We only need to decode the 'files' sequence, not 'chunkhashes' or 'chunklengthes'
-    manager.DownloadSnapshotFileSequence(leftSnapshot, nil)
+	manager.DownloadSnapshotFileSequence(leftSnapshot, nil, false)
 	if rightSnapshot != nil && rightSnapshot.Revision != 0 {
-        manager.DownloadSnapshotFileSequence(rightSnapshot, nil)
+		manager.DownloadSnapshotFileSequence(rightSnapshot, nil, false)
 	}
 
 	maxSize := int64(9)
@@ -1279,19 +1359,19 @@ func (manager *SnapshotManager) Diff(top string, snapshotID string, revisions []
 	// Find the max Size value in order for pretty alignment.
 	for _, file := range leftSnapshot.Files {
 		for !file.IsDir() && file.Size > maxSize {
-            maxSize = maxSize * 10 + 9
+			maxSize = maxSize*10 + 9
 			maxSizeDigits += 1
 		}
 	}
 
 	for _, file := range rightSnapshot.Files {
 		for !file.IsDir() && file.Size > maxSize {
-            maxSize = maxSize * 10 + 9
+			maxSize = maxSize*10 + 9
 			maxSizeDigits += 1
 		}
 	}
 
-    buffer := make([]byte, 32 * 1024)
+	buffer := make([]byte, 32*1024)
 
 	var i, j int
 	for i < len(leftSnapshot.Files) || j < len(rightSnapshot.Files) {
@@ -1374,7 +1454,7 @@ func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revis
 	sort.Ints(revisions)
 	for _, revision := range revisions {
 		snapshot := manager.DownloadSnapshot(snapshotID, revision)
-        manager.DownloadSnapshotFileSequence(snapshot, nil)
+		manager.DownloadSnapshotFileSequence(snapshot, nil, false)
 		file := manager.FindFile(snapshot, filePath, true)
 
 		if file != nil {
@@ -1393,7 +1473,6 @@ func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revis
 			LOG_INFO("SNAPSHOT_HISTORY", "%7d:", revision)
 		}
 
-
 	}
 
 	stat, err := os.Stat(joinPath(top, filePath))
@@ -1404,7 +1483,7 @@ func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revis
 			modifiedFlag = "*"
 		}
 		if showLocalHash {
-            localFile.Hash = manager.config.ComputeFileHash(joinPath(top, filePath), make([]byte, 32 * 1024))
+			localFile.Hash = manager.config.ComputeFileHash(joinPath(top, filePath), make([]byte, 32*1024))
 			if lastVersion.Hash != localFile.Hash {
 				modifiedFlag = "*"
 			}
@@ -1419,7 +1498,7 @@ func (manager *SnapshotManager) ShowHistory(top string, snapshotID string, revis
 
 // fossilizeChunk turns the chunk into a fossil.
 func (manager *SnapshotManager) fossilizeChunk(chunkID string, filePath string,
-                                               exclusive bool, collection *FossilCollection) (bool) {
+	exclusive bool, collection *FossilCollection) bool {
 	if exclusive {
 		err := manager.storage.DeleteFile(0, filePath)
 		if err != nil {
@@ -1455,7 +1534,7 @@ func (manager *SnapshotManager) fossilizeChunk(chunkID string, filePath string,
 }
 
 // resurrectChunk turns the fossil back into a chunk
-func (manager *SnapshotManager) resurrectChunk(fossilPath string, chunkID string) (bool) {
+func (manager *SnapshotManager) resurrectChunk(fossilPath string, chunkID string) bool {
 	chunkPath, exist, _, err := manager.storage.FindChunk(0, chunkID, false)
 	if err != nil {
 		LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", chunkID, err)
@@ -1478,8 +1557,6 @@ func (manager *SnapshotManager) resurrectChunk(fossilPath string, chunkID string
 	return true
 }
 
-
-
 // PruneSnapshots deletes snapshots by revisions, tags, or a retention policy.  The main idea is two-step
 // fossil collection.
 // 1. Delete snapshots specified by revision, retention policy, with a tag.  Find any resulting unreferenced
@@ -1501,7 +1578,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 	dryRun bool, deleteOnly bool, collectOnly bool) bool {
 
 	LOG_DEBUG("DELETE_PARAMETERS",
-              "id: %s, revisions: %v, tags: %v, retentions: %v, exhaustive: %t, exclusive: %t, " +
+		"id: %s, revisions: %v, tags: %v, retentions: %v, exhaustive: %t, exclusive: %t, "+
 			"dryrun: %t, deleteOnly: %t, collectOnly: %t",
 		snapshotID, revisionsToBeDeleted, tags, retentions,
 		exhaustive, exclusive, dryRun, deleteOnly, collectOnly)
@@ -1514,13 +1591,13 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 	logDir := path.Join(preferencePath, "logs")
 	os.Mkdir(logDir, 0700)
 	logFileName := path.Join(logDir, time.Now().Format("prune-log-20060102-150405"))
-    logFile, err := os.OpenFile(logFileName, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0600)
+	logFile, err := os.OpenFile(logFileName, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600)
 
 	defer func() {
 		if logFile != nil {
 			logFile.Close()
 		}
-    } ()
+	}()
 
 	// A retention policy is specified in the form 'interval:age', where both 'interval' and 'age' are numbers of
 	// days. A retention policy applies to a snapshot if the snapshot is older than the age.  For snapshots older
@@ -1536,7 +1613,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		Age      int
 		Interval int
 	}
-    var retentionPolicies [] RetentionPolicy
+	var retentionPolicies []RetentionPolicy
 
 	// Parse the retention policy if needed.
 	if len(revisionsToBeDeleted) == 0 && len(retentions) > 0 {
@@ -1561,9 +1638,9 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 				return false
 			}
 
-            policy :=  RetentionPolicy {
-                Age : age,
-                Interval : interval,
+			policy := RetentionPolicy{
+				Age:      age,
+				Interval: interval,
 			}
 
 			retentionPolicies = append(retentionPolicies, policy)
@@ -1575,7 +1652,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		}
 
 		for i, policy := range retentionPolicies {
-            if i == 0 || policy.Age < retentionPolicies[i - 1].Age {
+			if i == 0 || policy.Age < retentionPolicies[i-1].Age {
 				if policy.Interval == 0 {
 					LOG_INFO("RETENTION_POLICY", "Keep no snapshots older than %d days", policy.Age)
 				} else {
@@ -1586,7 +1663,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		}
 	}
 
-    allSnapshots := make(map[string] [] *Snapshot)
+	allSnapshots := make(map[string][]*Snapshot)
 
 	// We must find all snapshots for all ids even if only one snapshot is specified to be deleted,
 	// because we need to find out which chunks are not referenced.
@@ -1604,7 +1681,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		}
 
 		sort.Ints(revisions)
-        var snapshots [] *Snapshot
+		var snapshots []*Snapshot
 		for _, revision := range revisions {
 			snapshot := manager.DownloadSnapshot(id, revision)
 			if snapshot != nil {
@@ -1629,7 +1706,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 
 	referencedFossils := make(map[string]bool)
 
-    // Find fossil collections previsouly created, and delete fossils and temporary files in them if they are
+	// Find fossil collections previously created, and delete fossils and temporary files in them if they are
 	// deletable.
 	for _, collectionName := range collections {
 
@@ -1638,7 +1715,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		}
 
 		matched := collectionRegex.FindStringSubmatch(collectionName)
-        if matched == nil{
+		if matched == nil {
 			continue
 		}
 
@@ -1764,6 +1841,8 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 			// If revisions are specified ignore tags and the retention policy.
 			for _, snapshot := range snapshots {
 				if _, found := revisionMap[snapshot.Revision]; found {
+					LOG_DEBUG("SNAPSHOT_DELETE", "Snapshot %s at revision %d to be deleted - specified in command",
+						snapshot.ID, snapshot.Revision)
 					snapshot.Flag = true
 					toBeDeleted++
 				}
@@ -1781,19 +1860,19 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 			i := 0
 			for j, snapshot := range snapshots {
 
-                if !exclusive && j == len(snapshots) - 1 {
+				if !exclusive && j == len(snapshots)-1 {
 					continue
 				}
 
 				if len(tagMap) > 0 {
-                    if _, found := tagMap[snapshot.Tag]; found {
+					if _, found := tagMap[snapshot.Tag]; !found {
 						continue
 					}
 				}
 
 				// Find out which retent policy applies based on the age.
 				for i < len(retentionPolicies) &&
-                  int(now - snapshot.StartTime) <retentionPolicies[i].Age * secondsInDay {
+					int(now-snapshot.StartTime) < retentionPolicies[i].Age*secondsInDay {
 					i++
 					lastSnapshotTime = 0
 				}
@@ -1801,12 +1880,16 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 				if i < len(retentionPolicies) {
 					if retentionPolicies[i].Interval == 0 {
 						// No snapshots to keep if interval is 0
+						LOG_DEBUG("SNAPSHOT_DELETE", "Snapshot %s at revision %d to be deleted - older than %d days",
+							snapshot.ID, snapshot.Revision, retentionPolicies[i].Age)
 						snapshot.Flag = true
 						toBeDeleted++
 					} else if lastSnapshotTime != 0 &&
-                      int(snapshot.StartTime - lastSnapshotTime) < retentionPolicies[i].Interval * secondsInDay - 600 {
+						int(snapshot.StartTime-lastSnapshotTime) < retentionPolicies[i].Interval*secondsInDay-600 {
 						// Delete the snapshot if it is too close to the last kept one.  Note that a tolerance of 10
 						// minutes was subtracted from the interval.
+						LOG_DEBUG("SNAPSHOT_DELETE", "Snapshot %s at revision %d to be deleted - older than %d days, less than %d days from previous",
+							snapshot.ID, snapshot.Revision, retentionPolicies[i].Age, retentionPolicies[i].Interval)
 						snapshot.Flag = true
 						toBeDeleted++
 					} else {
@@ -1842,7 +1925,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 	for _, snapshots := range allSnapshots {
 
 		if len(snapshots) > 0 {
-            latest := snapshots[len(snapshots) - 1]
+			latest := snapshots[len(snapshots)-1]
 			if latest.Flag && !exclusive {
 				LOG_ERROR("SNAPSHOT_DELETE",
 					"The latest snapshot %s at revision %d can't be deleted in non-exclusive mode",
@@ -1873,7 +1956,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 		// In exhaustive, we scan the entire chunk tree to find dangling chunks and temporaries.
 		allFiles, _ := manager.ListAllFiles(manager.storage, chunkDir)
 		for _, file := range allFiles {
-            if file[len(file) - 1] == '/' {
+			if file[len(file)-1] == '/' {
 				continue
 			}
 
@@ -1888,7 +1971,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 
 				if exclusive {
 					// In exclusive mode, we assume no other restore operation is running concurrently.
-                    err := manager.storage.DeleteFile(0, chunkDir + file)
+					err := manager.storage.DeleteFile(0, chunkDir+file)
 					if err != nil {
 						LOG_ERROR("CHUNK_TEMPORARY", "Failed to remove the temporary file %s: %v", file, err)
 						return false
@@ -1913,9 +1996,9 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 					chunk = strings.Replace(chunk, ".fsl", "", -1)
 
 					if _, found := referencedChunks[chunk]; found {
-                        manager.resurrectChunk(chunkDir + file, chunk)
+						manager.resurrectChunk(chunkDir+file, chunk)
 					} else {
-                        err := manager.storage.DeleteFile(0, chunkDir + file)
+						err := manager.storage.DeleteFile(0, chunkDir+file)
 						if err != nil {
 							LOG_WARN("FOSSIL_DELETE", "Failed to remove the unreferenced fossil %s: %v", file, err)
 						} else {
@@ -1942,7 +2025,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 					continue
 				}
 
-                manager.fossilizeChunk(chunk, chunkDir + file, exclusive, collection)
+				manager.fossilizeChunk(chunk, chunkDir+file, exclusive, collection)
 				if exclusive {
 					fmt.Fprintf(logFile, "Deleted chunk %s (exclusive mode)\n", chunk)
 				} else {
@@ -1960,7 +2043,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 				}
 
 				// This is a redundant chunk file (for instance D3/495A8D and D3/49/5A8D )
-                err := manager.storage.DeleteFile(0, chunkDir + file)
+				err := manager.storage.DeleteFile(0, chunkDir+file)
 				if err != nil {
 					LOG_WARN("CHUNK_DELETE", "Failed to remove the redundant chunk file %s: %v", file, err)
 				} else {
@@ -2020,7 +2103,6 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 			}
 		}
 
-
 	}
 
 	// Save the fossil collection if it is not empty.
@@ -2065,7 +2147,7 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 					snapshot.ID, snapshot.Revision)
 			}
 			manager.snapshotCache.DeleteFile(0, snapshotPath)
-            fmt.Fprintf(logFile, "Deleted snapshot %s at revision %d\n", snapshot.ID, snapshot.Revision)
+			fmt.Fprintf(logFile, "Deleted cached snapshot %s at revision %d\n", snapshot.ID, snapshot.Revision)
 		}
 	}
 
@@ -2074,17 +2156,20 @@ func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string,
 			"No fossil collection has been created since deleted snapshots did not reference any unique chunks")
 	}
 
-    var latestSnapshot *Snapshot
+	var latestSnapshot *Snapshot = nil
 	if len(allSnapshots[selfID]) > 0 {
-        latestSnapshot = allSnapshots[selfID][len(allSnapshots[selfID]) - 1]
+		latestSnapshot = allSnapshots[selfID][len(allSnapshots[selfID])-1]
 	}
 
+	if latestSnapshot != nil && !latestSnapshot.Flag {
 		manager.CleanSnapshotCache(latestSnapshot, allSnapshots)
+	} else {
+		manager.CleanSnapshotCache(nil, allSnapshots)
+	}
 
 	return true
 }
 
-
 // CheckSnapshot performs sanity checks on the given snapshot.
 func (manager *SnapshotManager) CheckSnapshot(snapshot *Snapshot) (err error) {
 
@@ -2136,7 +2221,7 @@ func (manager *SnapshotManager) CheckSnapshot(snapshot *Snapshot) (err error) {
 					entry.Path, entry.StartChunk, lastChunk)
 			}
 
-            if entry.StartChunk > lastChunk + 1 {
+			if entry.StartChunk > lastChunk+1 {
 				return fmt.Errorf("The file %s starts at chunk %d while the last chunk is %d",
 					entry.Path, entry.StartChunk, lastChunk)
 			}
@@ -2178,11 +2263,11 @@ func (manager *SnapshotManager) CheckSnapshot(snapshot *Snapshot) (err error) {
 	}
 
 	if len(entries) > 0 && entries[0].StartChunk != 0 {
-        return fmt.Errorf("The first file starts at chunk %d", entries[0].StartChunk )
+		return fmt.Errorf("The first file starts at chunk %d", entries[0].StartChunk)
 	}
 
 	// There may be a last chunk whose size is 0 so we allow this to happen
-    if lastChunk < numberOfChunks - 2 {
+	if lastChunk < numberOfChunks-2 {
 		return fmt.Errorf("The last file ends at chunk %d but the number of chunks is %d", lastChunk, numberOfChunks)
 	}
 
@@ -2209,6 +2294,10 @@ func (manager *SnapshotManager) DownloadFile(path string, derivationKey string)
 		return nil
 	}
 
+	if len(derivationKey) > 64 {
+		derivationKey = derivationKey[len(derivationKey) - 64:]
+	}
+
 	err = manager.fileChunk.Decrypt(manager.config.FileKey, derivationKey)
 	if err != nil {
 		LOG_ERROR("DOWNLOAD_DECRYPT", "Failed to decrypt the file %s: %v", path, err)
@@ -2239,6 +2328,10 @@ func (manager *SnapshotManager) UploadFile(path string, derivationKey string, co
 		}
 	}
 
+	if len(derivationKey) > 64 {
+		derivationKey = derivationKey[len(derivationKey) - 64:]
+	}
+
 	err := manager.fileChunk.Encrypt(manager.config.FileKey, derivationKey)
 	if err != nil {
 		LOG_ERROR("UPLOAD_File", "Failed to encrypt the file %s: %v", path, err)

src/duplicacy_snapshotmanager_test.go

@@ -5,19 +5,19 @@
 package duplicacy
 
 import (
-    "testing"
-    "os"
+	"crypto/rand"
+	"encoding/hex"
+	"encoding/json"
 	"fmt"
-    "time"
+	"os"
 	"path"
 	"strings"
-    "crypto/rand"
-    "encoding/json"
-    "encoding/hex"
+	"testing"
+	"time"
 )
 
-func createDummySnapshot(snapshotID string, revision int, endTime int64) * Snapshot {
-    return &Snapshot {
+func createDummySnapshot(snapshotID string, revision int, endTime int64) *Snapshot {
+	return &Snapshot{
 		ID:       snapshotID,
 		Revision: revision,
 		EndTime:  endTime,
@@ -31,15 +31,15 @@ func TestIsDeletable(t *testing.T) {
 	now := time.Now().Unix()
 	day := int64(3600 * 24)
 
-    allSnapshots := make(map[string][] *Snapshot)
-    allSnapshots["host1"] = append([]*Snapshot{}, createDummySnapshot("host1", 1, now - 2 * day))
-    allSnapshots["host2"] = append([]*Snapshot{}, createDummySnapshot("host2", 1, now - 2 * day))
-    allSnapshots["host1"] = append(allSnapshots["host1"], createDummySnapshot("host1", 2, now - 1 * day))
-    allSnapshots["host2"] = append(allSnapshots["host2"], createDummySnapshot("host2", 2, now - 1 * day))
+	allSnapshots := make(map[string][]*Snapshot)
+	allSnapshots["host1"] = append([]*Snapshot{}, createDummySnapshot("host1", 1, now-2*day))
+	allSnapshots["host2"] = append([]*Snapshot{}, createDummySnapshot("host2", 1, now-2*day))
+	allSnapshots["host1"] = append(allSnapshots["host1"], createDummySnapshot("host1", 2, now-1*day))
+	allSnapshots["host2"] = append(allSnapshots["host2"], createDummySnapshot("host2", 2, now-1*day))
 
-    collection := & FossilCollection {
+	collection := &FossilCollection{
 		EndTime:       now - day - 3600,
-        LastRevisions: make(map[string] int),
+		LastRevisions: make(map[string]int),
 	}
 
 	collection.LastRevisions["host1"] = 1
@@ -51,21 +51,21 @@ func TestIsDeletable(t *testing.T) {
 	}
 
 	collection.LastRevisions["host3"] = 1
-    allSnapshots["host3"] = append([]*Snapshot{}, createDummySnapshot("host3", 1, now - 2 * day))
+	allSnapshots["host3"] = append([]*Snapshot{}, createDummySnapshot("host3", 1, now-2*day))
 
 	isDeletable, newSnapshots = collection.IsDeletable(true, nil, allSnapshots)
 	if isDeletable {
 		t.Errorf("Scenario 2: should not be deletable")
 	}
 
-    allSnapshots["host3"] = append(allSnapshots["host3"], createDummySnapshot("host3", 2, now - day))
+	allSnapshots["host3"] = append(allSnapshots["host3"], createDummySnapshot("host3", 2, now-day))
 	isDeletable, newSnapshots = collection.IsDeletable(true, nil, allSnapshots)
 	if !isDeletable || len(newSnapshots) != 3 {
 		t.Errorf("Scenario 3: should be deletable, 3 new snapshots")
 	}
 
 	collection.LastRevisions["host4"] = 1
-    allSnapshots["host4"] = append([]*Snapshot{}, createDummySnapshot("host4", 1, now - 8 * day))
+	allSnapshots["host4"] = append([]*Snapshot{}, createDummySnapshot("host4", 1, now-8*day))
 
 	isDeletable, newSnapshots = collection.IsDeletable(true, nil, allSnapshots)
 	if !isDeletable || len(newSnapshots) != 3 {
@@ -73,17 +73,17 @@ func TestIsDeletable(t *testing.T) {
 	}
 
 	collection.LastRevisions["repository1@host5"] = 1
-    allSnapshots["repository1@host5"] = append([]*Snapshot{}, createDummySnapshot("repository1@host5", 1, now - 3 * day))
+	allSnapshots["repository1@host5"] = append([]*Snapshot{}, createDummySnapshot("repository1@host5", 1, now-3*day))
 
 	collection.LastRevisions["repository2@host5"] = 1
-    allSnapshots["repository2@host5"] = append([]*Snapshot{}, createDummySnapshot("repository2@host5", 1, now - 2 * day))
+	allSnapshots["repository2@host5"] = append([]*Snapshot{}, createDummySnapshot("repository2@host5", 1, now-2*day))
 
 	isDeletable, newSnapshots = collection.IsDeletable(true, nil, allSnapshots)
 	if isDeletable {
 		t.Errorf("Scenario 5: should not be deletable")
 	}
 
-    allSnapshots["repository1@host5"] = append(allSnapshots["repository1@host5"], createDummySnapshot("repository1@host5", 2, now - day))
+	allSnapshots["repository1@host5"] = append(allSnapshots["repository1@host5"], createDummySnapshot("repository1@host5", 2, now-day))
 	isDeletable, newSnapshots = collection.IsDeletable(true, nil, allSnapshots)
 	if !isDeletable || len(newSnapshots) != 4 {
 		t.Errorf("Scenario 6: should be deletable, 4 new snapshots")
@@ -95,18 +95,21 @@ func createTestSnapshotManager(testDir string) *SnapshotManager {
 	os.RemoveAll(testDir)
 	os.MkdirAll(testDir, 0700)
 
-    storage, _ := CreateFileStorage(testDir, 2, false, 1)
+	storage, _ := CreateFileStorage(testDir, false, 1)
 	storage.CreateDirectory(0, "chunks")
 	storage.CreateDirectory(0, "snapshots")
 	config := CreateConfig()
 	snapshotManager := CreateSnapshotManager(config, storage)
 
 	cacheDir := path.Join(testDir, "cache")
-    snapshotCache, _ := CreateFileStorage(cacheDir, 2, false, 1)
+	snapshotCache, _ := CreateFileStorage(cacheDir, false, 1)
 	snapshotCache.CreateDirectory(0, "chunks")
 	snapshotCache.CreateDirectory(0, "snapshots")
 
 	snapshotManager.snapshotCache = snapshotCache
+
+	SetDuplicacyPreferencePath(testDir + "/.duplicacy")
+
 	return snapshotManager
 }
 
@@ -140,39 +143,40 @@ func uploadRandomChunk(manager *SnapshotManager, chunkSize int) string {
 	return uploadTestChunk(manager, content)
 }
 
-func createTestSnapshot(manager *SnapshotManager, snapshotID string, revision int, startTime int64, endTime int64, chunkHashes []string) {
+func createTestSnapshot(manager *SnapshotManager, snapshotID string, revision int, startTime int64, endTime int64, chunkHashes []string, tag string) {
 
-    snapshot :=  &Snapshot {
+	snapshot := &Snapshot{
 		ID:          snapshotID,
 		Revision:    revision,
 		StartTime:   startTime,
 		EndTime:     endTime,
 		ChunkHashes: chunkHashes,
+		Tag:         tag,
 	}
 
-    var chunkHashesInHex [] string
+	var chunkHashesInHex []string
 	for _, chunkHash := range chunkHashes {
 		chunkHashesInHex = append(chunkHashesInHex, hex.EncodeToString([]byte(chunkHash)))
 	}
 
 	sequence, _ := json.Marshal(chunkHashesInHex)
-    snapshot.ChunkSequence = []string { uploadTestChunk(manager, sequence) }
+	snapshot.ChunkSequence = []string{uploadTestChunk(manager, sequence)}
 
 	description, _ := snapshot.MarshalJSON()
 	path := fmt.Sprintf("snapshots/%s/%d", snapshotID, snapshot.Revision)
-    manager.storage.CreateDirectory(0, "snapshots/" + snapshotID)
+	manager.storage.CreateDirectory(0, "snapshots/"+snapshotID)
 	manager.UploadFile(path, path, description)
 }
 
 func checkTestSnapshots(manager *SnapshotManager, expectedSnapshots int, expectedFossils int) {
 
-    var snapshotIDs [] string
+	var snapshotIDs []string
 	var err error
 
 	chunks := make(map[string]bool)
 	files, _ := manager.ListAllFiles(manager.storage, "chunks/")
 	for _, file := range files {
-        if file[len(file) - 1] == '/' {
+		if file[len(file)-1] == '/' {
 			continue
 		}
 		chunk := strings.Replace(file, "/", "", -1)
@@ -239,12 +243,12 @@ func TestSingleRepositoryPrune(t *testing.T) {
 	now := time.Now().Unix()
 	day := int64(24 * 3600)
 	t.Logf("Creating 1 snapshot")
-    createTestSnapshot(snapshotManager, "repository1", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash1, chunkHash2 })
+	createTestSnapshot(snapshotManager, "repository1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
 	checkTestSnapshots(snapshotManager, 1, 2)
 
 	t.Logf("Creating 2 snapshots")
-    createTestSnapshot(snapshotManager, "repository1", 2, now - 2 * day - 3600, now - 2 * day - 60, []string { chunkHash2, chunkHash3 })
-    createTestSnapshot(snapshotManager, "repository1", 3, now - 1 * day - 3600, now - 1 * day - 60, []string { chunkHash3, chunkHash4 })
+	createTestSnapshot(snapshotManager, "repository1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	createTestSnapshot(snapshotManager, "repository1", 3, now-1*day-3600, now-1*day-60, []string{chunkHash3, chunkHash4}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 0)
 
 	t.Logf("Removing snapshot repository1 revision 1 with --exclusive")
@@ -257,7 +261,7 @@ func TestSingleRepositoryPrune(t *testing.T) {
 
 	t.Logf("Creating 1 snapshot")
 	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
-    createTestSnapshot(snapshotManager, "repository1", 4, now + 1 * day - 3600 , now + 1 * day, []string { chunkHash4, chunkHash5 })
+	createTestSnapshot(snapshotManager, "repository1", 4, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash5}, "tag")
 	checkTestSnapshots(snapshotManager, 2, 2)
 
 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
@@ -282,9 +286,9 @@ func TestSingleHostPrune(t *testing.T) {
 	now := time.Now().Unix()
 	day := int64(24 * 3600)
 	t.Logf("Creating 3 snapshots")
-    createTestSnapshot(snapshotManager, "vm1@host1", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash1, chunkHash2 })
-    createTestSnapshot(snapshotManager, "vm1@host1", 2, now - 2 * day - 3600, now - 2 * day - 60, []string { chunkHash2, chunkHash3 })
-    createTestSnapshot(snapshotManager, "vm2@host1", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash3, chunkHash4 })
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	createTestSnapshot(snapshotManager, "vm2@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash3, chunkHash4}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 0)
 
 	t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
@@ -297,7 +301,7 @@ func TestSingleHostPrune(t *testing.T) {
 
 	t.Logf("Creating 1 snapshot")
 	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
-    createTestSnapshot(snapshotManager, "vm2@host1", 2, now + 1 * day - 3600 , now + 1 * day, []string { chunkHash4, chunkHash5 })
+	createTestSnapshot(snapshotManager, "vm2@host1", 2, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash5}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 2)
 
 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
@@ -323,9 +327,9 @@ func TestMultipleHostPrune(t *testing.T) {
 	now := time.Now().Unix()
 	day := int64(24 * 3600)
 	t.Logf("Creating 3 snapshot")
-    createTestSnapshot(snapshotManager, "vm1@host1", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash1, chunkHash2 })
-    createTestSnapshot(snapshotManager, "vm1@host1", 2, now - 2 * day - 3600, now - 2 * day - 60, []string { chunkHash2, chunkHash3 })
-    createTestSnapshot(snapshotManager, "vm2@host2", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash3, chunkHash4 })
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
+	createTestSnapshot(snapshotManager, "vm2@host2", 1, now-3*day-3600, now-3*day-60, []string{chunkHash3, chunkHash4}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 0)
 
 	t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
@@ -338,7 +342,7 @@ func TestMultipleHostPrune(t *testing.T) {
 
 	t.Logf("Creating 1 snapshot")
 	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
-    createTestSnapshot(snapshotManager, "vm2@host2", 2, now + 1 * day - 3600 , now + 1 * day, []string {chunkHash4, chunkHash5})
+	createTestSnapshot(snapshotManager, "vm2@host2", 2, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash5}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 2)
 
 	t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
@@ -347,7 +351,7 @@ func TestMultipleHostPrune(t *testing.T) {
 
 	t.Logf("Creating 1 snapshot")
 	chunkHash6 := uploadRandomChunk(snapshotManager, chunkSize)
-    createTestSnapshot(snapshotManager, "vm1@host1", 3, now + 1 * day - 3600 , now + 1 * day, []string {chunkHash5, chunkHash6})
+	createTestSnapshot(snapshotManager, "vm1@host1", 3, now+1*day-3600, now+1*day, []string{chunkHash5, chunkHash6}, "tag")
 	checkTestSnapshots(snapshotManager, 4, 2)
 
 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
@@ -371,8 +375,8 @@ func TestPruneAndResurrect(t *testing.T) {
 	now := time.Now().Unix()
 	day := int64(24 * 3600)
 	t.Logf("Creating 2 snapshots")
-    createTestSnapshot(snapshotManager, "vm1@host1", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash1, chunkHash2})
-    createTestSnapshot(snapshotManager, "vm1@host1", 2, now - 2 * day - 3600, now - 2 * day - 60, []string { chunkHash2, chunkHash3})
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
 	checkTestSnapshots(snapshotManager, 2, 0)
 
 	t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
@@ -381,7 +385,7 @@ func TestPruneAndResurrect(t *testing.T) {
 
 	t.Logf("Creating 1 snapshot")
 	chunkHash4 := uploadRandomChunk(snapshotManager, chunkSize)
-    createTestSnapshot(snapshotManager, "vm1@host1", 4, now + 1 * day - 3600 , now + 1 * day, []string { chunkHash4, chunkHash1})
+	createTestSnapshot(snapshotManager, "vm1@host1", 4, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash1}, "tag")
 	checkTestSnapshots(snapshotManager, 2, 2)
 
 	t.Logf("Prune without removing any snapshots -- one fossil will be resurrected")
@@ -406,10 +410,10 @@ func TestInactiveHostPrune(t *testing.T) {
 	now := time.Now().Unix()
 	day := int64(24 * 3600)
 	t.Logf("Creating 3 snapshot")
-    createTestSnapshot(snapshotManager, "vm1@host1", 1, now - 3 * day - 3600, now - 3 * day - 60, []string { chunkHash1, chunkHash2} )
-    createTestSnapshot(snapshotManager, "vm1@host1", 2, now - 2 * day - 3600, now - 2 * day - 60, []string { chunkHash2, chunkHash3} )
+	createTestSnapshot(snapshotManager, "vm1@host1", 1, now-3*day-3600, now-3*day-60, []string{chunkHash1, chunkHash2}, "tag")
+	createTestSnapshot(snapshotManager, "vm1@host1", 2, now-2*day-3600, now-2*day-60, []string{chunkHash2, chunkHash3}, "tag")
 	// Host2 is inactive
-    createTestSnapshot(snapshotManager, "vm2@host2", 1, now - 7 * day - 3600, now - 7 * day - 60, []string { chunkHash3, chunkHash4} )
+	createTestSnapshot(snapshotManager, "vm2@host2", 1, now-7*day-3600, now-7*day-60, []string{chunkHash3, chunkHash4}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 0)
 
 	t.Logf("Removing snapshot vm1@host1 revision 1")
@@ -422,7 +426,7 @@ func TestInactiveHostPrune(t *testing.T) {
 
 	t.Logf("Creating 1 snapshot")
 	chunkHash5 := uploadRandomChunk(snapshotManager, chunkSize)
-    createTestSnapshot(snapshotManager, "vm1@host1", 3, now + 1 * day - 3600 , now + 1 * day, []string { chunkHash4, chunkHash5} )
+	createTestSnapshot(snapshotManager, "vm1@host1", 3, now+1*day-3600, now+1*day, []string{chunkHash4, chunkHash5}, "tag")
 	checkTestSnapshots(snapshotManager, 3, 2)
 
 	t.Logf("Prune without removing any snapshots -- fossils will be deleted")
@@ -439,7 +443,7 @@ func TestRetentionPolicy(t *testing.T) {
 	snapshotManager := createTestSnapshotManager(testDir)
 
 	chunkSize := 1024
-    var chunkHashes [] string
+	var chunkHashes []string
 	for i := 0; i < 30; i++ {
 		chunkHashes = append(chunkHashes, uploadRandomChunk(snapshotManager, chunkSize))
 	}
@@ -448,7 +452,7 @@ func TestRetentionPolicy(t *testing.T) {
 	day := int64(24 * 3600)
 	t.Logf("Creating 30 snapshots")
 	for i := 0; i < 30; i++ {
-        createTestSnapshot(snapshotManager, "vm1@host1", i + 1, now - int64(30 - i) * day - 3600, now - int64(30 - i) * day - 60, []string { chunkHashes[i] })
+		createTestSnapshot(snapshotManager, "vm1@host1", i+1, now-int64(30-i)*day-3600, now-int64(30-i)*day-60, []string{chunkHashes[i]}, "tag")
 	}
 
 	checkTestSnapshots(snapshotManager, 30, 0)
@@ -465,3 +469,35 @@ func TestRetentionPolicy(t *testing.T) {
 	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"3:14", "2:7"}, false, true, []string{}, false, false, false)
 	checkTestSnapshots(snapshotManager, 12, 0)
 }
+
+func TestRetentionPolicyAndTag(t *testing.T) {
+
+	setTestingT(t)
+
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "snapshot_test")
+
+	snapshotManager := createTestSnapshotManager(testDir)
+
+	chunkSize := 1024
+	var chunkHashes []string
+	for i := 0; i < 30; i++ {
+		chunkHashes = append(chunkHashes, uploadRandomChunk(snapshotManager, chunkSize))
+	}
+
+	now := time.Now().Unix()
+	day := int64(24 * 3600)
+	t.Logf("Creating 30 snapshots")
+	for i := 0; i < 30; i++ {
+		tag := "auto"
+		if i % 3 == 0 {
+			tag = "manual"
+		}
+		createTestSnapshot(snapshotManager, "vm1@host1", i+1, now-int64(30-i)*day-3600, now-int64(30-i)*day-60, []string{chunkHashes[i]}, tag)
+	}
+
+	checkTestSnapshots(snapshotManager, 30, 0)
+
+	t.Logf("Removing snapshot vm1@host1 0:20 with --exclusive and --tag manual")
+	snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{"manual"}, []string{"0:7"}, false, true, []string{}, false, false, false)
+	checkTestSnapshots(snapshotManager, 22, 0)
+}

src/duplicacy_storage.go

@@ -5,23 +5,26 @@
 package duplicacy
 
 import (
+	"encoding/json"
 	"fmt"
-    "regexp"
-    "strings"
-    "strconv"
-    "os"
-    "net"
-    "path"
 	"io/ioutil"
+	"net"
+	"os"
+	"path"
+	"regexp"
 	"runtime"
+	"strconv"
+	"strings"
 
 	"golang.org/x/crypto/ssh"
 	"golang.org/x/crypto/ssh/agent"
 )
 
 type Storage interface {
-    // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
-    ListFiles(threadIndex int, dir string) (files []string, size []int64, err error)
+	// ListFiles return the list of files and subdirectories under 'dir'.  A subdirectories returned must have a trailing '/', with
+	// a size of 0.  If 'dir' is 'snapshots', only subdirectories will be returned.  If 'dir' is 'snapshots/repository_id', then only
+	// files will be returned.  If 'dir' is 'chunks', the implementation can return the list either recusively or non-recusively.
+	ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error)
 
 	// DeleteFile deletes the file or directory at 'filePath'.
 	DeleteFile(threadIndex int, filePath string) (err error)
@@ -45,18 +48,21 @@ type Storage interface {
 	// UploadFile writes 'content' to the file at 'filePath'.
 	UploadFile(threadIndex int, filePath string, content []byte) (err error)
 
+	// SetNestingLevels sets up the chunk nesting structure.
+	SetNestingLevels(config *Config)
+
 	// If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 	// managing snapshots.
-    IsCacheNeeded() (bool)
+	IsCacheNeeded() bool
 
 	// If the 'MoveFile' method is implemented.
-    IsMoveFileImplemented() (bool)
+	IsMoveFileImplemented() bool
 
 	// If the storage can guarantee strong consistency.
-    IsStrongConsistent() (bool)
+	IsStrongConsistent() bool
 
 	// If the storage supports fast listing of files names.
-    IsFastListing() (bool)
+	IsFastListing() bool
 
 	// Enable the test mode.
 	EnableTestMode()
@@ -65,21 +71,104 @@ type Storage interface {
 	SetRateLimits(downloadRateLimit int, uploadRateLimit int)
 }
 
-type RateLimitedStorage struct {
-    DownloadRateLimit int
-    UploadRateLimit int
+// StorageBase is the base struct from which all storages are derived from
+type StorageBase struct {
+	DownloadRateLimit int // Maximum download rate (bytes/seconds)
+	UploadRateLimit   int // Maximum upload reate (bytes/seconds)
+
+	DerivedStorage Storage // Used as the pointer to the derived storage class
+
+	readLevels []int // At which nesting level to find the chunk with the given id
+	writeLevel int   // Store the uploaded chunk to this level
 }
 
-func (storage *RateLimitedStorage) SetRateLimits(downloadRateLimit int, uploadRateLimit int) {
+// SetRateLimits sets the maximum download and upload rates
+func (storage *StorageBase) SetRateLimits(downloadRateLimit int, uploadRateLimit int) {
 	storage.DownloadRateLimit = downloadRateLimit
 	storage.UploadRateLimit = uploadRateLimit
 }
 
+// SetDefaultNestingLevels sets the default read and write levels.  This is usually called by
+// derived storages to set the levels with old values so that storages initialied by ealier versions
+// will continue to work.
+func (storage *StorageBase) SetDefaultNestingLevels(readLevels []int, writeLevel int) {
+	storage.readLevels = readLevels
+	storage.writeLevel = writeLevel
+}
+
+// SetNestingLevels sets the new read and write levels (normally both at 1) if the 'config' file has
+// the 'fixed-nesting' key, or if a file named 'nesting' exists on the storage.
+func (storage *StorageBase) SetNestingLevels(config *Config) {
+
+	// 'FixedNesting' is true only for the 'config' file with the new format (2.0.10+)
+	if config.FixedNesting {
+
+		storage.readLevels = nil
+
+		// Check if the 'nesting' file exist
+		exist, _, _, err := storage.DerivedStorage.GetFileInfo(0, "nesting")
+		if err == nil && exist {
+			nestingFile := CreateChunk(CreateConfig(), true)
+			if storage.DerivedStorage.DownloadFile(0, "nesting", nestingFile) == nil {
+				var nesting struct {
+					ReadLevels []int `json:"read-levels"`
+					WriteLevel int   `json:"write-level"`
+				}
+				if json.Unmarshal(nestingFile.GetBytes(), &nesting) == nil {
+					storage.readLevels = nesting.ReadLevels
+					storage.writeLevel = nesting.WriteLevel
+				}
+			}
+		}
+
+		if len(storage.readLevels) == 0 {
+			storage.readLevels = []int{1}
+			storage.writeLevel = 1
+		}
+	}
+
+	LOG_DEBUG("STORAGE_NESTING", "Chunk read levels: %v, write level: %d", storage.readLevels, storage.writeLevel)
+	for _, level := range storage.readLevels {
+		if storage.writeLevel == level {
+			return
+		}
+	}
+	LOG_ERROR("STORAGE_NESTING", "The write level %d isn't in the read levels %v", storage.readLevels, storage.writeLevel)
+}
+
+// FindChunk finds the chunk with the specified id at the levels one by one as specified by 'readLevels'.
+func (storage *StorageBase) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
+	chunkPaths := make([]string, 0)
+	for _, level := range storage.readLevels {
+		chunkPath := "chunks/"
+		for i := 0; i < level; i++ {
+			chunkPath += chunkID[2*i:2*i+2] + "/"
+		}
+		chunkPath += chunkID[2*level:]
+		if isFossil {
+			chunkPath += ".fsl"
+		}
+		exist, _, size, err = storage.DerivedStorage.GetFileInfo(threadIndex, chunkPath)
+		if err == nil && exist {
+			return chunkPath, exist, size, err
+		}
+		chunkPaths = append(chunkPaths, chunkPath)
+	}
+	for i, level := range storage.readLevels {
+		if storage.writeLevel == level {
+			return chunkPaths[i], false, 0, nil
+		}
+	}
+	return "", false, 0, fmt.Errorf("Invalid chunk nesting setup")
+}
+
 func checkHostKey(hostname string, remote net.Addr, key ssh.PublicKey) error {
 
-    preferencePath := GetDuplicacyPreferencePath()
+	if preferencePath == "" {
+		return fmt.Errorf("Can't verify SSH host since the preference path is not set")
+	}
 	hostFile := path.Join(preferencePath, "known_hosts")
-    file, err := os.OpenFile(hostFile, os.O_RDWR | os.O_CREATE, 0600)
+	file, err := os.OpenFile(hostFile, os.O_RDWR|os.O_CREATE, 0600)
 	if err != nil {
 		return err
 	}
@@ -96,7 +185,7 @@ func checkHostKey(hostname string, remote net.Addr, key ssh.PublicKey) error {
 	keyString = strings.Replace(keyString, "\n", "", -1)
 	remoteAddress := remote.String()
 	if strings.HasSuffix(remoteAddress, ":22") {
-        remoteAddress = remoteAddress[:len(remoteAddress) - len(":22")]
+		remoteAddress = remoteAddress[:len(remoteAddress)-len(":22")]
 	}
 
 	for i, line := range strings.Split(string(content), "\n") {
@@ -146,7 +235,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 	}
 
 	if isFileStorage {
-        fileStorage, err := CreateFileStorage(storageURL, 2, isCacheNeeded, threads)
+		fileStorage, err := CreateFileStorage(storageURL, isCacheNeeded, threads)
 		if err != nil {
 			LOG_ERROR("STORAGE_CREATE", "Failed to load the file storage at %s: %v", storageURL, err)
 			return nil
@@ -155,7 +244,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 	}
 
 	if strings.HasPrefix(storageURL, "flat://") {
-        fileStorage, err := CreateFileStorage(storageURL[7:], 0, false, threads)
+		fileStorage, err := CreateFileStorage(storageURL[7:], false, threads)
 		if err != nil {
 			LOG_ERROR("STORAGE_CREATE", "Failed to load the file storage at %s: %v", storageURL, err)
 			return nil
@@ -164,7 +253,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 	}
 
 	if strings.HasPrefix(storageURL, "samba://") {
-        fileStorage, err := CreateFileStorage(storageURL[8:], 2, true, threads)
+		fileStorage, err := CreateFileStorage(storageURL[8:], true, threads)
 		if err != nil {
 			LOG_ERROR("STORAGE_CREATE", "Failed to load the file storage at %s: %v", storageURL, err)
 			return nil
@@ -187,7 +276,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 
 		if strings.Contains(server, ":") {
 			index := strings.Index(server, ":")
-            port, _ = strconv.Atoi(server[index + 1:])
+			port, _ = strconv.Atoi(server[index+1:])
 			server = server[:index]
 		}
 
@@ -197,7 +286,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		}
 
 		if username != "" {
-            username = username[:len(username) - 1]
+			username = username[:len(username)-1]
 		}
 
 		// If ssh_key_file is set, skip password-based login
@@ -210,12 +299,12 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 			return password, nil
 		}
 
-        keyboardInteractive := func (user, instruction string, questions []string, echos []bool) (answers []string,
+		keyboardInteractive := func(user, instruction string, questions []string, echos []bool) (answers []string,
 			err error) {
 			if len(questions) == 1 {
 				LOG_DEBUG("SSH_INTERACTIVE", "Attempting keyboard interactive login")
 				password = GetPassword(preference, "ssh_password", "Enter SSH password:", false, resetPassword)
-                answers = []string { password }
+				answers = []string{password}
 				return answers, nil
 			} else {
 				return nil, nil
@@ -225,7 +314,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		publicKeysCallback := func() ([]ssh.Signer, error) {
 			LOG_DEBUG("SSH_PUBLICKEY", "Attempting public key authentication")
 
-            signers := []ssh.Signer {}
+			signers := []ssh.Signer{}
 
 			agentSock := os.Getenv("SSH_AUTH_SOCK")
 			if agentSock != "" {
@@ -237,6 +326,8 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 					signers, err = sshAgent.Signers()
 					if err != nil {
 						LOG_DEBUG("SSH_AGENT", "Can't log in using public key authentication via agent: %v", err)
+					} else if len(signers) == 0 {
+						LOG_DEBUG("SSH_AGENT", "SSH agent doesn't return any signer")
 					}
 				}
 			}
@@ -274,13 +365,12 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 
 		}
 
-        authMethods := [] ssh.AuthMethod {
-        }
-        passwordAuthMethods := [] ssh.AuthMethod {
+		authMethods := []ssh.AuthMethod{}
+		passwordAuthMethods := []ssh.AuthMethod{
 			ssh.PasswordCallback(passwordCallback),
 			ssh.KeyboardInteractive(keyboardInteractive),
 		}
-        keyFileAuthMethods := [] ssh.AuthMethod {
+		keyFileAuthMethods := []ssh.AuthMethod{
 			ssh.PublicKeysCallback(publicKeysCallback),
 		}
 		if keyFile != "" {
@@ -298,7 +388,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 				keyFileKey = preference.Name + "_" + keyFileKey
 			}
 
-            authMethods = [] ssh.AuthMethod {}
+			authMethods = []ssh.AuthMethod{}
 			if keyringGet(passwordKey) != "" {
 				authMethods = append(authMethods, ssh.PasswordCallback(passwordCallback))
 				authMethods = append(authMethods, ssh.KeyboardInteractive(keyboardInteractive))
@@ -312,7 +402,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 			return checkHostKey(hostname, remote, key)
 		}
 
-        sftpStorage, err := CreateSFTPStorage(server, port, username, storageDir, authMethods, hostKeyChecker, threads)
+		sftpStorage, err := CreateSFTPStorage(server, port, username, storageDir, 2, authMethods, hostKeyChecker, threads)
 		if err != nil {
 			LOG_ERROR("STORAGE_CREATE", "Failed to load the SFTP storage at %s: %v", storageURL, err)
 			return nil
@@ -333,7 +423,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		bucket := matched[5]
 
 		if region != "" {
-            region = region[:len(region) - 1]
+			region = region[:len(region)-1]
 		}
 
 		if strings.EqualFold(endpoint, "amazon") || strings.EqualFold(endpoint, "amazon.com") {
@@ -343,7 +433,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		storageDir := ""
 		if strings.Contains(bucket, "/") {
 			firstSlash := strings.Index(bucket, "/")
-            storageDir = bucket[firstSlash + 1:]
+			storageDir = bucket[firstSlash+1:]
 			bucket = bucket[:firstSlash]
 		}
 
@@ -374,7 +464,7 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 	} else if matched[1] == "dropbox" {
 		storageDir := matched[3] + matched[5]
 		token := GetPassword(preference, "dropbox_token", "Enter Dropbox access token:", true, resetPassword)
-        dropboxStorage, err := CreateDropboxStorage(token, storageDir, threads)
+		dropboxStorage, err := CreateDropboxStorage(token, storageDir, 1, threads)
 		if err != nil {
 			LOG_ERROR("STORAGE_CREATE", "Failed to load the dropbox storage: %v", err)
 			return nil
@@ -470,6 +560,16 @@ func CreateStorage(preference Preference, resetPassword bool, threads int) (stor
 		}
 		SavePassword(preference, "hubic_token", tokenFile)
 		return hubicStorage
+	} else if matched[1] == "swift" {
+		prompt := fmt.Sprintf("Enter the OpenStack Swift key:")
+		key := GetPassword(preference, "swift_key", prompt, true, resetPassword)
+		swiftStorage, err := CreateSwiftStorage(storageURL[8:], key, threads)
+		if err != nil {
+			LOG_ERROR("STORAGE_CREATE", "Failed to load the OpenStack Swift storage at %s: %v", storageURL, err)
+			return nil
+		}
+		SavePassword(preference, "swift_key", key)
+		return swiftStorage
 	} else {
 		LOG_ERROR("STORAGE_CREATE", "The storage type '%s' is not supported", matched[1])
 		return nil

src/duplicacy_storage_test.go

@@ -5,19 +5,18 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "time"
-    "flag"
-    "path"
-    "testing"
-    "strings"
-    "strconv"
-    "io/ioutil"
 	"crypto/sha256"
 	"encoding/hex"
 	"encoding/json"
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
 	"runtime/debug"
+	"strconv"
+	"testing"
+	"time"
 
 	crypto_rand "crypto/rand"
 	"math/rand"
@@ -41,61 +40,108 @@ func init() {
 func loadStorage(localStoragePath string, threads int) (Storage, error) {
 
 	if testStorageName == "" || testStorageName == "file" {
-        return CreateFileStorage(localStoragePath, 2, false, threads)
+		storage, err := CreateFileStorage(localStoragePath, false, threads)
+		if storage != nil {
+			// Use a read level of at least 2 because this will catch more errors than a read level of 1.
+			storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		}
+		return storage, err
 	}
 
-    config, err := ioutil.ReadFile("test_storage.conf")
+	description, err := ioutil.ReadFile("test_storage.conf")
 	if err != nil {
 		return nil, err
 	}
 
-    storages := make(map[string]map[string]string)
+	configs := make(map[string]map[string]string)
 
-    err = json.Unmarshal(config, &storages)
+	err = json.Unmarshal(description, &configs)
 	if err != nil {
 		return nil, err
 	}
 
-    storage, found := storages[testStorageName]
+	config, found := configs[testStorageName]
 	if !found {
 		return nil, fmt.Errorf("No storage named '%s' found", testStorageName)
 	}
 
 	if testStorageName == "flat" {
-        return CreateFileStorage(localStoragePath, 0, false, threads)
+		storage, err := CreateFileStorage(localStoragePath, false, threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "samba" {
-        return CreateFileStorage(localStoragePath, 2, true, threads)
+		storage, err := CreateFileStorage(localStoragePath, true, threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "sftp" {
-        port, _ := strconv.Atoi(storage["port"])
-        return CreateSFTPStorageWithPassword(storage["server"], port, storage["username"], storage["directory"], storage["password"], threads)
+		port, _ := strconv.Atoi(config["port"])
+		storage, err := CreateSFTPStorageWithPassword(config["server"], port, config["username"], config["directory"], 2, config["password"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "s3" || testStorageName == "wasabi" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, true, false)
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, false)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "s3c" {
-        return CreateS3CStorage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads)
+		storage, err := CreateS3CStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "digitalocean" {
+		storage, err := CreateS3CStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "minio" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, false, true)
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, false, true)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "minios" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, true, true)
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, true)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "dropbox" {
-        return CreateDropboxStorage(storage["token"], storage["directory"], threads)
+		storage, err := CreateDropboxStorage(config["token"], config["directory"], 1, threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "b2" {
-        return CreateB2Storage(storage["account"], storage["key"], storage["bucket"], threads)
+		storage, err := CreateB2Storage(config["account"], config["key"], config["bucket"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "gcs-s3" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, true, false)
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, false)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "gcs" {
-        return CreateGCSStorage(storage["token_file"], storage["bucket"], storage["directory"], threads)
+		storage, err := CreateGCSStorage(config["token_file"], config["bucket"], config["directory"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "gcs-sa" {
-        return CreateGCSStorage(storage["token_file"], storage["bucket"], storage["directory"], threads)
+		storage, err := CreateGCSStorage(config["token_file"], config["bucket"], config["directory"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "azure" {
-        return CreateAzureStorage(storage["account"], storage["key"], storage["container"], threads)
+		storage, err := CreateAzureStorage(config["account"], config["key"], config["container"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "acd" {
-        return CreateACDStorage(storage["token_file"], storage["storage_path"], threads)
+		storage, err := CreateACDStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "gcd" {
-        return CreateGCDStorage(storage["token_file"], storage["storage_path"], threads)
+		storage, err := CreateGCDStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "one" {
-        return CreateOneDriveStorage(storage["token_file"], storage["storage_path"], threads)
+		storage, err := CreateOneDriveStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else if testStorageName == "hubic" {
-        return CreateHubicStorage(storage["token_file"], storage["storage_path"], threads)
+		storage, err := CreateHubicStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "memset" {
+		storage, err := CreateSwiftStorage(config["storage_url"], config["key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
 	} else {
 		return nil, fmt.Errorf("Invalid storage named: %s", testStorageName)
 	}
@@ -111,8 +157,8 @@ func cleanStorage(storage Storage) {
 	LOG_INFO("STORAGE_LIST", "Listing snapshots in the storage")
 	for len(directories) > 0 {
 
-        dir := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 
 		files, _, err := storage.ListFiles(0, dir)
 		if err != nil {
@@ -121,10 +167,10 @@ func cleanStorage(storage Storage) {
 		}
 
 		for _, file := range files {
-            if len(file) > 0 && file[len(file) - 1] == '/' {
-                directories = append(directories, dir + file)
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
 			} else {
-                snapshots = append(snapshots, dir + file)
+				snapshots = append(snapshots, dir+file)
 			}
 		}
 	}
@@ -135,7 +181,7 @@ func cleanStorage(storage Storage) {
 	}
 
 	for _, chunk := range listChunks(storage) {
-        storage.DeleteFile(0, "chunks/" + chunk)
+		storage.DeleteFile(0, "chunks/"+chunk)
 	}
 
 	storage.DeleteFile(0, "config")
@@ -151,8 +197,8 @@ func listChunks(storage Storage) (chunks []string) {
 
 	for len(directories) > 0 {
 
-        dir := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 
 		files, _, err := storage.ListFiles(0, dir)
 		if err != nil {
@@ -161,8 +207,8 @@ func listChunks(storage Storage) (chunks []string) {
 		}
 
 		for _, file := range files {
-            if len(file) > 0 && file[len(file) - 1] == '/' {
-                directories = append(directories, dir + file)
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
 			} else {
 				chunk := dir + file
 				chunk = chunk[len("chunks/"):]
@@ -185,7 +231,7 @@ func moveChunk(t *testing.T, storage Storage, chunkID string, isFossil bool, del
 
 	to := filePath + ".fsl"
 	if isFossil {
-        to = filePath[:len(filePath) - len(".fsl")]
+		to = filePath[:len(filePath)-len(".fsl")]
 	}
 
 	err = storage.MoveFile(0, filePath, to)
@@ -232,7 +278,7 @@ func TestStorage(t *testing.T) {
 				debug.PrintStack()
 			}
 		}
-    } ()
+	}()
 
 	testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
 	os.RemoveAll(testDir)
@@ -256,7 +302,7 @@ func TestStorage(t *testing.T) {
 		delay = 2
 	}
 
-    for _, dir := range []string { "chunks", "snapshots" }  {
+	for _, dir := range []string{"chunks", "snapshots"} {
 		err = storage.CreateDirectory(0, dir)
 		if err != nil {
 			t.Errorf("Failed to create directory %s: %v", dir, err)
@@ -266,6 +312,33 @@ func TestStorage(t *testing.T) {
 
 	storage.CreateDirectory(0, "snapshots/repository1")
 	storage.CreateDirectory(0, "snapshots/repository2")
+
+	storage.CreateDirectory(0, "shared")
+
+	// Upload to the same directory by multiple goroutines
+	count := 8
+	finished := make(chan int, count)
+	for i := 0; i < count; i++ {
+		go func(name string) {
+			err := storage.UploadFile(0, name, []byte("this is a test file"))
+			if err != nil {
+				t.Errorf("Error to upload '%s': %v", name, err)
+			}
+			finished <- 0
+		}(fmt.Sprintf("shared/a/b/c/%d", i))
+	}
+
+	for i := 0; i < count; i++ {
+		<-finished
+	}
+
+	for i := 0; i < count; i++ {
+		storage.DeleteFile(0, fmt.Sprintf("shared/a/b/c/%d", i))
+	}
+	storage.DeleteFile(0, "shared/a/b/c")
+	storage.DeleteFile(0, "shared/a/b")
+	storage.DeleteFile(0, "shared/a")
+
 	time.Sleep(time.Duration(delay) * time.Second)
 	{
 
@@ -299,10 +372,10 @@ func TestStorage(t *testing.T) {
 		return
 	}
 
-    snapshotIDs := []string {}
+	snapshotIDs := []string{}
 	for _, snapshotDir := range snapshotDirs {
-        if len(snapshotDir) > 0 && snapshotDir[len(snapshotDir) - 1] == '/' {
-            snapshotIDs = append(snapshotIDs, snapshotDir[:len(snapshotDir) - 1])
+		if len(snapshotDir) > 0 && snapshotDir[len(snapshotDir)-1] == '/' {
+			snapshotIDs = append(snapshotIDs, snapshotDir[:len(snapshotDir)-1])
 		}
 	}
 
@@ -312,13 +385,13 @@ func TestStorage(t *testing.T) {
 	}
 
 	for _, snapshotID := range snapshotIDs {
-        snapshots, _, err := storage.ListFiles(0, "snapshots/" + snapshotID)
+		snapshots, _, err := storage.ListFiles(0, "snapshots/"+snapshotID)
 		if err != nil {
 			t.Errorf("Failed to list snapshots for %s: %v", snapshotID, err)
 			return
 		}
 		for _, snapshot := range snapshots {
-            storage.DeleteFile(0, "snapshots/" + snapshotID + "/" + snapshot)
+			storage.DeleteFile(0, "snapshots/"+snapshotID+"/"+snapshot)
 		}
 	}
 
@@ -326,7 +399,7 @@ func TestStorage(t *testing.T) {
 
 	storage.DeleteFile(0, "config")
 
-    for _, file := range []string { "snapshots/repository1/1", "snapshots/repository2/1"} {
+	for _, file := range []string{"snapshots/repository1/1", "snapshots/repository2/1"} {
 		exist, _, _, err := storage.GetFileInfo(0, file)
 		if err != nil {
 			t.Errorf("Failed to get file info for %s: %v", file, err)
@@ -338,7 +411,7 @@ func TestStorage(t *testing.T) {
 		}
 	}
 
-    numberOfFiles := 20
+	numberOfFiles := 10
 	maxFileSize := 64 * 1024
 
 	if testQuickMode {
@@ -348,7 +421,7 @@ func TestStorage(t *testing.T) {
 	chunks := []string{}
 
 	for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
+		content := make([]byte, rand.Int()%maxFileSize+1)
 		_, err = crypto_rand.Read(content)
 		if err != nil {
 			t.Errorf("Error generating random content: %v", err)
@@ -374,15 +447,7 @@ func TestStorage(t *testing.T) {
 			t.Errorf("Failed to upload the file %s: %v", filePath, err)
 			return
 		}
-        LOG_INFO("STORAGE_CHUNK", "Uploaded chunk: %s, size: %d", chunkID, len(content))
-    }
-
-    allChunks := [] string {}
-    for _, file := range listChunks(storage) {
-        file = strings.Replace(file, "/", "", -1)
-        if len(file) == 64 {
-            allChunks = append(allChunks, file)
-        }
+		LOG_INFO("STORAGE_CHUNK", "Uploaded chunk: %s, size: %d", filePath, len(content))
 	}
 
 	LOG_INFO("STORAGE_FOSSIL", "Making %s a fossil", chunks[0])
@@ -392,11 +457,10 @@ func TestStorage(t *testing.T) {
 
 	config := CreateConfig()
 	config.MinimumChunkSize = 100
-    config.chunkPool = make(chan *Chunk, numberOfFiles * 2)
+	config.chunkPool = make(chan *Chunk, numberOfFiles*2)
 
 	chunk := CreateChunk(config, true)
 
-
 	for _, chunkID := range chunks {
 
 		chunk.Reset(false)
@@ -413,7 +477,7 @@ func TestStorage(t *testing.T) {
 				t.Errorf("Error downloading file %s: %v", filePath, err)
 				continue
 			}
-            LOG_INFO("STORAGE_CHUNK", "Downloaded chunk: %s, size: %d", chunkID, chunk.GetLength())
+			LOG_INFO("STORAGE_CHUNK", "Downloaded chunk: %s, size: %d", filePath, chunk.GetLength())
 		}
 
 		hasher := sha256.New()
@@ -448,9 +512,14 @@ func TestStorage(t *testing.T) {
 		}
 	}
 
+	allChunks := []string{}
+	for _, file := range listChunks(storage) {
+		allChunks = append(allChunks, file)
+	}
+
 	for _, file := range allChunks {
 
-        err = storage.DeleteFile(0, "chunks/" + file)
+		err = storage.DeleteFile(0, "chunks/"+file)
 		if err != nil {
 			t.Errorf("Failed to delete the file %s: %v", file, err)
 			return
@@ -474,7 +543,7 @@ func TestCleanStorage(t *testing.T) {
 				debug.PrintStack()
 			}
 		}
-    } ()
+	}()
 
 	testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
 	os.RemoveAll(testDir)
@@ -494,8 +563,8 @@ func TestCleanStorage(t *testing.T) {
 
 	for len(directories) > 0 {
 
-        dir := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 
 		LOG_INFO("LIST_FILES", "Listing %s", dir)
 
@@ -506,10 +575,10 @@ func TestCleanStorage(t *testing.T) {
 		}
 
 		for _, file := range files {
-            if len(file) > 0 && file[len(file) - 1] == '/' {
-                directories = append(directories, dir + file)
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
 			} else {
-                storage.DeleteFile(0, dir + file)
+				storage.DeleteFile(0, dir+file)
 				LOG_INFO("DELETE_FILE", "Deleted file %s", file)
 			}
 		}
@@ -518,4 +587,12 @@ func TestCleanStorage(t *testing.T) {
 	storage.DeleteFile(0, "config")
 	LOG_INFO("DELETE_FILE", "Deleted config")
 
+
+	files, _, err := storage.ListFiles(0, "chunks/")
+	for _, file := range files {
+			if len(file) > 0 && file[len(file)-1] != '/' {
+				LOG_DEBUG("FILE_EXIST", "File %s exists after deletion", file)
+			}
+	}
+
 }

src/duplicacy_swiftstorage.go

@@ -0,0 +1,251 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+package duplicacy
+
+import (
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/ncw/swift"
+)
+
+type SwiftStorage struct {
+	StorageBase
+
+	connection *swift.Connection
+	container  string
+	storageDir string
+	threads    int
+}
+
+// CreateSwiftStorage creates an OpenStack Swift storage object.  storageURL is in the form of 
+// `user@authURL/container/path?arg1=value1&arg2=value2``
+func CreateSwiftStorage(storageURL string, key string, threads int) (storage *SwiftStorage, err error) {
+
+	// This is the map to store all arguments
+	arguments := make(map[string]string)
+
+	// Check if there are arguments provided as a query string
+	if strings.Contains(storageURL, "?") {
+		urlAndArguments := strings.SplitN(storageURL, "?", 2)
+		storageURL = urlAndArguments[0]
+		for _, pair := range strings.Split(urlAndArguments[1], "&") {
+			if strings.Contains(pair, "=") {
+				keyAndValue := strings.Split(pair, "=")
+				arguments[keyAndValue[0]] = keyAndValue[1]
+			}
+		}
+	}
+
+	// Take out the user name if there is one
+	if strings.Contains(storageURL, "@") {
+		userAndURL := strings.Split(storageURL, "@")
+		arguments["user"] = userAndURL[0]
+		storageURL = userAndURL[1]
+	}
+
+	// The version is used to split authURL and container/path
+	versions := []string{"/v1/", "/v1.0/", "/v2/", "/v2.0/", "/v3/", "/v3.0/", "/v4/", "/v4.0/"}
+	storageDir := ""
+	for _, version := range versions {
+		if strings.Contains(storageURL, version) {
+			urlAndStorageDir := strings.SplitN(storageURL, version, 2)
+			storageURL = urlAndStorageDir[0] + version[0:len(version)-1]
+			storageDir = urlAndStorageDir[1]
+		}
+	}
+
+	// If no container/path is specified, find them from the arguments
+	if storageDir == "" {
+		storageDir = arguments["storage_dir"]
+	}
+
+	// Now separate the container name from the storage path
+	container := ""
+	if strings.Contains(storageDir, "/") {
+		containerAndStorageDir := strings.SplitN(storageDir, "/", 2)
+		container = containerAndStorageDir[0]
+		storageDir = containerAndStorageDir[1]
+		if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
+			storageDir += "/"
+		}
+	} else {
+		container = storageDir
+		storageDir = ""
+	}
+
+	// Number of retries on err
+	retries := 4
+	if value, ok := arguments["retries"]; ok {
+		retries, _ = strconv.Atoi(value)
+	}
+
+	// Connect channel timeout
+	connectionTimeout := 10
+	if value, ok := arguments["connection_timeout"]; ok {
+		connectionTimeout, _ = strconv.Atoi(value)
+	}
+
+	// Data channel timeout
+	timeout := 60
+	if value, ok := arguments["timeout"]; ok {
+		timeout, _ = strconv.Atoi(value)
+	}
+
+	// Auth version; default to auto-detect
+	authVersion := 0
+	if value, ok := arguments["auth_version"]; ok {
+		authVersion, _ = strconv.Atoi(value)
+	}
+
+	// Allow http to be used by setting "protocol=http" in arguments
+	if _, ok := arguments["protocol"]; !ok {
+		arguments["protocol"] = "https"
+	}
+
+	// Please refer to https://godoc.org/github.com/ncw/swift#Connection
+	connection := swift.Connection{
+		Domain:         arguments["domain"],
+		DomainId:       arguments["domain_id"],
+		UserName:       arguments["user"],
+		UserId:         arguments["user_id"],
+		ApiKey:         key,
+		AuthUrl:        arguments["protocol"] + "://" + storageURL,
+		Retries:        retries,
+		UserAgent:      arguments["user_agent"],
+		ConnectTimeout: time.Duration(connectionTimeout) * time.Second,
+		Timeout:        time.Duration(timeout) * time.Second,
+		Region:         arguments["region"],
+		AuthVersion:    authVersion,
+		Internal:       false,
+		Tenant:         arguments["tenant"],
+		TenantId:       arguments["tenant_id"],
+		EndpointType:   swift.EndpointType(arguments["endpiont_type"]),
+		TenantDomain:   arguments["tenant_domain"],
+		TenantDomainId: arguments["tenant_domain_id"],
+		TrustId:        arguments["trust_id"],
+	}
+
+	_, _, err = connection.Container(container)
+	if err != nil {
+		return nil, err
+	}
+
+	storage = &SwiftStorage{
+		connection: &connection,
+		container:  container,
+		storageDir: storageDir,
+		threads:    threads,
+	}
+
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{1}, 1)
+	return storage, nil
+}
+
+// ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
+func (storage *SwiftStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
+	isSnapshotDir := dir == "snapshots/"
+	dir = storage.storageDir + dir
+
+	options := swift.ObjectsOpts{
+		Prefix: dir,
+		Limit:  1000,
+	}
+
+	if isSnapshotDir {
+		options.Delimiter = '/'
+	}
+
+	objects, err := storage.connection.ObjectsAll(storage.container, &options)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	for _, obj := range objects {
+		if isSnapshotDir {
+			if obj.SubDir != "" {
+				files = append(files, obj.SubDir[len(dir):])
+				sizes = append(sizes, 0)
+			}
+		} else {
+			files = append(files, obj.Name[len(dir):])
+			sizes = append(sizes, obj.Bytes)
+		}
+	}
+
+	return files, sizes, nil
+}
+
+// DeleteFile deletes the file or directory at 'filePath'.
+func (storage *SwiftStorage) DeleteFile(threadIndex int, filePath string) (err error) {
+	return storage.connection.ObjectDelete(storage.container, storage.storageDir+filePath)
+}
+
+// MoveFile renames the file.
+func (storage *SwiftStorage) MoveFile(threadIndex int, from string, to string) (err error) {
+	return storage.connection.ObjectMove(storage.container, storage.storageDir+from,
+		storage.container, storage.storageDir+to)
+}
+
+// CreateDirectory creates a new directory.
+func (storage *SwiftStorage) CreateDirectory(threadIndex int, dir string) (err error) {
+	// Does nothing as directories do not exist in OpenStack Swift
+	return nil
+}
+
+// GetFileInfo returns the information about the file or directory at 'filePath'.
+func (storage *SwiftStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
+	object, _, err := storage.connection.Object(storage.container, storage.storageDir+filePath)
+
+	if err != nil {
+		if err == swift.ObjectNotFound {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
+
+	return true, false, object.Bytes, nil
+}
+
+// DownloadFile reads the file at 'filePath' into the chunk.
+func (storage *SwiftStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
+
+	file, _, err := storage.connection.ObjectOpen(storage.container, storage.storageDir+filePath, false, nil)
+	if err != nil {
+		return err
+	}
+	_, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit/storage.threads)
+	return err
+}
+
+// UploadFile writes 'content' to the file at 'filePath'.
+func (storage *SwiftStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.threads)
+	_, err = storage.connection.ObjectPut(storage.container, storage.storageDir+filePath, reader, true, "", "application/duplicacy", nil)
+	return err
+}
+
+// If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
+// managing snapshots.
+func (storage *SwiftStorage) IsCacheNeeded() bool { return true }
+
+// If the 'MoveFile' method is implemented.
+func (storage *SwiftStorage) IsMoveFileImplemented() bool { return true }
+
+// If the storage can guarantee strong consistency.
+func (storage *SwiftStorage) IsStrongConsistent() bool { return false }
+
+// If the storage supports fast listing of files names.
+func (storage *SwiftStorage) IsFastListing() bool { return true }
+
+// Enable the test mode.
+func (storage *SwiftStorage) EnableTestMode() {
+}

src/duplicacy_utils.go

@@ -5,21 +5,21 @@
 package duplicacy
 
 import (
-    "fmt"
-    "os"
 	"bufio"
+	"crypto/sha256"
+	"fmt"
 	"io"
-    "time"
+	"os"
 	"path"
 	"path/filepath"
 	"regexp"
-    "strings"
-    "strconv"
 	"runtime"
-    "crypto/sha256"
+	"strconv"
+	"strings"
+	"time"
 
-    "golang.org/x/crypto/pbkdf2"
 	"github.com/gilbertchen/gopass"
+	"golang.org/x/crypto/pbkdf2"
 )
 
 var RunInBackground bool = false
@@ -31,15 +31,60 @@ type RateLimitedReader struct {
 	StartTime time.Time
 }
 
-func CreateRateLimitedReader(content []byte, rate int) (*RateLimitedReader) {
-    return &RateLimitedReader {
+var RegexMap map[string]*regexp.Regexp
+
+func init() {
+
+	if RegexMap == nil {
+		RegexMap = make(map[string]*regexp.Regexp)
+	}
+
+}
+
+func CreateRateLimitedReader(content []byte, rate int) *RateLimitedReader {
+	return &RateLimitedReader{
 		Content: content,
 		Rate:    float64(rate * 1024),
 		Next:    0,
 	}
 }
 
-func (reader *RateLimitedReader) Length() (int64) {
+func IsEmptyFilter(pattern string) bool {
+	if pattern == "+" || pattern == "-" || pattern == "i:" || pattern == "e:" {
+		return true
+	} else {
+		return false
+	}
+}
+
+func IsUnspecifiedFilter(pattern string) bool {
+	if pattern[0] != '+' && pattern[0] != '-' && pattern[0] != 'i' && pattern[0] != 'e' {
+		return true
+	} else {
+		return false
+	}
+}
+
+func IsValidRegex(pattern string) (valid bool, err error) {
+
+	var re *regexp.Regexp = nil
+
+	if re, valid = RegexMap[pattern]; valid && re != nil {
+		return true, nil
+	}
+
+	re, err = regexp.Compile(pattern)
+
+	if err != nil {
+		return false, err
+	} else {
+		RegexMap[pattern] = re
+		LOG_DEBUG("REGEX_STORED", "Saved compiled regex for pattern \"%s\", regex=%#v", pattern, re)
+		return true, err
+	}
+}
+
+func (reader *RateLimitedReader) Length() int64 {
 	return int64(len(reader.Content))
 }
 
@@ -78,19 +123,19 @@ func (reader *RateLimitedReader) Read(p []byte) (n int, err error) {
 	}
 
 	elapsed := time.Since(reader.StartTime).Seconds()
-    delay := float64(reader.Next) / reader.Rate - elapsed
-    end := reader.Next + int(reader.Rate / 5)
+	delay := float64(reader.Next)/reader.Rate - elapsed
+	end := reader.Next + int(reader.Rate/5)
 	if delay > 0 {
 		time.Sleep(time.Duration(delay * float64(time.Second)))
 	} else {
-        end += - int(delay * reader.Rate)
+		end += -int(delay * reader.Rate)
 	}
 
 	if end > len(reader.Content) {
 		end = len(reader.Content)
 	}
 
-    n = copy(p, reader.Content[reader.Next : end])
+	n = copy(p, reader.Content[reader.Next:end])
 	reader.Next += n
 	return n, nil
 }
@@ -100,7 +145,7 @@ func RateLimitedCopy(writer io.Writer, reader io.Reader, rate int) (written int6
 		return io.Copy(writer, reader)
 	}
 	for range time.Tick(time.Second / 5) {
-        n, err := io.CopyN(writer, reader, int64(rate * 1024 / 5))
+		n, err := io.CopyN(writer, reader, int64(rate*1024/5))
 		written += n
 		if err != nil {
 			if err == io.EOF {
@@ -114,12 +159,12 @@ func RateLimitedCopy(writer io.Writer, reader io.Reader, rate int) (written int6
 }
 
 // GenerateKeyFromPassword generates a key from the password.
-func GenerateKeyFromPassword(password string) []byte {
-    return pbkdf2.Key([]byte(password), DEFAULT_KEY, 16384, 32, sha256.New)
+func GenerateKeyFromPassword(password string, salt []byte, iterations int) []byte {
+	return pbkdf2.Key([]byte(password), salt, iterations, 32, sha256.New)
 }
 
 // Get password from preference, env, but don't start any keyring request
-func GetPasswordFromPreference(preference Preference, passwordType string) (string) {
+func GetPasswordFromPreference(preference Preference, passwordType string) string {
 	passwordID := passwordType
 	if preference.Name != "default" {
 		passwordID = preference.Name + "_" + passwordID
@@ -137,12 +182,12 @@ func GetPasswordFromPreference(preference Preference, passwordType string) (stri
 	// (i.e., preference.Name) in the key, so the key name should really be passwordType rather
 	// than passwordID; we're using passwordID here only for backward compatibility
 	if len(preference.Keys) > 0 && len(preference.Keys[passwordID]) > 0 {
-        LOG_DEBUG("PASSWORD_KEYCHAIN", "Reading %s from preferences", passwordID)
+		LOG_DEBUG("PASSWORD_PREFERENCE", "Reading %s from preferences", passwordID)
 		return preference.Keys[passwordID]
 	}
 
 	if len(preference.Keys) > 0 && len(preference.Keys[passwordType]) > 0 {
-        LOG_DEBUG("PASSWORD_KEYCHAIN", "Reading %s from preferences", passwordType)
+		LOG_DEBUG("PASSWORD_PREFERENCE", "Reading %s from preferences", passwordType)
 		return preference.Keys[passwordType]
 	}
 
@@ -151,11 +196,16 @@ func GetPasswordFromPreference(preference Preference, passwordType string) (stri
 
 // GetPassword attempts to get the password from KeyChain/KeyRing, environment variables, or keyboard input.
 func GetPassword(preference Preference, passwordType string, prompt string,
-                 showPassword bool, resetPassword bool) (string) {
+	showPassword bool, resetPassword bool) string {
 	passwordID := passwordType
-    password := GetPasswordFromPreference(preference,passwordType)
-    if password != "" {
-        return password
+
+	preferencePassword := GetPasswordFromPreference(preference, passwordType)
+	if preferencePassword != "" {
+		return preferencePassword
+	}
+
+	if preference.Name != "default" {
+		passwordID = preference.Name + "_" + passwordID
 	}
 
 	if resetPassword && !RunInBackground {
@@ -163,6 +213,7 @@ func GetPassword(preference Preference, passwordType string, prompt string,
 	} else {
 		password := keyringGet(passwordID)
 		if password != "" {
+			LOG_DEBUG("PASSWORD_KEYCHAIN", "Reading %s from keychain/keyring", passwordType)
 			return password
 		}
 
@@ -173,7 +224,7 @@ func GetPassword(preference Preference, passwordType string, prompt string,
 
 	}
 
-    password = ""
+	password := ""
 	fmt.Printf("%s", prompt)
 	if showPassword {
 		scanner := bufio.NewScanner(os.Stdin)
@@ -282,11 +333,15 @@ func matchPattern(text string, pattern string) bool {
 // either '+' or '-', whereas '-' indicates exclusion and '+' indicates inclusion.  Wildcards like '*' and '?' may
 // appear in the patterns.  In case no matching pattern is found, the file will be excluded if all patterns are
 // include patterns, and included otherwise.
-func MatchPath(filePath string, patterns [] string) (included bool) {
+func MatchPath(filePath string, patterns []string) (included bool) {
+
+	var re *regexp.Regexp = nil
+	var found bool
+	var matched bool
 
 	allIncludes := true
-    for _, pattern := range patterns {
 
+	for _, pattern := range patterns {
 		if pattern[0] == '+' {
 			if matchPattern(filePath, pattern[1:]) {
 				return true
@@ -296,6 +351,24 @@ func MatchPath(filePath string, patterns [] string) (included bool) {
 			if matchPattern(filePath, pattern[1:]) {
 				return false
 			}
+		} else if strings.HasPrefix(pattern, "i:") || strings.HasPrefix(pattern, "e:") {
+			if re, found = RegexMap[pattern[2:]]; found {
+				matched = re.MatchString(filePath)
+			} else {
+				re, err := regexp.Compile(pattern)
+				if err != nil {
+					LOG_ERROR("REGEX_ERROR", "Invalid regex encountered for pattern \"%s\" - %v", pattern[2:], err)
+				}
+				RegexMap[pattern] = re
+				matched = re.MatchString(filePath)
+			}
+			if matched {
+				return strings.HasPrefix(pattern, "i:")
+			} else {
+				if strings.HasPrefix(pattern, "e:") {
+					allIncludes = false
+				}
+			}
 		}
 	}
 
@@ -315,52 +388,52 @@ func joinPath(components ...string) string {
 	return combinedPath
 }
 
-func PrettyNumber(number int64) (string) {
+func PrettyNumber(number int64) string {
 
 	G := int64(1024 * 1024 * 1024)
 	M := int64(1024 * 1024)
 	K := int64(1024)
 
-    if number > 1000 * G {
-        return fmt.Sprintf("%dG", number / G)
+	if number > 1000*G {
+		return fmt.Sprintf("%dG", number/G)
 	} else if number > G {
-        return fmt.Sprintf("%d,%03dM", number / (1000 * M), (number / M) % 1000)
+		return fmt.Sprintf("%d,%03dM", number/(1000*M), (number/M)%1000)
 	} else if number > M {
-        return fmt.Sprintf("%d,%03dK", number / (1000 * K), (number / K) % 1000)
+		return fmt.Sprintf("%d,%03dK", number/(1000*K), (number/K)%1000)
 	} else if number > K {
-        return fmt.Sprintf("%dK", number / K)
+		return fmt.Sprintf("%dK", number/K)
 	} else {
 		return fmt.Sprintf("%d", number)
 	}
 }
 
-func PrettySize(size int64) (string) {
-    if size > 1024 * 1024 {
-        return fmt.Sprintf("%.2fM", float64(size) / (1024.0 * 1024.0))
+func PrettySize(size int64) string {
+	if size > 1024*1024 {
+		return fmt.Sprintf("%.2fM", float64(size)/(1024.0*1024.0))
 	} else if size > 1024 {
-        return fmt.Sprintf("%.0fK", float64(size) / 1024.0)
+		return fmt.Sprintf("%.0fK", float64(size)/1024.0)
 	} else {
 		return fmt.Sprintf("%d", size)
 	}
 }
 
-func PrettyTime(seconds int64) (string) {
+func PrettyTime(seconds int64) string {
 
 	day := int64(3600 * 24)
 
-    if seconds > day * 2 {
+	if seconds > day*2 {
 		return fmt.Sprintf("%d days %02d:%02d:%02d",
-                           seconds / day, (seconds % day) / 3600, (seconds % 3600) / 60, seconds % 60)
+			seconds/day, (seconds%day)/3600, (seconds%3600)/60, seconds%60)
 	} else if seconds > day {
-        return fmt.Sprintf("1 day %02d:%02d:%02d", (seconds % day) / 3600, (seconds % 3600) / 60, seconds % 60)
+		return fmt.Sprintf("1 day %02d:%02d:%02d", (seconds%day)/3600, (seconds%3600)/60, seconds%60)
 	} else if seconds > 0 {
-        return fmt.Sprintf("%02d:%02d:%02d", seconds / 3600, (seconds % 3600) / 60, seconds % 60)
+		return fmt.Sprintf("%02d:%02d:%02d", seconds/3600, (seconds%3600)/60, seconds%60)
 	} else {
 		return "n/a"
 	}
 }
 
-func AtoSize(sizeString string) (int) {
+func AtoSize(sizeString string) int {
 	sizeString = strings.ToLower(sizeString)
 
 	sizeRegex := regexp.MustCompile(`^([0-9]+)([mk])?$`)
@@ -379,3 +452,10 @@ func AtoSize(sizeString string) (int) {
 
 	return size
 }
+
+func MinInt(x, y int) int {
+	if x < y {
+		return x
+	}
+	return y
+}

src/duplicacy_utils_others.go

@@ -7,10 +7,10 @@
 package duplicacy
 
 import (
-    "os"
 	"bytes"
-    "syscall"
+	"os"
 	"path/filepath"
+	"syscall"
 
 	"github.com/gilbertchen/xattr"
 )
@@ -31,7 +31,7 @@ func GetOwner(entry *Entry, fileInfo *os.FileInfo) {
 	}
 }
 
-func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) (bool) {
+func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool {
 	stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
 	if ok && stat != nil && (int(stat.Uid) != entry.UID || int(stat.Gid) != entry.GID) {
 		if entry.UID != -1 && entry.GID != -1 {
@@ -69,7 +69,7 @@ func (entry *Entry) SetAttributesToFile(fullPath string) {
 		newAttribute, found := entry.Attributes[name]
 		if found {
 			oldAttribute, _ := xattr.Getxattr(fullPath, name)
-            if bytes.Equal(oldAttribute, newAttribute) {
+			if !bytes.Equal(oldAttribute, newAttribute) {
 				xattr.Setxattr(fullPath, name, newAttribute)
 			}
 			delete(entry.Attributes, name)

src/duplicacy_utils_test.go

@@ -5,15 +5,14 @@
 package duplicacy
 
 import (
+	"bytes"
 	"io"
 	"io/ioutil"
 	"time"
-    "bytes"
 
 	crypto_rand "crypto/rand"
 
 	"testing"
-
 )
 
 func TestMatchPattern(t *testing.T) {
@@ -21,70 +20,70 @@ func TestMatchPattern(t *testing.T) {
 	// Test cases were copied from Matching Wildcards: An Empirical Way to Tame an Algorithm
 	// By Kirk J. Krauss, October 07, 2014
 
-    DATA := [] struct {
+	DATA := []struct {
 		text    string
 		pattern string
 		matched bool
-    } {
+	}{
 		// Cases with repeating character sequences.
-        { "abcccd", "*ccd", true },
-        { "mississipissippi", "*issip*ss*", true },
-        { "xxxx*zzzzzzzzy*f", "xxxx*zzy*fffff", false },
-        { "xxxx*zzzzzzzzy*f", "xxx*zzy*f", true },
-        { "xxxxzzzzzzzzyf", "xxxx*zzy*fffff", false },
-        { "xxxxzzzzzzzzyf", "xxxx*zzy*f", true },
-        { "xyxyxyzyxyz", "xy*z*xyz", true },
-        { "mississippi", "*sip*", true },
-        { "xyxyxyxyz", "xy*xyz", true },
-        { "mississippi", "mi*sip*", true },
-        { "ababac", "*abac*", true },
-        { "ababac", "*abac*", true },
-        { "aaazz", "a*zz*", true },
-        { "a12b12", "*12*23", false },
-        { "a12b12", "a12b", false },
-        { "a12b12", "*12*12*", true },
+		{"abcccd", "*ccd", true},
+		{"mississipissippi", "*issip*ss*", true},
+		{"xxxx*zzzzzzzzy*f", "xxxx*zzy*fffff", false},
+		{"xxxx*zzzzzzzzy*f", "xxx*zzy*f", true},
+		{"xxxxzzzzzzzzyf", "xxxx*zzy*fffff", false},
+		{"xxxxzzzzzzzzyf", "xxxx*zzy*f", true},
+		{"xyxyxyzyxyz", "xy*z*xyz", true},
+		{"mississippi", "*sip*", true},
+		{"xyxyxyxyz", "xy*xyz", true},
+		{"mississippi", "mi*sip*", true},
+		{"ababac", "*abac*", true},
+		{"ababac", "*abac*", true},
+		{"aaazz", "a*zz*", true},
+		{"a12b12", "*12*23", false},
+		{"a12b12", "a12b", false},
+		{"a12b12", "*12*12*", true},
 
 		// More double wildcard scenarios.
-        { "XYXYXYZYXYz", "XY*Z*XYz", true },
-        { "missisSIPpi", "*SIP*", true },
-        { "mississipPI", "*issip*PI", true },
-        { "xyxyxyxyz", "xy*xyz", true },
-        { "miSsissippi", "mi*sip*", true },
-        { "miSsissippi", "mi*Sip*", false },
-        { "abAbac", "*Abac*", true },
-        { "abAbac", "*Abac*", true },
-        { "aAazz", "a*zz*", true },
-        { "A12b12", "*12*23", false },
-        { "a12B12", "*12*12*", true },
-        { "oWn", "*oWn*", true },
+		{"XYXYXYZYXYz", "XY*Z*XYz", true},
+		{"missisSIPpi", "*SIP*", true},
+		{"mississipPI", "*issip*PI", true},
+		{"xyxyxyxyz", "xy*xyz", true},
+		{"miSsissippi", "mi*sip*", true},
+		{"miSsissippi", "mi*Sip*", false},
+		{"abAbac", "*Abac*", true},
+		{"abAbac", "*Abac*", true},
+		{"aAazz", "a*zz*", true},
+		{"A12b12", "*12*23", false},
+		{"a12B12", "*12*12*", true},
+		{"oWn", "*oWn*", true},
 
 		// Completely tame (no wildcards) cases.
-        { "bLah", "bLah", true },
-        { "bLah", "bLaH", false },
+		{"bLah", "bLah", true},
+		{"bLah", "bLaH", false},
 
 		// Simple mixed wildcard tests suggested by IBMer Marlin Deckert.
-        { "a", "*?", true },
-        { "ab", "*?", true },
-        { "abc", "*?", true },
+		{"a", "*?", true},
+		{"ab", "*?", true},
+		{"abc", "*?", true},
 
 		// More mixed wildcard tests including coverage for false positives.
-        { "a", "??", false },
-        { "ab", "?*?", true },
-        { "ab", "*?*?*", true },
-        { "abc", "?*?*?", true },
-        { "abc", "?*?*&?", false },
-        { "abcd", "?b*??", true },
-        { "abcd", "?a*??", false },
-        { "abcd", "?*?c?", true },
-        { "abcd", "?*?d?", false },
-        { "abcde", "?*b*?*d*?", true },
+		{"a", "??", false},
+		{"ab", "?*?", true},
+		{"ab", "*?*?*", true},
+		{"abc", "?*?*?", true},
+		{"abc", "?*?*&?", false},
+		{"abcd", "?b*??", true},
+		{"abcd", "?a*??", false},
+		{"abcd", "?*?c?", true},
+		{"abcd", "?*?d?", false},
+		{"abcde", "?*b*?*d*?", true},
 
 		// Single-character-match cases.
-        { "bLah", "bL?h", true },
-        { "bLaaa", "bLa?", false },
-        { "bLah", "bLa?", true },
-        { "bLaH", "?Lah", false },
-        { "bLaH", "?LaH", true },
+		{"bLah", "bL?h", true},
+		{"bLaaa", "bLa?", false},
+		{"bLah", "bLa?", true},
+		{"bLaH", "?Lah", false},
+		{"bLaH", "?LaH", true},
 	}
 
 	for _, data := range DATA {
@@ -96,7 +95,7 @@ func TestMatchPattern(t *testing.T) {
 }
 
 func TestRateLimit(t *testing.T) {
-    content := make([]byte, 100 * 1024)
+	content := make([]byte, 100*1024)
 	_, err := crypto_rand.Read(content)
 	if err != nil {
 		t.Errorf("Error generating random content: %v", err)

src/duplicacy_utils_windows.go

@@ -5,8 +5,8 @@
 package duplicacy
 
 import (
-    "os"
 	"fmt"
+	"os"
 	"syscall"
 	"unsafe"
 )
@@ -36,6 +36,7 @@ type reparseDataBuffer struct {
 	// GenericReparseBuffer
 	reparseBuffer byte
 }
+
 const (
 	FSCTL_GET_REPARSE_POINT          = 0x900A8
 	MAXIMUM_REPARSE_DATA_BUFFER_SIZE = 16 * 1024
@@ -75,17 +76,17 @@ func Readlink(path string) (isRegular bool, s string, err error) {
 		data := (*symbolicLinkReparseBuffer)(unsafe.Pointer(&rdb.reparseBuffer))
 		p := (*[0xffff]uint16)(unsafe.Pointer(&data.PathBuffer[0]))
 		if data.PrintNameLength > 0 {
-            s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength + data.PrintNameOffset)/2])
+			s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength+data.PrintNameOffset)/2])
 		} else {
-            s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength + data.SubstituteNameOffset)/2])
+			s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength+data.SubstituteNameOffset)/2])
 		}
 	case IO_REPARSE_TAG_MOUNT_POINT:
 		data := (*mountPointReparseBuffer)(unsafe.Pointer(&rdb.reparseBuffer))
 		p := (*[0xffff]uint16)(unsafe.Pointer(&data.PathBuffer[0]))
 		if data.PrintNameLength > 0 {
-            s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength + data.PrintNameOffset)/2])
+			s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength+data.PrintNameOffset)/2])
 		} else {
-            s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength + data.SubstituteNameOffset)/2])
+			s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength+data.SubstituteNameOffset)/2])
 		}
 	case IO_REPARSE_TAG_DEDUP:
 		return true, "", nil
@@ -103,7 +104,7 @@ func GetOwner(entry *Entry, fileInfo *os.FileInfo) {
 	entry.GID = -1
 }
 
-func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) (bool) {
+func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool {
 	return true
 }