Bump version to 2.0.4

Don't compute file hashes when DUPLICACY_SKIP_FILE_HASH is set; handle vertical backup-style hashes in restore
Merge branch 'master' of https://github.com/gilbertchen/duplicacy
2026-01-03 04:04:45 -06:00 · 2017-06-29 22:22:59 -04:00 · 2017-06-29 22:19:41 -04:00 · 2017-06-29 13:19:53 -04:00 · 2017-06-29 13:11:28 -04:00 · 2017-06-26 11:08:38 -04:00
28 changed files with 852 additions and 234 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,3 @@
+.idea
+duplicacy_main
+
--- a/ACKNOWLEDGEMENTS.md
+++ b/ACKNOWLEDGEMENTS.md
@@ -7,7 +7,7 @@ Duplicacy is based on the following open source project:
 |https://github.com/bkaradzic/go-lz4        |  BSD-2-Clause |
 |https://github.com/Azure/azure-sdk-for-go  |  Apache-2.0 |
 |https://github.com/tj/go-dropbox           |  MIT |
-|https://github.com/goamz/goamz             |  LGPL-3.0 with static compilation excpetions |
+|https://github.com/aws/aws-sdk-go          |  Apache-2.0 |
 |https://github.com/howeyc/gopass           |  ISC |
 |https://github.com/tmc/keyring             |  ISC |
 |https://github.com/pcwizz/xattr            |  BSD-2-Clause | 
--- a/GUIDE.md
+++ b/GUIDE.md
@@ -16,6 +16,7 @@ OPTIONS:
   -chunk-size, -c 4M              the average size of chunks
   -max-chunk-size, -max 16M       the maximum size of chunks (defaults to chunk-size * 4)
   -min-chunk-size, -min 1M        the minimum size of chunks (defaults to chunk-size / 4)
+   -pref-dir <preference directory path> 	Specify alternate location for .duplicacy preferences directory
 ```

 The *init* command first connects to the storage specified by the storage URL.  If the storage has been already been
@@ -27,12 +28,14 @@ for those commands.  This default storage actually has a name, *default*.
 After that, it will prepare the the current working directory as the repository to be backed up.  Under the hood, it will create a directory
 named *.duplicacy* in the repository and put a file named *preferences* that stores the snapshot id and encryption and storage options.

-The snapshot id is an id used to distinguish different repositories connected to the same storage.  Each repository must have a unique snapshot id.
+The snapshot id is an id used to distinguish different repositories connected to the same storage.  Each repository must have a unique snapshot id.  A snapshot id must contain only characters valid in Linux and Windows paths (alphabet, digits, underscore, dash, etc), but cannot include `/`, `\`, or `@`.

 The -e option controls whether or not encryption will be enabled for the storage.  If encryption is enabled, you will be prompted to enter a storage password.

 The three chunk size parameters are passed to the variable-size chunking algorithm.  Their values are important to the overall performance, especially for cloud storages.  If the chunk size is too small, a lot of overhead will be in sending requests and receiving responses.  If the chunk size is too large, the effect of deduplication will be less obvious as more data will need to be transferred with each chunk.

+The -pref-dir controls the location of the preferences directory. If not specified, a directory named .duplicacy is created in the repository. If specified, it must point to a non-existing directory. The directory is created and a .duplicacy file is created in the repository. The .duplicacy file contains the absolute path name to the preferences directory.
+
 Once a storage has been initialized with these parameters, these parameters cannot be modified any more.

 #### Backup
@@ -499,4 +502,4 @@ Note that the passwords stored in the environment variable and the preference ne

 ## Scripts

-You can instruct Duplicay to run a script before or after executing a command.  For example, if you create a bash script with the name *pre-prune* under the *.duplicacy/scripts* directory, this bash script will be run before the *prune* command starts.  A script named *post-prune* will be run after the *prune* command finishes.  This rule applies to all commands except *init*.
+You can instruct Duplicacy to run a script before or after executing a command.  For example, if you create a bash script with the name *pre-prune* under the *.duplicacy/scripts* directory, this bash script will be run before the *prune* command starts.  A script named *post-prune* will be run after the *prune* command finishes.  This rule applies to all commands except *init*.
--- a/LICENSE.md
+++ b/LICENSE.md
@@ -0,0 +1,20 @@
+Copyright © 2017 Acrosync LLC
+
+Licensor: Acrosync LLC
+
+Software: Dulicacy
+
+Use Limitation: 5 users
+
+License Grant. Licensor hereby grants to each recipient of the Software (“you”) a non-exclusive, non-transferable, royalty-free and fully-paid-up license, under all of the Licensor’s copyright and patent rights, to use, copy, distribute, prepare derivative works of, publicly perform and display the Software, subject to the Use Limitation and the conditions set forth below.
+
+Use Limitation. The license granted above allows use by up to the number of users per entity set forth above (the “Use Limitation”). For determining the number of users, “you” includes all affiliates, meaning legal entities controlling, controlled by, or under common control with you. If you exceed the Use Limitation, your use is subject to payment of Licensor’s then-current list price for licenses.
+
+Conditions. Redistribution in source code or other forms must include a copy of this license document to be provided in a reasonable manner. Any redistribution of the Software is only allowed subject to this license.
+
+Trademarks. This license does not grant you any right in the trademarks, service marks, brand names or logos of Licensor.
+
+DISCLAIMER. THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OR CONDITION, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. LICENSORS HEREBY DISCLAIM ALL LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE.
+
+Termination. If you violate the terms of this license, your rights will terminate automatically and will not be reinstated without the prior written consent of Licensor. Any such termination will not affect the right of others who may have received copies of the Software from you.
+
--- a/README.md
+++ b/README.md
@@ -4,6 +4,8 @@ Duplicacy is a new generation cross-platform cloud backup tool based on the idea

 The repository hosts source code, design documents, and binary releases of the command line version.  There is also a Duplicacy GUI frontend built for Windows and Mac OS X available from https://duplicacy.com.

+There is a special edition of Duplicacy developed for VMware vSphere (ESXi) named [Vertical Backup](https://www.verticalbackup.com) that can back up virtual machine files on ESXi to local drives, network or cloud storages.
+
 ## Features

 Duplicacy currently supports major cloud storage providers (Amazon S3, Google Cloud Storage, Microsoft Azure, Dropbox, Backblaze, Google Drive, Microsoft OneDrive, and Hubic) and offers all essential features of a modern backup tool:
@@ -26,16 +28,13 @@ The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/D

 ## Getting Started

-Duplicacy is written in Go.  You can build the executable by running the following commands:
+Duplicacy is written in Go.  You can run the following command to build the executable (which will be created under `$GOPATH/bin`):

 ```
-git clone https://github.com/gilbertchen/duplicacy.git
-cd duplicacy
-go get ./...
-go build main/duplicacy_main.go
+go get -u github.com/gilbertchen/duplicacy/...
 ```

-You can also visit the [releases page](https://github.com/gilbertchen/duplicacy-cli/releases/latest) to download the version suitable for your platform.  Installation is not needed.
+You can also visit the [releases page](https://github.com/gilbertchen/duplicacy-cli/releases/latest) to download the pre-built binary suitable for your platform..

 Once you have the Duplicacy executable on your path, you can change to the directory that you want to back up (called *repository*) and run the *init* command:

@@ -155,7 +154,7 @@ You'll need to input an access key and a secret key to access your Amazon S3 sto
 Storage URL:  gcs://bucket/path/to/storage
 ```

-Starting from version 2.0.0, a new Google Cloud Storage backend is added which is implemented using the [official Google client library](https://godoc.org/cloud.google.com/go/storage).  You must first obtain a credential file by [authorizing](https://duplicacy.com/gcp_start) Dupliacy to access your Google Cloud Storage account or by [downloading](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts) a service account credential file.
+Starting from version 2.0.0, a new Google Cloud Storage backend is added which is implemented using the [official Google client library](https://godoc.org/cloud.google.com/go/storage).  You must first obtain a credential file by [authorizing](https://duplicacy.com/gcp_start) Duplicacy to access your Google Cloud Storage account or by [downloading](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts) a service account credential file.
 
 You can also use the s3 protocol to access Google Cloud Storage.  To do this, you must enable the [s3 interoperability](https://cloud.google.com/storage/docs/migrating#migration-simple) in your Google Cloud Storage settings and set the storage url as `s3://storage.googleapis.com/bucket/path/to/storage`.

@@ -175,7 +174,7 @@ Storage URL: b2://bucket

 You'll need to input the account id and application key.

-Backblaze's B2 storage is not only the least expensive (at 0.5 cent per GB per month), but also the fastest.  We have been working closely with their developers to leverage the full potentials provided by the B2 API in order to maximumize the transfer speed.
+Backblaze's B2 storage is not only the least expensive (at 0.5 cent per GB per month), but also the fastest.  We have been working closely with their developers to leverage the full potentials provided by the B2 API in order to maximize the transfer speed.

 #### Google Drive

@@ -224,9 +223,9 @@ It is unclear if the lack of cloud backends is due to difficulties in porting th
 [not recommended](http://librelist.com/browser//attic/2014/11/11/backing-up-multiple-servers-into-a-single-repository/#e96345aa5a3469a87786675d65da492b) by the developer due to chunk indices being kept in a local cache. 
 Concurrent access is not only a convenience; it is a necessity for better deduplication.  For instance, if multiple machines with the same OS installed can back up their entire drives to the same storage, only one copy of the system files needs to be stored, greatly reducing the storage space regardless of the number of machines.  Attic still adopts the traditional approach of using a centralized indexing database to manage chunks, and relies heavily on caching to improve performance.  The presence of exclusive locking makes it hard to be adapted for cloud storage APIs and reduces the level of deduplication.

-[restic](https://restic.github.io) is a more recent addition.  It is worth mentioning here because, like Duplicacy, it is written in Go.  It uses a format similar to the git packfile format, but not exactly the same.  Multiple clients backing up to the same storage are still guarded by 
-[locks](https://github.com/restic/restic/blob/master/doc/Design.md#locks).
-A command to delete old backups is in the developer's [plan](https://github.com/restic/restic/issues/18). S3 storage is supported, although it is unclear how hard it is to support other cloud storage APIs because of the need for locking.  Overall, it still falls in the same category as Attic.  Whether it will eventually reach the same level as Attic remains to be seen.
+[restic](https://restic.github.io) is a more recent addition.  It is worth mentioning here because, like Duplicacy, it is written in Go.  It uses a format similar to the git packfile format.  Multiple clients backing up to the same storage are still guarded by 
+[locks](https://github.com/restic/restic/blob/master/doc/Design.md#locks).  A prune operation will therefore completely block all other clients connected to the storage from doing their regular backups.  Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is.  A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss.  This is the exact problem that we avoided by taking the lock-free approach.
+

 The following table compares the feature lists of all these backup tools:

@@ -239,7 +238,7 @@ The following table compares the feature lists of all these backup tools:
 | Encryption         | Yes       | Yes | Yes               | Yes             | Yes               | **Yes**       |
 | Deletion           | No        | No  | Yes               | Yes             | No                | **Yes**       |
 | Concurrent Access  | No        | No  | Exclusive locking | Not recommended | Exclusive locking | **Lock-free** |
-| Cloud Support      | Extensive | No  | No                | No              | S3 only           | **S3, GCS, Azure, Dropbox, Backblaze, Google Drive, OneDrive, and Hubic**|
+| Cloud Support      | Extensive | No  | No                | No              | S3, B2, OpenStack | **S3, GCS, Azure, Dropbox, Backblaze B2, Google Drive, OneDrive, and Hubic**|
 | Snapshot Migration | No        | No  | No                | No              | No                | **Yes**       |


--- a/duplicacy/duplicacy_main.go
+++ b/duplicacy/duplicacy_main.go
@@ -13,11 +13,13 @@ import (
    "strings"
    "strconv"
    "os/exec"
+    "os/signal"
    "encoding/json"

    "github.com/gilbertchen/cli"

    "github.com/gilbertchen/duplicacy/src"
+    "io/ioutil"
 )

 const (
@@ -36,14 +38,14 @@ func getRepositoryPreference(context *cli.Context, storageName string) (reposito
    }

    for {
-        stat, err := os.Stat(path.Join(repository, duplicacy.DUPLICACY_DIRECTORY))
+        stat, err := os.Stat(path.Join(repository, duplicacy.DUPLICACY_DIRECTORY)) //TOKEEP
        if err != nil && !os.IsNotExist(err) {
            duplicacy.LOG_ERROR("REPOSITORY_PATH", "Failed to retrieve the information about the directory %s: %v",
                                repository, err)
            return "", nil
        }

-        if stat != nil && stat.IsDir() {
+        if stat != nil && (stat.IsDir() || stat.Mode().IsRegular()) {
            break
        }

@@ -54,10 +56,10 @@ func getRepositoryPreference(context *cli.Context, storageName string) (reposito
        }
        repository = parent
    }
-
    duplicacy.LoadPreferences(repository)
    
-    duplicacy.SetKeyringFile(path.Join(repository, duplicacy.DUPLICACY_DIRECTORY, "keyring"))
+    preferencePath := duplicacy.GetDuplicacyPreferencePath()
+    duplicacy.SetKeyringFile(path.Join(preferencePath, "keyring"))

    if storageName == "" {
        storageName = context.String("storage")
@@ -137,13 +139,14 @@ func setGlobalOptions(context *cli.Context) {
    duplicacy.RunInBackground = context.GlobalBool("background")
 }

-func runScript(context *cli.Context, repository string, storageName string, phase string) bool {
+func runScript(context *cli.Context, storageName string, phase string) bool {

    if !ScriptEnabled {
        return false
    }
    
-    scriptDir, _ := filepath.Abs(path.Join(repository, duplicacy.DUPLICACY_DIRECTORY, "scripts"))
+    preferencePath := duplicacy.GetDuplicacyPreferencePath()
+    scriptDir, _ := filepath.Abs(path.Join(preferencePath, "scripts"))
    scriptName := phase + "-" + context.Command.Name

    script := path.Join(scriptDir, scriptName)
@@ -174,14 +177,14 @@ func runScript(context *cli.Context, repository string, storageName string, phas
 }

 func initRepository(context *cli.Context) {
-    configRespository(context, true)
+    configRepository(context, true)
 }

 func addStorage(context *cli.Context) {
-    configRespository(context, false)
+    configRepository(context, false)
 }

-func configRespository(context *cli.Context, init bool) {
+func configRepository(context *cli.Context, init bool) {

    setGlobalOptions(context)
    defer duplicacy.CatchLogException()
@@ -221,20 +224,36 @@ func configRespository(context *cli.Context, init bool) {
            return
        }
    
-        duplicacyDirectory := path.Join(repository, duplicacy.DUPLICACY_DIRECTORY)
-        if stat, _ := os.Stat(path.Join(duplicacyDirectory, "preferences")); stat != nil {
+        preferencePath :=  context.String("pref-dir")
+        if preferencePath == "" {
+            preferencePath = path.Join(repository, duplicacy.DUPLICACY_DIRECTORY) // TOKEEP
+        }
+        
+        
+        if stat, _ := os.Stat(path.Join(preferencePath, "preferences")); stat != nil {
            duplicacy.LOG_ERROR("REPOSITORY_INIT", "The repository %s has already been initialized", repository)
            return
        }

-        err = os.Mkdir(duplicacyDirectory, 0744)
+        err = os.Mkdir(preferencePath, 0744)
        if err != nil && !os.IsExist(err) {
            duplicacy.LOG_ERROR("REPOSITORY_INIT", "Failed to create the directory %s: %v",
-                                duplicacy.DUPLICACY_DIRECTORY, err)
+                preferencePath, err)
            return
        }
-
-        duplicacy.SetKeyringFile(path.Join(duplicacyDirectory, "keyring"))
+        if context.String("pref-dir") != "" {
+            // out of tree preference file
+            // write real path into .duplicacy file inside repository
+            duplicacyFileName := path.Join(repository, duplicacy.DUPLICACY_FILE)
+            d1 := []byte(preferencePath)
+            err = ioutil.WriteFile(duplicacyFileName, d1, 0644)
+            if err != nil {
+                duplicacy.LOG_ERROR("REPOSITORY_PATH", "Failed to write %s file inside repository  %v", duplicacyFileName, err)
+                return
+            }
+        }
+        duplicacy.SetDuplicacyPreferencePath(preferencePath)
+        duplicacy.SetKeyringFile(path.Join(preferencePath, "keyring"))

    } else {
        repository, _ = getRepositoryPreference(context, "")
@@ -251,7 +270,7 @@ func configRespository(context *cli.Context, init bool) {
        Encrypted: context.Bool("encrypt"),
    }

-    storage := duplicacy.CreateStorage(repository, preference, true, 1)
+    storage := duplicacy.CreateStorage(preference, true, 1)
    storagePassword := ""
    if preference.Encrypted {
        prompt := fmt.Sprintf("Enter storage password for %s:", preference.StorageURL)
@@ -341,7 +360,7 @@ func configRespository(context *cli.Context, init bool) {

            }

-            otherStorage := duplicacy.CreateStorage(repository, *otherPreference, false, 1)
+            otherStorage := duplicacy.CreateStorage(*otherPreference, false, 1)

            otherPassword := ""
            if otherPreference.Encrypted {
@@ -368,7 +387,7 @@ func configRespository(context *cli.Context, init bool) {

    duplicacy.Preferences = append(duplicacy.Preferences, preference)

-    duplicacy.SavePreferences(repository)
+    duplicacy.SavePreferences()

    duplicacy.LOG_INFO("REPOSITORY_INIT", "%s will be backed up to %s with id %s",
                       repository, preference.StorageURL, preference.SnapshotID)
@@ -489,7 +508,7 @@ func setPreference(context *cli.Context) {
                           oldPreference.StorageURL)
    } else {
        *oldPreference = newPreference
-        duplicacy.SavePreferences(repository)
+        duplicacy.SavePreferences()
        duplicacy.LOG_INFO("STORAGE_SET", "New options for storage %s have been saved", oldPreference.StorageURL)
    }
 }
@@ -506,9 +525,9 @@ func changePassword(context *cli.Context) {
        os.Exit(ArgumentExitCode)
    }

-    repository, preference := getRepositoryPreference(context, "")
+    _, preference := getRepositoryPreference(context, "")

-    storage := duplicacy.CreateStorage(repository, *preference, false, 1)
+    storage := duplicacy.CreateStorage(*preference, false, 1)
    if storage == nil {
        return
    }
@@ -547,7 +566,6 @@ func changePassword(context *cli.Context) {
    duplicacy.LOG_INFO("STORAGE_SET", "The password for storage %s has been changed", preference.StorageURL)
 }

-
 func backupRepository(context *cli.Context) {
    setGlobalOptions(context)
    defer duplicacy.CatchLogException()
@@ -566,7 +584,7 @@ func backupRepository(context *cli.Context) {
        return
    }

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    threads := context.Int("threads")
    if threads < 1 {
@@ -574,7 +592,7 @@ func backupRepository(context *cli.Context) {
    }

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-    storage := duplicacy.CreateStorage(repository, *preference, false, threads)
+    storage := duplicacy.CreateStorage(*preference, false, threads)
    if storage == nil {
        return
    }
@@ -598,10 +616,10 @@ func backupRepository(context *cli.Context) {
    backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
    duplicacy.SavePassword(*preference, "password", password)

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)
    backupManager.Backup(repository, quickMode, threads, context.String("t"), showStatistics, enableVSS)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func restoreRepository(context *cli.Context) {
@@ -623,7 +641,7 @@ func restoreRepository(context *cli.Context) {
        return
    }

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    threads := context.Int("threads")
    if threads < 1 {
@@ -631,7 +649,7 @@ func restoreRepository(context *cli.Context) {
    }

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-    storage := duplicacy.CreateStorage(repository, *preference, false, threads)
+    storage := duplicacy.CreateStorage(*preference, false, threads)
    if storage == nil {
        return
    }
@@ -673,10 +691,10 @@ func restoreRepository(context *cli.Context) {
    backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
    duplicacy.SavePassword(*preference, "password", password)

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)
    backupManager.Restore(repository, revision, true, quickMode, threads, overwrite, deleteMode, showStatistics, patterns)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func listSnapshots(context *cli.Context) {
@@ -693,10 +711,10 @@ func listSnapshots(context *cli.Context) {

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    resetPassword := context.Bool("reset-passwords")
-    storage := duplicacy.CreateStorage(repository, *preference, resetPassword, 1)
+    storage := duplicacy.CreateStorage(*preference, resetPassword, 1)
    if storage == nil {
        return
    }
@@ -723,10 +741,10 @@ func listSnapshots(context *cli.Context) {
    showFiles := context.Bool("files")
    showChunks := context.Bool("chunks")

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)
    backupManager.SnapshotManager.ListSnapshots(id, revisions, tag, showFiles, showChunks)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func checkSnapshots(context *cli.Context) {
@@ -743,9 +761,9 @@ func checkSnapshots(context *cli.Context) {

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

-    storage := duplicacy.CreateStorage(repository, *preference, false, 1)
+    storage := duplicacy.CreateStorage(*preference, false, 1)
    if storage == nil {
        return
    }
@@ -773,10 +791,10 @@ func checkSnapshots(context *cli.Context) {
    searchFossils := context.Bool("fossils")
    resurrect := context.Bool("resurrect")

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)
    backupManager.SnapshotManager.CheckSnapshots(id, revisions, tag, showStatistics, checkFiles, searchFossils, resurrect)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func printFile(context *cli.Context) {
@@ -791,11 +809,11 @@ func printFile(context *cli.Context) {

    repository, preference := getRepositoryPreference(context, "")

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    // Do not print out storage for this command
    //duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-    storage := duplicacy.CreateStorage(repository, *preference, false, 1)
+    storage := duplicacy.CreateStorage(*preference, false, 1)
    if storage == nil {
        return
    }
@@ -815,7 +833,7 @@ func printFile(context *cli.Context) {
    backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
    duplicacy.SavePassword(*preference, "password", password)

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)

    file := ""
    if len(context.Args()) > 0 {
@@ -823,7 +841,7 @@ func printFile(context *cli.Context) {
    }
    backupManager.SnapshotManager.PrintFile(snapshotID, revision, file)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func diff(context *cli.Context) {
@@ -838,10 +856,10 @@ func diff(context *cli.Context) {

    repository, preference := getRepositoryPreference(context, "")

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-    storage := duplicacy.CreateStorage(repository, *preference, false, 1)
+    storage := duplicacy.CreateStorage(*preference, false, 1)
    if storage == nil {
        return
    }
@@ -872,10 +890,10 @@ func diff(context *cli.Context) {
    backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
    duplicacy.SavePassword(*preference, "password", password)

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)
    backupManager.SnapshotManager.Diff(repository, snapshotID, revisions, path, compareByHash)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func showHistory(context *cli.Context) {
@@ -890,10 +908,10 @@ func showHistory(context *cli.Context) {

    repository, preference := getRepositoryPreference(context, "")

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-    storage := duplicacy.CreateStorage(repository, *preference, false, 1)
+    storage := duplicacy.CreateStorage(*preference, false, 1)
    if storage == nil {
        return
    }
@@ -915,10 +933,10 @@ func showHistory(context *cli.Context) {
    backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
    duplicacy.SavePassword(*preference, "password", password)

-    backupManager.SetupSnapshotCache(repository, preference.Name)
+    backupManager.SetupSnapshotCache(preference.Name)
    backupManager.SnapshotManager.ShowHistory(repository, snapshotID, revisions, path, showLocalHash)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func pruneSnapshots(context *cli.Context) {
@@ -933,10 +951,10 @@ func pruneSnapshots(context *cli.Context) {

    repository, preference := getRepositoryPreference(context, "")

-    runScript(context, repository,  preference.Name, "pre")
+    runScript(context, preference.Name, "pre")

    duplicacy.LOG_INFO("STORAGE_SET", "Storage set to %s", preference.StorageURL)
-    storage := duplicacy.CreateStorage(repository, *preference, false, 1)
+    storage := duplicacy.CreateStorage(*preference, false, 1)
    if storage == nil {
        return
    }
@@ -973,11 +991,11 @@ func pruneSnapshots(context *cli.Context) {
    backupManager := duplicacy.CreateBackupManager(preference.SnapshotID, storage, repository, password)
    duplicacy.SavePassword(*preference, "password", password)

-    backupManager.SetupSnapshotCache(repository, preference.Name)
-    backupManager.SnapshotManager.PruneSnapshots(repository, selfID, snapshotID, revisions, tags, retentions,
+    backupManager.SetupSnapshotCache(preference.Name)
+    backupManager.SnapshotManager.PruneSnapshots(selfID, snapshotID, revisions, tags, retentions,
                                                 exhaustive, exclusive, ignoredIDs, dryRun, deleteOnly, collectOnly)

-    runScript(context, repository,  preference.Name, "post")
+    runScript(context, preference.Name, "post")
 }

 func copySnapshots(context *cli.Context) {
@@ -992,10 +1010,10 @@ func copySnapshots(context *cli.Context) {

    repository, source := getRepositoryPreference(context, context.String("from"))

-    runScript(context, repository,  source.Name, "pre")
+    runScript(context, source.Name, "pre")

    duplicacy.LOG_INFO("STORAGE_SET", "Source storage set to %s", source.StorageURL)
-    sourceStorage := duplicacy.CreateStorage(repository, *source, false, 1)
+    sourceStorage := duplicacy.CreateStorage(*source, false, 1)
    if sourceStorage == nil {
        return
    }
@@ -1006,7 +1024,7 @@ func copySnapshots(context *cli.Context) {
    }

    sourceManager := duplicacy.CreateBackupManager(source.SnapshotID, sourceStorage, repository, sourcePassword)
-    sourceManager.SetupSnapshotCache(repository, source.Name)
+    sourceManager.SetupSnapshotCache(source.Name)
    duplicacy.SavePassword(*source, "password", sourcePassword)


@@ -1025,7 +1043,7 @@ func copySnapshots(context *cli.Context) {


    duplicacy.LOG_INFO("STORAGE_SET", "Destination storage set to %s", destination.StorageURL)
-    destinationStorage := duplicacy.CreateStorage(repository, *destination, false, 1)
+    destinationStorage := duplicacy.CreateStorage(*destination, false, 1)
    if destinationStorage == nil {
        return
    }
@@ -1042,7 +1060,7 @@ func copySnapshots(context *cli.Context) {
    destinationManager := duplicacy.CreateBackupManager(destination.SnapshotID, destinationStorage, repository,
                                                        destinationPassword)
    duplicacy.SavePassword(*destination, "password", destinationPassword)
-    destinationManager.SetupSnapshotCache(repository, destination.Name)
+    destinationManager.SetupSnapshotCache(destination.Name)

    revisions := getRevisions(context)
    snapshotID := ""
@@ -1056,7 +1074,7 @@ func copySnapshots(context *cli.Context) {
    }

    sourceManager.CopySnapshots(destinationManager, snapshotID, revisions, threads)
-    runScript(context, repository,  source.Name, "post")
+    runScript(context, source.Name, "post")
 }

 func infoStorage(context *cli.Context) {
@@ -1071,7 +1089,9 @@ func infoStorage(context *cli.Context) {

    repository := context.String("repository")
    if repository != "" {
-        duplicacy.SetKeyringFile(path.Join(repository, duplicacy.DUPLICACY_DIRECTORY, "keyring"))
+        preferencePath := path.Join(repository, duplicacy.DUPLICACY_DIRECTORY)
+        duplicacy.SetDuplicacyPreferencePath(preferencePath)
+        duplicacy.SetKeyringFile(path.Join(preferencePath, "keyring"))
    }

    isEncrypted := context.Bool("e")
@@ -1088,7 +1108,7 @@ func infoStorage(context *cli.Context) {
        password = duplicacy.GetPassword(preference, "password", "Enter the storage password:", false, false)
    }

-    storage := duplicacy.CreateStorage("", preference, context.Bool("reset-passwords"), 1)
+    storage := duplicacy.CreateStorage(preference, context.Bool("reset-passwords"), 1)
    config, isStorageEncrypted, err := duplicacy.DownloadConfig(storage, password)

    if isStorageEncrypted {
@@ -1132,6 +1152,11 @@ func main() {
                    Usage: "the minimum size of chunks (defaults to chunk-size / 4)",
                    Argument: "1M",
                },
+                cli.StringFlag{
+                    Name: "pref-dir",
+                    Usage: "Specify alternate location for .duplicacy preferences directory (absolute or relative to current directory)",
+                    Argument: "<preferences directory path>",
+                },
            },
            Usage:  "Initialize the storage if necessary and the current directory as the repository",
            ArgsUsage: "<snapshot id> <storage url>",
@@ -1658,7 +1683,18 @@ func main() {
    app.Name = "duplicacy"
    app.HelpName = "duplicacy"
    app.Usage = "A new generation cloud backup tool based on lock-free deduplication"
-    app.Version = "2.0.2"
+    app.Version = "2.0.4"
+
+    // If the program is interrupted, call the RunAtError function.
+    c := make(chan os.Signal, 1)                                       
+    signal.Notify(c, os.Interrupt)                                     
+    go func() {                                                        
+        for _ = range c {
+            duplicacy.RunAtError()                                             
+            os.Exit(1)                                                     
+        }                                                                
+    }()        
+
    err := app.Run(os.Args)
    if err != nil {
        os.Exit(2)
--- a/integration_tests/fixed_test.sh
+++ b/integration_tests/fixed_test.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Sanity test for the fixed-size chunking algorithm
+
+. ./test_functions.sh
+
+fixture
+
+pushd ${TEST_REPO}
+${DUPLICACY} init integration-tests $TEST_STORAGE -c 64 -max 64 -min 64 
+
+add_file file3
+add_file file4
+
+
+${DUPLICACY} backup 
+${DUPLICACY} check --files -stats
+popd
--- a/integration_tests/resume_test.sh
+++ b/integration_tests/resume_test.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+
+
+. ./test_functions.sh
+
+fixture
+
+pushd ${TEST_REPO}
+${DUPLICACY} init integration-tests $TEST_STORAGE -c 4k
+
+# Create 10 20k files
+add_file file1 20000
+add_file file2 20000
+add_file file3 20000
+add_file file4 20000
+add_file file5 20000
+add_file file6 20000
+add_file file7 20000
+add_file file8 20000
+add_file file9 20000
+add_file file10 20000
+
+# Limit the rate to 10k/s so the backup will take about 10 seconds
+${DUPLICACY} backup -limit-rate 10 -threads 4 &
+# Kill the backup after 3 seconds
+DUPLICACY_PID=$!
+sleep 3
+kill -2 ${DUPLICACY_PID}
+
+# Try it again to test the multiple-resume case
+${DUPLICACY} backup -limit-rate 10  -threads 4&
+DUPLICACY_PID=$!
+sleep 3
+kill -2 ${DUPLICACY_PID}
+
+# Fail the backup before uploading the snapshot
+env DUPLICACY_FAIL_SNAPSHOT=true ${DUPLICACY} backup
+
+# Now complete the backup
+${DUPLICACY} backup
+${DUPLICACY} check --files
+popd
--- a/integration_tests/test.sh
+++ b/integration_tests/test.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+
+. ./test_functions.sh
+
+fixture
+init_repo_pref_dir
+
+backup
+add_file file3
+backup
+add_file file4
+backup
+add_file file5
+restore
+check
+
--- a/integration_tests/test_functions.sh
+++ b/integration_tests/test_functions.sh
@@ -0,0 +1,123 @@
+#!/bin/bash
+
+get_abs_filename() {
+  # $1 : relative filename
+  echo "$(cd "$(dirname "$1")" && pwd)/$(basename "$1")"
+}
+
+pushd () {
+    command pushd "$@" > /dev/null
+}
+
+popd () {
+    command popd "$@" > /dev/null
+}
+
+
+# Functions used to create integration tests suite
+
+DUPLICACY=$(get_abs_filename ../duplicacy_main)
+
+# Base directory where test repositories will be created
+TEST_ZONE=$HOME/DUPLICACY_TEST_ZONE
+# Test Repository
+TEST_REPO=$TEST_ZONE/TEST_REPO
+
+# Storage for test ( For now, only local path storage is supported by test suite)
+TEST_STORAGE=$TEST_ZONE/TEST_STORAGE
+
+# Extra storage for copy operation
+SECONDARY_STORAGE=$TEST_ZONE/SECONDARY_STORAGE
+
+# Preference directory ( for testing the -pref-dir option)
+DUPLICACY_PREF_DIR=$TEST_ZONE/TEST_DUPLICACY_PREF_DIR
+
+# Scratch pad for testing restore
+TEST_RESTORE_POINT=$TEST_ZONE/RESTORE_POINT
+
+# Make sure $TEST_ZONE is in know state
+
+
+
+function fixture()
+{
+  # clean TEST_RESTORE_POINT
+  rm -rf $TEST_RESTORE_POINT
+  mkdir -p $TEST_RESTORE_POINT
+
+  # clean TEST_STORAGE
+  rm -rf $TEST_STORAGE
+  mkdir -p $TEST_STORAGE
+
+  # clean SECONDARY_STORAGE
+  rm -rf $SECONDARY_STORAGE
+  mkdir -p $SECONDARY_STORAGE
+  
+
+  # clean TEST_DOT_DUPLICACY
+  rm -rf $DUPLICACY_PREF_DIR
+  mkdir -p $DUPLICACY_PREF_DIR
+
+  # Create test repository
+  rm -rf ${TEST_REPO}
+  mkdir -p ${TEST_REPO}
+  pushd ${TEST_REPO}
+    echo "file1" > file1
+    mkdir dir1
+    echo "file2" > dir1/file2
+  popd
+}
+
+function init_repo()
+{
+  pushd ${TEST_REPO}
+  ${DUPLICACY} init integration-tests $TEST_STORAGE
+  ${DUPLICACY} add -copy default secondary integration-tests $SECONDARY_STORAGE
+  ${DUPLICACY} backup
+  popd
+
+}
+
+function init_repo_pref_dir()
+{
+  pushd ${TEST_REPO}
+  ${DUPLICACY} init -pref-dir "${DUPLICACY_PREF_DIR}"  integration-tests ${TEST_STORAGE}
+  ${DUPLICACY} add -copy default secondary integration-tests $SECONDARY_STORAGE
+  ${DUPLICACY} backup
+  popd
+
+}
+
+function add_file()
+{
+  FILE_NAME=$1
+  FILE_SIZE=${2:-20000000}
+  pushd ${TEST_REPO}
+  dd if=/dev/urandom of=${FILE_NAME} bs=1 count=$(($RANDOM % ${FILE_SIZE})) &> /dev/null
+  popd
+}
+
+
+function backup()
+{
+  pushd ${TEST_REPO}
+  ${DUPLICACY} backup
+  ${DUPLICACY} copy -from default -to secondary
+  popd
+}
+
+
+function restore()
+{
+  pushd ${TEST_REPO}
+  ${DUPLICACY} restore -r 2 -delete
+  popd
+}
+
+function check()
+{
+  pushd ${TEST_REPO}
+  ${DUPLICACY} check -files
+  ${DUPLICACY} check -storage secondary -files
+  popd
+}
--- a/integration_tests/threaded_test.sh
+++ b/integration_tests/threaded_test.sh
@@ -0,0 +1,17 @@
+#!/bin/bash
+
+
+. ./test_functions.sh
+
+fixture
+
+pushd ${TEST_REPO}
+${DUPLICACY} init integration-tests $TEST_STORAGE -c 1k
+
+add_file file3
+add_file file4
+
+
+${DUPLICACY} backup -threads 16 
+${DUPLICACY} check --files -stats
+popd
--- a/src/duplicacy_backupmanager.go
+++ b/src/duplicacy_backupmanager.go
@@ -13,6 +13,7 @@ import (
    "path"
    "time"
    "sort"
+    "sync"
    "sync/atomic"
    "strings"
    "strconv"
@@ -70,9 +71,10 @@ func CreateBackupManager(snapshotID string, storage Storage, top string, passwor

 // SetupSnapshotCache creates the snapshot cache, which is merely a local storage under the default .duplicacy
 // directory
-func (manager *BackupManager) SetupSnapshotCache(top string, storageName string) bool {
+func (manager *BackupManager) SetupSnapshotCache(storageName string) bool {
    
-    cacheDir := path.Join(top, DUPLICACY_DIRECTORY, "cache", storageName)
+    preferencePath := GetDuplicacyPreferencePath()
+    cacheDir := path.Join(preferencePath, "cache", storageName)

    storage, err := CreateFileStorage(cacheDir, 1)
    if err != nil {
@@ -93,11 +95,19 @@ func (manager *BackupManager) SetupSnapshotCache(top string, storageName string)
    return true
 }

+// setEntryContent sets the 4 content pointers for each entry in 'entries'.  'offset' indicates the value
+// to be added to the StartChunk and EndChunk points, used when intending to append 'entries' to the
+// original unchanged entry list.
+//
+// This function assumes the Size field of each entry is equal to the length of the chunk content that belong
+// to the file.
 func setEntryContent(entries[] *Entry, chunkLengths[]int, offset int) {
    if len(entries) == 0 {
        return
    }

+    // The following code works by iterating over 'entries' and 'chunkLength' and keeping track of the
+    // accumulated total file size and the accumulated total chunk size.
    i := 0
    totalChunkSize := int64(0)
    totalFileSize := entries[i].Size
@@ -114,6 +124,8 @@ func setEntryContent(entries[] *Entry, chunkLengths[]int, offset int) {
                break
            }

+            // If the current file ends at the end of the current chunk, the next file will
+            // start at the next chunk
            if totalChunkSize + int64(length) == totalFileSize {
                entries[i].StartChunk = j + 1 + offset
                entries[i].StartOffset = 0
@@ -125,6 +137,9 @@ func setEntryContent(entries[] *Entry, chunkLengths[]int, offset int) {
            totalFileSize += entries[i].Size
        }

+        if i >= len(entries) {
+            break
+        }
        totalChunkSize += int64(length)
    }
 }
@@ -149,7 +164,6 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta

    remoteSnapshot := manager.SnapshotManager.downloadLatestSnapshot(manager.snapshotID)
    if remoteSnapshot == nil {
-        quickMode = false
        remoteSnapshot = CreateEmptySnapshot(manager.snapshotID)
        LOG_INFO("BACKUP_START", "No previous backup found")
    } else {
@@ -170,13 +184,24 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
    // UploadChunk.
    chunkCache := make(map[string]bool)

+    var incompleteSnapshot *Snapshot
+
+    // A revision number of 0 means this is the initial backup
    if remoteSnapshot.Revision > 0 {
-        // Add all chunks in the last snapshot to the
+        // Add all chunks in the last snapshot to the cache
        for _, chunkID := range manager.SnapshotManager.GetSnapshotChunks(remoteSnapshot) {
            chunkCache[chunkID] = true
        }
-    } else if manager.storage.IsFastListing() {
-        // If the listing operation is fast, list all chunks and put them in the cache.
+    } else {
+        
+        // In quick mode, attempt to load the incomplete snapshot from last incomplete backup if there is one.
+        if quickMode {
+            incompleteSnapshot = LoadIncompleteSnapshot()
+        }
+
+        // If the listing operation is fast or there is an incomplete snapshot, list all chunks and
+        // put them in the cache.
+        if manager.storage.IsFastListing() || incompleteSnapshot != nil {
            LOG_INFO("BACKUP_LIST", "Listing all chunks")
            allChunks, _ := manager.SnapshotManager.ListAllFiles(manager.storage, "chunks/")

@@ -194,6 +219,39 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
            }
        }

+
+        if incompleteSnapshot != nil {
+
+            // This is the last chunk from the incomplete snapshot that can be found in the cache
+            lastCompleteChunk := -1
+            for i, chunkHash := range incompleteSnapshot.ChunkHashes {
+                chunkID := manager.config.GetChunkIDFromHash(chunkHash)
+                if _, ok := chunkCache[chunkID]; ok {
+                    lastCompleteChunk = i
+                } else {
+                    break
+                }
+            }
+
+            // Only keep those files whose chunks exist in the cache
+            var files []*Entry
+            for _, file := range incompleteSnapshot.Files {
+                if file.StartChunk <= lastCompleteChunk && file.EndChunk <= lastCompleteChunk {
+                    files = append(files, file)
+                } else {
+                    break
+                }
+            }
+            incompleteSnapshot.Files = files
+
+            // Remove incomplete chunks (they may not have been uploaded)
+            incompleteSnapshot.ChunkHashes = incompleteSnapshot.ChunkHashes[:lastCompleteChunk + 1]
+            incompleteSnapshot.ChunkLengths = incompleteSnapshot.ChunkLengths[:lastCompleteChunk + 1]            
+            remoteSnapshot = incompleteSnapshot
+        }
+
+    }
+
    var numberOfNewFileChunks int                 // number of new file chunks
    var totalUploadedFileChunkLength int64        // total length of uploaded file chunks
    var totalUploadedFileChunkBytes int64         // how many actual bytes have been uploaded
@@ -210,10 +268,11 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
    var modifiedEntries [] *Entry    // Files that has been modified or newly created
    var preservedEntries [] *Entry   // Files unchanges

-    // If the quick mode is enabled, we simply treat all files as if they were new, and break them into chunks.
+    // If the quick mode is disable and there isn't an incomplete snapshot from last (failed) backup, 
+    // we simply treat all files as if they were new, and break them into chunks.
    // Otherwise, we need to find those that are new or recently modified

-    if !quickMode {
+    if remoteSnapshot.Revision == 0 && incompleteSnapshot == nil {
        modifiedEntries = localSnapshot.Files
        for _, entry := range modifiedEntries {
            totalModifiedFileSize += entry.Size
@@ -267,7 +326,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
    var preservedChunkHashes []string
    var preservedChunkLengths []int

-    // For each preserved file, adjust the indices StartChunk and EndChunk.  This is done by finding gaps
+    // For each preserved file, adjust the StartChunk and EndChunk pointers.  This is done by finding gaps
    // between these indices and subtracting the number of deleted chunks.
    last := -1
    deletedChunks := 0
@@ -294,6 +353,7 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
    var uploadedEntries [] *Entry
    var uploadedChunkHashes []string
    var uploadedChunkLengths []int
+    var uploadedChunkLock = &sync.Mutex{}

    // the file reader implements the Reader interface. When an EOF is encounter, it opens the next file unless it
    // is the last file.
@@ -317,6 +377,37 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
    chunkMaker := CreateChunkMaker(manager.config, false)
    chunkUploader := CreateChunkUploader(manager.config, manager.storage, nil, threads, nil)

+    localSnapshotReady := false
+    var once sync.Once
+
+    if remoteSnapshot.Revision == 0 {
+        // In case an error occurs during the initial backup, save the incomplete snapshot
+        RunAtError = func() {
+            once.Do(
+                func() {
+                    if !localSnapshotReady {
+                        // Lock it to gain exclusive access to uploadedChunkHashes and uploadedChunkLengths
+                        uploadedChunkLock.Lock()
+                        for _, entry := range uploadedEntries {
+                            entry.EndChunk = -1
+                        }
+                        setEntryContent(uploadedEntries, uploadedChunkLengths, len(preservedChunkHashes))
+                        if len(preservedChunkHashes) > 0 {
+                            localSnapshot.ChunkHashes = preservedChunkHashes
+                            localSnapshot.ChunkHashes = append(localSnapshot.ChunkHashes, uploadedChunkHashes...)
+                            localSnapshot.ChunkLengths = preservedChunkLengths
+                            localSnapshot.ChunkLengths = append(localSnapshot.ChunkLengths, uploadedChunkLengths...)
+                        } else {
+                            localSnapshot.ChunkHashes = uploadedChunkHashes
+                            localSnapshot.ChunkLengths = uploadedChunkLengths
+                        }
+                        uploadedChunkLock.Unlock()
+                    }
+                    SaveIncompleteSnapshot(localSnapshot)
+                })
+        }
+    }
+
    if fileReader.CurrentFile != nil {

        LOG_TRACE("PACK_START", "Packing %s", fileReader.CurrentEntry.Path)
@@ -397,8 +488,11 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
                    chunkUploader.StartChunk(chunk, chunkIndex)
                }

+                // Must lock it because the RunAtError function called by other threads may access these two slices
+                uploadedChunkLock.Lock()
                uploadedChunkHashes = append(uploadedChunkHashes, hash)
                uploadedChunkLengths = append(uploadedChunkLengths, chunkSize)
+                uploadedChunkLock.Unlock()

            },
            func (fileSize int64, hash string) (io.Reader, bool) {
@@ -444,6 +538,8 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
        localSnapshot.ChunkLengths = uploadedChunkLengths
    }

+    localSnapshotReady = true
+
    localSnapshot.EndTime = time.Now().Unix()

    err = manager.SnapshotManager.CheckSnapshot(localSnapshot)
@@ -454,10 +550,15 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta

    localSnapshot.Tag = tag
    localSnapshot.Options = ""
-    if !quickMode {
+    if !quickMode || remoteSnapshot.Revision == 0 {
        localSnapshot.Options = "-hash"
    }

+    if _, found := os.LookupEnv("DUPLICACY_FAIL_SNAPSHOT"); found {
+        LOG_ERROR("SNAPSHOT_FAIL", "Artificially fail the backup for testing purposes")
+        return false
+    }
+
    if shadowCopy {
        if localSnapshot.Options == "" {
            localSnapshot.Options = "-vss"
@@ -504,6 +605,8 @@ func (manager *BackupManager) Backup(top string, quickMode bool, threads int, ta
    manager.SnapshotManager.CleanSnapshotCache(localSnapshot, nil)
    LOG_INFO("BACKUP_END", "Backup for %s at revision %d completed", top, localSnapshot.Revision)

+    RunAtError = func() {}
+    RemoveIncompleteSnapshot()

    totalSnapshotChunks := len(localSnapshot.FileSequence) + len(localSnapshot.ChunkSequence) +
                        len(localSnapshot.LengthSequence)
@@ -600,6 +703,7 @@ func (manager *BackupManager) Restore(top string, revision int, inPlace bool, qu
        }
    }

+    // How will behave restore when repo created using -repo-dir ,??
    err = os.Mkdir(path.Join(top, DUPLICACY_DIRECTORY), 0744)
    if err != nil && !os.IsExist(err) {
        LOG_ERROR("RESTORE_MKDIR", "Failed to create the preference directory: %v", err)
@@ -979,7 +1083,8 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
    var existingFile, newFile *os.File
    var err error
    
-    temporaryPath := path.Join(top, DUPLICACY_DIRECTORY, "temporary")
+    preferencePath := GetDuplicacyPreferencePath()
+    temporaryPath := path.Join(preferencePath, "temporary")
    fullPath := joinPath(top, entry.Path)

    defer func() {
@@ -1027,7 +1132,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
                fileHash = hash
                return nil, false
            })
-        if fileHash == entry.Hash {
+        if fileHash == entry.Hash && fileHash != "" {
            LOG_TRACE("DOWNLOAD_SKIP", "File %s unchanged (by hash)", entry.Path)
            return false
        }
@@ -1128,7 +1233,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun

        // Verify the download by hash
        hash := hex.EncodeToString(hasher.Sum(nil))
-        if hash != entry.Hash {
+        if hash != entry.Hash && hash != "" && entry.Hash != "" && !strings.HasPrefix(entry.Hash, "#") {
            LOG_ERROR("DOWNLOAD_HASH", "File %s has a mismatched hash: %s instead of %s (in-place)",
                      fullPath, "", entry.Hash)
            return false
@@ -1201,7 +1306,7 @@ func (manager *BackupManager) RestoreFile(chunkDownloader *ChunkDownloader, chun
        }

        hash := hex.EncodeToString(hasher.Sum(nil))
-        if hash != entry.Hash {
+        if hash != entry.Hash && hash != "" && entry.Hash != "" && !strings.HasPrefix(entry.Hash, "#") {
            LOG_ERROR("DOWNLOAD_HASH", "File %s has a mismatched hash: %s instead of %s",
                      entry.Path, hash, entry.Hash)
            return false
@@ -1334,6 +1439,7 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho
            } else {
                LOG_INFO("SNAPSHOT_COPY", "Copied chunk %s (%d/%d)", chunk.GetID(), chunkIndex, len(chunks))
            }
+            otherManager.config.PutChunk(chunk)
        })
    chunkUploader.Start()

@@ -1347,7 +1453,10 @@ func (manager *BackupManager) CopySnapshots(otherManager *BackupManager, snapsho

        i := chunkDownloader.AddChunk(chunkHash)
        chunk := chunkDownloader.WaitForChunk(i)
-        chunkUploader.StartChunk(chunk, chunkIndex)
+        newChunk := otherManager.config.GetChunk()
+        newChunk.Reset(true)
+        newChunk.Write(chunk.GetBytes())
+        chunkUploader.StartChunk(newChunk, chunkIndex)
    }

    chunkDownloader.Stop()
--- a/src/duplicacy_backupmanager_test.go
+++ b/src/duplicacy_backupmanager_test.go
@@ -215,8 +215,9 @@ func TestBackupManager(t *testing.T) {

    time.Sleep(time.Duration(delay) * time.Second)

+    SetDuplicacyPreferencePath(testDir + "/repository1")
    backupManager := CreateBackupManager("host1", storage, testDir, password)
-    backupManager.SetupSnapshotCache(testDir + "/repository1", "default")
+    backupManager.SetupSnapshotCache("default")

    backupManager.Backup(testDir + "/repository1", /*quickMode=*/true, threads, "first", false, false)
    time.Sleep(time.Duration(delay) * time.Second)
--- a/src/duplicacy_chunkmaker.go
+++ b/src/duplicacy_chunkmaker.go
@@ -146,7 +146,6 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
        }

        for {
-            startNewChunk()
            maker.bufferStart = 0
            for maker.bufferStart < maker.minimumChunkSize && !isEOF {
                count, err := reader.Read(maker.buffer[maker.bufferStart : maker.minimumChunkSize])
@@ -174,6 +173,7 @@ func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *C
                    return
                } else {
                    endOfChunk(chunk, false)
+                    startNewChunk()
                    fileSize = 0
                    fileHasher = maker.config.NewFileHasher()
                    isEOF = false
--- a/src/duplicacy_config.go
+++ b/src/duplicacy_config.go
@@ -225,8 +225,41 @@ func (config *Config) NewKeyedHasher(key []byte) hash.Hash {
    }
 }

+var SkipFileHash = false
+
+func init() {
+    if value, found := os.LookupEnv("DUPLICACY_SKIP_FILE_HASH"); found && value != "" && value != "0" {
+        SkipFileHash = true
+    }
+}
+
+// Implement a dummy hasher to be used when SkipFileHash is true.
+type DummyHasher struct {
+}
+
+func (hasher *DummyHasher) Write(p []byte) (int, error) {
+    return len(p), nil
+}
+
+func (hasher *DummyHasher) Sum(b []byte) []byte {
+    return []byte("")
+}
+
+func (hasher *DummyHasher) Reset() {
+}
+
+func (hasher *DummyHasher) Size() int {
+    return 0
+}
+
+func (hasher *DummyHasher) BlockSize() int {
+    return 0
+}
+
 func (config *Config) NewFileHasher() hash.Hash {
-    if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
+    if SkipFileHash {
+        return &DummyHasher {}
+    } else if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
        hasher, _ := blake2.New(&blake2.Config{ Size: 32 })
        return hasher
    } else {
--- a/src/duplicacy_entry.go
+++ b/src/duplicacy_entry.go
@@ -22,6 +22,7 @@ import (

 // This is the hidden directory in the repository for storing various files.
 var DUPLICACY_DIRECTORY = ".duplicacy"
+var DUPLICACY_FILE = ".duplicacy"

 // Regex for matching 'StartChunk:StartOffset:EndChunk:EndOffset'
 var contentRegex = regexp.MustCompile(`^([0-9]+):([0-9]+):([0-9]+):([0-9]+)`)
--- a/src/duplicacy_filestorage.go
+++ b/src/duplicacy_filestorage.go
@@ -158,8 +158,12 @@ func (storage *FileStorage) FindChunk(threadIndex int, chunkID string, isFossil

            err = os.Mkdir(subDir, 0744)
            if err != nil {
+                // The directory may have been created by other threads so check it again.
+                stat, _ := os.Stat(subDir)
+                if stat == nil || !stat.IsDir() {
                    return "", false, 0, err
                }
+            }

            dir = subDir
            continue
--- a/src/duplicacy_log.go
+++ b/src/duplicacy_log.go
@@ -160,6 +160,9 @@ const (
    otherExitCode     = 101
 )

+// This is the function to be called before exiting when an error occurs.
+var RunAtError func() = func() {}
+
 func CatchLogException() {
    if r := recover(); r != nil {
        switch e := r.(type) {
@@ -167,10 +170,12 @@ func CatchLogException() {
            if printStackTrace {
                debug.PrintStack()
            }
+            RunAtError()
            os.Exit(duplicacyExitCode)
        default:
            fmt.Fprintf(os.Stderr, "%v\n", e)
            debug.PrintStack()
+            RunAtError()
            os.Exit(otherExitCode)
        }
    }
--- a/src/duplicacy_oneclient.go
+++ b/src/duplicacy_oneclient.go
@@ -128,12 +128,6 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
            Error:  OneDriveError { Status: response.StatusCode },
        }

-        if err := json.NewDecoder(response.Body).Decode(errorResponse); err != nil {
-            return nil, 0, OneDriveError { Status: response.StatusCode, Message: fmt.Sprintf("Unexpected response"), }
-        }
-
-        errorResponse.Error.Status = response.StatusCode
-
        if response.StatusCode == 401 {

            if url == OneDriveRefreshTokenURL {
@@ -152,6 +146,12 @@ func (client *OneDriveClient) call(url string, method string, input interface{},
            backoff *= 2
            continue
        } else {
+            if err := json.NewDecoder(response.Body).Decode(errorResponse); err != nil {
+                return nil, 0, OneDriveError { Status: response.StatusCode, Message: fmt.Sprintf("Unexpected response"), }
+            }
+
+            errorResponse.Error.Status = response.StatusCode
+
            return nil, 0, errorResponse.Error
        }
    }
--- a/src/duplicacy_preference.go
+++ b/src/duplicacy_preference.go
@@ -9,6 +9,7 @@ import (
    "path"
    "io/ioutil"
    "reflect"
+    "os"
 )

 // Preference stores options for each storage.
@@ -23,11 +24,39 @@ type Preference struct {
    Keys map[string]string    `json:"keys"`
 }

+var preferencePath string
 var Preferences [] Preference

-func LoadPreferences(repository string) (bool) {
+func LoadPreferences(repository string) bool {
    
-    description, err := ioutil.ReadFile(path.Join(repository, DUPLICACY_DIRECTORY, "preferences"))
+    preferencePath = path.Join(repository, DUPLICACY_DIRECTORY)
+    
+    stat, err := os.Stat(preferencePath)
+    if err != nil {
+        LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", repository, err)
+        return false
+    }
+    
+    if !stat.IsDir() {
+        content, err := ioutil.ReadFile(preferencePath)
+        if err != nil {
+            LOG_ERROR("DOT_DUPLICACY_PATH", "Failed to locate the preference path: %v", err)
+            return false
+        }
+        realPreferencePath := string(content)
+        stat, err := os.Stat(realPreferencePath)
+        if err != nil {
+            LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", content, err)
+            return false
+        }
+        if !stat.IsDir() {
+            LOG_ERROR("PREFERENCE_PATH", "The preference path %s is not a directory",  realPreferencePath)
+        }
+
+        preferencePath = realPreferencePath
+    }
+    
+    description, err := ioutil.ReadFile(path.Join(preferencePath, "preferences"))
    if err != nil {
        LOG_ERROR("PREFERENCE_OPEN", "Failed to read the preference file from repository %s: %v", repository, err)
        return false
@@ -47,14 +76,28 @@ func LoadPreferences(repository string) (bool) {
    return true
 }

-func SavePreferences(repository string) (bool) {
+func GetDuplicacyPreferencePath() string {
+    if preferencePath == "" {
+        LOG_ERROR("PREFERENCE_PATH", "The preference path has not been set")
+        return ""
+    }
+    return preferencePath
+}
+
+// Normally 'preferencePath' is set in LoadPreferences; however, if LoadPreferences is not called, this function
+// provide another change to set 'preferencePath'
+func SetDuplicacyPreferencePath(p string) {
+    preferencePath = p
+}
+
+func SavePreferences() (bool) {
    description, err := json.MarshalIndent(Preferences, "", "    ")
    if err != nil {
        LOG_ERROR("PREFERENCE_MARSHAL", "Failed to marshal the repository preferences: %v", err)
        return false
    }
+    preferenceFile := path.Join(GetDuplicacyPreferencePath(), "preferences")
    
-    preferenceFile := path.Join(repository, DUPLICACY_DIRECTORY, "/preferences")
    err = ioutil.WriteFile(preferenceFile, description, 0644)
    if err != nil {
        LOG_ERROR("PREFERENCE_WRITE", "Failed to save the preference file %s: %v", preferenceFile, err)
--- a/src/duplicacy_s3storage.go
+++ b/src/duplicacy_s3storage.go
@@ -5,46 +5,54 @@
 package duplicacy

 import (
-    "time"
-    "github.com/gilbertchen/goamz/aws"
-    "github.com/gilbertchen/goamz/s3"
+    "github.com/aws/aws-sdk-go/aws"
+    "github.com/aws/aws-sdk-go/aws/awserr"
+    "github.com/aws/aws-sdk-go/aws/credentials"
+    "github.com/aws/aws-sdk-go/aws/session"
+    "github.com/aws/aws-sdk-go/service/s3"
 )

 type S3Storage struct {
    RateLimitedStorage

-    buckets []*s3.Bucket
+    client *s3.S3
+    bucket string
    storageDir string
+    numberOfThreads int
 }

 // CreateS3Storage creates a amazon s3 storage object.
 func CreateS3Storage(regionName string, endpoint string, bucketName string, storageDir string,
                     accessKey string, secretKey string, threads int) (storage *S3Storage, err error) {

-    var region aws.Region
+    token := ""
+    
+    auth := credentials.NewStaticCredentials(accessKey, secretKey, token)
+
+    if regionName == "" && endpoint == "" {
+        defaultRegionConfig := &aws.Config {
+            Region: aws.String("us-east-1"),
+            Credentials: auth,
+        }
+        
+        s3Client := s3.New(session.New(defaultRegionConfig))
+
+        response, err := s3Client.GetBucketLocation(&s3.GetBucketLocationInput{Bucket: aws.String(bucketName)})
+
+        if err != nil {
+            return nil, err
+        }
    
-    if endpoint == "" {
-        if regionName == "" {
        regionName = "us-east-1"
+        if response.LocationConstraint != nil {
+            regionName = *response.LocationConstraint
        }
-        region = aws.Regions[regionName]
-    } else {
-        region = aws.Region { Name: regionName, S3Endpoint:"https://" + endpoint }
    }
   
-    auth := aws.Auth{ AccessKey: accessKey, SecretKey: secretKey }
-
-    var buckets []*s3.Bucket
-    for i := 0; i < threads; i++ {
-        s3Client := s3.New(auth, region)
-        s3Client.AttemptStrategy = aws.AttemptStrategy{
-            Min:   8,
-            Total: 300 * time.Second,
-            Delay: 1000 * time.Millisecond,
-        }
-
-        bucket := s3Client.Bucket(bucketName)
-        buckets = append(buckets, bucket)
+    config := &aws.Config {
+        Region: aws.String(regionName),
+        Credentials: auth,
+        Endpoint: aws.String(endpoint),
    }
    
    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
@@ -52,8 +60,10 @@ func CreateS3Storage(regionName string, endpoint string, bucketName string, stor
    }

    storage = &S3Storage {
-        buckets: buckets,
+        client: s3.New(session.New(config)),
+        bucket: bucketName,
        storageDir: storageDir,
+        numberOfThreads: threads,
    }
    
    return storage, nil
@@ -65,67 +75,82 @@ func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string
        dir += "/"
    }

-    dirLength := len(storage.storageDir + dir)
    if dir == "snapshots/" {
-        results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "/", "", 100)
+        dir = storage.storageDir + dir
+        input := s3.ListObjectsInput {
+            Bucket: aws.String(storage.bucket),
+            Prefix: aws.String(dir),
+            Delimiter: aws.String("/"),
+            MaxKeys: aws.Int64(1000),
+        }
+        
+        output, err := storage.client.ListObjects(&input)
        if err != nil {
            return nil, nil, err
        }
        
-        for _, subDir := range results.CommonPrefixes {
-            files = append(files, subDir[dirLength:])
+        for _, subDir := range output.CommonPrefixes {
+            files = append(files, (*subDir.Prefix)[len(dir):])
        }
        return files, nil, nil
-    } else if dir == "chunks/" {
+    } else {
+        dir = storage.storageDir + dir
        marker := ""
        for {
-            results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "", marker, 1000)
+            input := s3.ListObjectsInput {
+                Bucket: aws.String(storage.bucket),
+                Prefix: aws.String(dir),
+                MaxKeys: aws.Int64(1000),
+                Marker: aws.String(marker),
+            }
+
+            output, err := storage.client.ListObjects(&input)
            if err != nil {
                return nil, nil, err
            }

-            for _, object := range results.Contents {
-                files = append(files, object.Key[dirLength:])
-                sizes = append(sizes, object.Size)
+            for _, object := range output.Contents {
+                files = append(files, (*object.Key)[len(dir):])
+                sizes = append(sizes, *object.Size)
            }

-            if !results.IsTruncated {
+            if !*output.IsTruncated {
                break
            }

-            marker = results.Contents[len(results.Contents) - 1].Key
+            marker = *output.Contents[len(output.Contents) - 1].Key            
        }
        return files, sizes, nil
-
-    } else {
-
-        results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "", "", 1000)
-        if err != nil {
-            return nil, nil, err
    }    

-        for _, object := range results.Contents {
-            files = append(files, object.Key[dirLength:])
-        }
-        return files, nil, nil
-    }
 }

 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *S3Storage) DeleteFile(threadIndex int, filePath string) (err error) {
-    return storage.buckets[threadIndex].Del(storage.storageDir + filePath)
+    input := &s3.DeleteObjectInput {
+        Bucket: aws.String(storage.bucket),
+        Key: aws.String(storage.storageDir + filePath),
+    }
+    _, err = storage.client.DeleteObject(input)
+    return err
 }

 // MoveFile renames the file.
 func (storage *S3Storage) MoveFile(threadIndex int, from string, to string) (err error) {

-    options := s3.CopyOptions { ContentType: "application/duplicacy" }
-    _, err = storage.buckets[threadIndex].PutCopy(storage.storageDir + to, s3.Private, options, storage.buckets[threadIndex].Name + "/" + storage.storageDir + from)
+    input := &s3.CopyObjectInput {
+        Bucket: aws.String(storage.bucket),
+        CopySource: aws.String(storage.bucket + "/" + storage.storageDir + from),
+        Key: aws.String(storage.storageDir + to),
+    }
+
+    _, err = storage.client.CopyObject(input)
    if err != nil {
-        return nil
+        return err
    }
  
    return storage.DeleteFile(threadIndex, from)
+
 }

 // CreateDirectory creates a new directory.
@@ -136,19 +161,24 @@ func (storage *S3Storage) CreateDirectory(threadIndex int, dir string) (err erro
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *S3Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {

-    response, err := storage.buckets[threadIndex].Head(storage.storageDir + filePath, nil)
+    input := &s3.HeadObjectInput {
+        Bucket: aws.String(storage.bucket),
+        Key: aws.String(storage.storageDir + filePath),
+    }
+
+    output, err := storage.client.HeadObject(input)
    if err != nil {
-        if e, ok := err.(*s3.Error); ok && (e.StatusCode == 403 || e.StatusCode == 404) {
+        if e, ok := err.(awserr.RequestFailure); ok && (e.StatusCode() == 403 || e.StatusCode() == 404) {
            return false, false, 0, nil
        } else {
            return false, false, 0, err
        }    
    }
    
-    if response.StatusCode == 403 || response.StatusCode == 404 {
+    if output == nil || output.ContentLength == nil {
        return false, false, 0, nil
    } else {
-        return true, false, response.ContentLength, nil
+        return true, false, *output.ContentLength, nil
    }
 }

@@ -174,14 +204,19 @@ func (storage *S3Storage) FindChunk(threadIndex int, chunkID string, isFossil bo
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *S3Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {

-    readCloser, err := storage.buckets[threadIndex].GetReader(storage.storageDir + filePath)
+   input := &s3.GetObjectInput {
+        Bucket: aws.String(storage.bucket),
+        Key: aws.String(storage.storageDir + filePath),
+    }
+    
+    output, err := storage.client.GetObject(input)
    if err != nil {
        return err
    }
    
-    defer readCloser.Close()
+    defer output.Body.Close()
    
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.buckets))
+    _, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit / len(storage.bucket))
    return err

 }
@@ -189,9 +224,16 @@ func (storage *S3Storage) DownloadFile(threadIndex int, filePath string, chunk *
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *S3Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {

-    options := s3.Options { }
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.buckets))
-    return storage.buckets[threadIndex].PutReader(storage.storageDir + filePath, reader, int64(len(content)), "application/duplicacy", s3.Private, options)
+    input := &s3.PutObjectInput {
+        Bucket: aws.String(storage.bucket),
+        Key: aws.String(storage.storageDir + filePath),
+        ACL: aws.String(s3.ObjectCannedACLPrivate),
+        Body: CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.bucket)),
+        ContentType: aws.String("application/duplicacy"),
+    }
+    
+    _, err = storage.client.PutObject(input)
+    return err
 }

 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
--- a/src/duplicacy_sftpstorage.go
+++ b/src/duplicacy_sftpstorage.go
@@ -215,8 +215,12 @@ func (storage *SFTPStorage) FindChunk(threadIndex int, chunkID string, isFossil

            err = storage.client.Mkdir(subDir)
            if err != nil {
+                // The directory may have been created by other threads so check it again.
+                stat, _ := storage.client.Stat(subDir)
+                if stat == nil || !stat.IsDir() {
                    return "", false, 0, fmt.Errorf("Failed to create the directory %s: %v", subDir, err)
                }
+            }

            dir = subDir
            continue
--- a/src/duplicacy_shadowcopy_windows.go
+++ b/src/duplicacy_shadowcopy_windows.go
@@ -9,7 +9,6 @@ import (
    "unsafe"
    "time"
    "os"
-    "path"
    "runtime"

    ole "github.com/gilbertchen/go-ole"
@@ -510,7 +509,8 @@ func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {

    snapshotPath := uint16ArrayToString(properties.SnapshotDeviceObject)
    
-    shadowLink = path.Join(top, DUPLICACY_DIRECTORY) + "\\shadow"
+    preferencePath := GetDuplicacyPreferencePath()
+    shadowLink =  preferencePath + "\\shadow"
    os.Remove(shadowLink)
    err = os.Symlink(snapshotPath + "\\", shadowLink)
    if err != nil {
--- a/src/duplicacy_snapshot.go
+++ b/src/duplicacy_snapshot.go
@@ -67,7 +67,8 @@ func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, ski
    }

    var patterns []string
-    patternFile, err := ioutil.ReadFile(path.Join(top, DUPLICACY_DIRECTORY, "filters"))
+    
+    patternFile, err := ioutil.ReadFile(path.Join(GetDuplicacyPreferencePath(), "filters"))
    if err == nil {
        for _, pattern := range strings.Split(string(patternFile), "\n") {
            pattern = strings.TrimSpace(pattern)
@@ -136,6 +137,96 @@ func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, ski
    return snapshot, skippedDirectories, skippedFiles, nil
 }

+// This is the struct used to save/load incomplete snapshots
+type IncompleteSnapshot struct {
+    Files [] *Entry
+    ChunkHashes []string
+    ChunkLengths [] int
+}
+
+// LoadIncompleteSnapshot loads the incomplete snapshot if it exists
+func LoadIncompleteSnapshot() (snapshot *Snapshot) {
+    snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")
+    description, err := ioutil.ReadFile(snapshotFile)    
+    if err != nil {
+        return nil
+    }
+
+    var incompleteSnapshot IncompleteSnapshot
+
+    err = json.Unmarshal(description, &incompleteSnapshot)
+    if err != nil {
+        return nil
+    }
+
+    var chunkHashes []string
+    for _, chunkHash := range incompleteSnapshot.ChunkHashes {
+        hash, err := hex.DecodeString(chunkHash)
+        if err != nil {
+            return nil
+        }
+        chunkHashes = append(chunkHashes, string(hash))
+    }
+
+    snapshot = &Snapshot {
+        Files: incompleteSnapshot.Files,
+        ChunkHashes: chunkHashes,
+        ChunkLengths: incompleteSnapshot.ChunkLengths,
+    }      
+    LOG_INFO("INCOMPLETE_LOAD", "Incomplete snpashot loaded from %s", snapshotFile)
+    return snapshot
+}
+
+// SaveIncompleteSnapshot saves the incomplete snapshot under the preference directory
+func SaveIncompleteSnapshot(snapshot *Snapshot) {
+    var files []*Entry
+    for _, file := range snapshot.Files {
+        if file.EndChunk >= 0 {
+            file.Attributes = nil
+            files = append(files, file)
+        } else {
+            break
+        }
+    }
+    var chunkHashes []string
+    for _, chunkHash := range snapshot.ChunkHashes {
+        chunkHashes = append(chunkHashes, hex.EncodeToString([]byte(chunkHash)))
+    }
+
+    incompleteSnapshot := IncompleteSnapshot {
+        Files: files,
+        ChunkHashes: chunkHashes,
+        ChunkLengths: snapshot.ChunkLengths,
+    }
+
+    description, err := json.Marshal(incompleteSnapshot)
+    if err != nil {
+        LOG_WARN("INCOMPLETE_ENCODE", "Failed to encode the incomplete snapshot: %v", err)
+        return
+    }
+
+    snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")    
+    err = ioutil.WriteFile(snapshotFile, description, 0644)
+    if err != nil {
+        LOG_WARN("INCOMPLETE_WRITE", "Failed to save the incomplete snapshot: %v", err)        
+        return
+    }
+
+    LOG_INFO("INCOMPLETE_SAVE", "Incomplete snapshot saved to %s", snapshotFile)
+}
+
+func RemoveIncompleteSnapshot() {
+    snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")    
+    if stat, err := os.Stat(snapshotFile); err == nil && !stat.IsDir() {
+        err = os.Remove(snapshotFile)
+        if err != nil {
+            LOG_INFO("INCOMPLETE_SAVE", "Failed to remove ncomplete snapshot: %v", err)
+        } else {
+            LOG_INFO("INCOMPLETE_SAVE", "Removed incomplete snapshot %s", snapshotFile)    
+        }
+    }
+}
+
 // CreateSnapshotFromDescription creates a snapshot from json decription.
 func CreateSnapshotFromDescription(description []byte) (snapshot *Snapshot, err error) {

--- a/src/duplicacy_snapshotmanager.go
+++ b/src/duplicacy_snapshotmanager.go
@@ -1084,7 +1084,7 @@ func (manager *SnapshotManager) RetrieveFile(snapshot *Snapshot, file *Entry, ou
    if alternateHash {
        fileHash = "#" + fileHash
    }
-    if strings.ToLower(fileHash) != strings.ToLower(file.Hash) {
+    if strings.ToLower(fileHash) != strings.ToLower(file.Hash) && !SkipFileHash {
        LOG_WARN("SNAPSHOT_HASH", "File %s has mismatched hashes: %s vs %s", file.Path, file.Hash, fileHash)
        return false
    }
@@ -1496,7 +1496,7 @@ func (manager *SnapshotManager) resurrectChunk(fossilPath string, chunkID string
 // Note that a snapshot being created when step 2 is in progress may reference a fossil.  To avoid this
 // problem, never remove the lastest revision (unless exclusive is true), and only cache chunks referenced
 // by the lastest revision.
-func (manager *SnapshotManager) PruneSnapshots(top string, selfID string, snapshotID string, revisionsToBeDeleted []int,
+func (manager *SnapshotManager) PruneSnapshots(selfID string, snapshotID string, revisionsToBeDeleted []int,
                                               tags []string, retentions []string,
                                               exhaustive bool, exclusive bool, ignoredIDs []string,
                                               dryRun bool, deleteOnly bool, collectOnly bool) bool {
@@ -1511,7 +1511,8 @@ func (manager *SnapshotManager) PruneSnapshots(top string, selfID string, snapsh
        LOG_WARN("DELETE_OPTIONS", "Tags or retention policy will be ignored if at least one revision is specified")
    }
    
-    logDir := path.Join(top, DUPLICACY_DIRECTORY, "logs")
+    preferencePath := GetDuplicacyPreferencePath()
+    logDir := path.Join(preferencePath, "logs")
    os.Mkdir(logDir, 0700)
    logFileName := path.Join(logDir, time.Now().Format("prune-log-20060102-150405"))
    logFile, err := os.OpenFile(logFileName, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0600)
--- a/src/duplicacy_snapshotmanager_test.go
+++ b/src/duplicacy_snapshotmanager_test.go
@@ -248,11 +248,11 @@ func TestSingleRepositoryPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 0)

    t.Logf("Removing snapshot repository1 revision 1 with --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "repository1", "repository1", []int{1}, []string{}, []string{}, false, true, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("repository1", "repository1", []int{1}, []string{}, []string{}, false, true, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 0)

    t.Logf("Removing snapshot repository1 revision 2 without --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "repository1", "repository1", []int{2}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("repository1", "repository1", []int{2}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 1, 2)

    t.Logf("Creating 1 snapshot")
@@ -261,7 +261,7 @@ func TestSingleRepositoryPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "repository1", "repository1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("repository1", "repository1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 0)
 }

@@ -288,11 +288,11 @@ func TestSingleHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 0)

    t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Creating 1 snapshot")
@@ -301,7 +301,7 @@ func TestSingleHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 2)

    t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 3, 0)

 }
@@ -329,11 +329,11 @@ func TestMultipleHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 0)

    t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Creating 1 snapshot")
@@ -342,7 +342,7 @@ func TestMultipleHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 2)

    t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 3, 2)

    t.Logf("Creating 1 snapshot")
@@ -351,7 +351,7 @@ func TestMultipleHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 4, 2)

    t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 4, 0)
 }

@@ -376,7 +376,7 @@ func TestPruneAndResurrect(t *testing.T) {
    checkTestSnapshots(snapshotManager, 2, 0)

    t.Logf("Removing snapshot vm1@host1 revision 1 without --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 1, 2)

    t.Logf("Creating 1 snapshot")
@@ -385,7 +385,7 @@ func TestPruneAndResurrect(t *testing.T) {
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Prune without removing any snapshots -- one fossil will be resurrected")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 0)
 }

@@ -413,11 +413,11 @@ func TestInactiveHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 0)

    t.Logf("Removing snapshot vm1@host1 revision 1")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{1}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Prune without removing any snapshots -- no fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 2, 2)

    t.Logf("Creating 1 snapshot")
@@ -426,7 +426,7 @@ func TestInactiveHostPrune(t *testing.T) {
    checkTestSnapshots(snapshotManager, 3, 2)

    t.Logf("Prune without removing any snapshots -- fossils will be deleted")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{}, false, false, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 3, 0)
 }

@@ -454,14 +454,14 @@ func TestRetentionPolicy(t *testing.T) {
    checkTestSnapshots(snapshotManager, 30, 0)

    t.Logf("Removing snapshot vm1@host1 0:20 with --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 19, 0)

    t.Logf("Removing snapshot vm1@host1 -k 0:20 with --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"0:20"}, false, true, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 19, 0)

    t.Logf("Removing snapshot vm1@host1 -k 3:14 -k 2:7 with --exclusive")
-    snapshotManager.PruneSnapshots(testDir, "vm1@host1", "vm1@host1", []int{}, []string{}, []string{"3:14", "2:7"}, false, true, []string{}, false, false, false)
+    snapshotManager.PruneSnapshots("vm1@host1", "vm1@host1", []int{}, []string{}, []string{"3:14", "2:7"}, false, true, []string{}, false, false, false)
    checkTestSnapshots(snapshotManager, 12, 0)
 }
--- a/src/duplicacy_storage.go
+++ b/src/duplicacy_storage.go
@@ -75,14 +75,10 @@ func (storage *RateLimitedStorage) SetRateLimits(downloadRateLimit int, uploadRa
    storage.UploadRateLimit = uploadRateLimit
 }

-func checkHostKey(repository string, hostname string, remote net.Addr, key ssh.PublicKey) error {
+func checkHostKey(hostname string, remote net.Addr, key ssh.PublicKey) error {
    
-    if len(repository) == 0 {
-        return nil
-    }
-
-    duplicacyDirectory := path.Join(repository, DUPLICACY_DIRECTORY)
-    hostFile := path.Join(duplicacyDirectory, "knowns_hosts")
+    preferencePath := GetDuplicacyPreferencePath()
+    hostFile := path.Join(preferencePath, "known_hosts")
    file, err := os.OpenFile(hostFile, os.O_RDWR | os.O_CREATE, 0600)
    if err != nil {
        return err
@@ -126,7 +122,7 @@ func checkHostKey(repository string, hostname string, remote net.Addr, key ssh.P
 }

 // CreateStorage creates a storage object based on the provide storage URL.
-func CreateStorage(repository string, preference Preference, resetPassword bool, threads int) (storage Storage) {
+func CreateStorage(preference Preference, resetPassword bool, threads int) (storage Storage) {

    storageURL := preference.StorageURL

@@ -282,7 +278,7 @@ func CreateStorage(repository string, preference Preference, resetPassword bool,
        }

        hostKeyChecker := func(hostname string, remote net.Addr, key ssh.PublicKey) error {
-            return checkHostKey(repository, hostname, remote, key)
+            return checkHostKey(hostname, remote, key)
        }

        sftpStorage, err := CreateSFTPStorage(server, port, username, storageDir, authMethods, hostKeyChecker, threads)
--- a/src/duplicacy_utils.go
+++ b/src/duplicacy_utils.go
@@ -48,6 +48,16 @@ func (reader *RateLimitedReader) Reset() {
    reader.Next = 0
 }

+func (reader *RateLimitedReader)  Seek(offset int64, whence int) (int64, error) {
+    if whence == io.SeekStart {
+        reader.Next = int(offset)
+    } else if whence == io.SeekCurrent {
+        reader.Next += int(offset)
+    } else {
+        reader.Next = len(reader.Content) - int(offset)
+    }
+    return int64(reader.Next), nil
+}

 func (reader *RateLimitedReader) Read(p []byte) (n int, err error) {
Author	SHA1	Message	Date
Gilbert Chen	817e36c7a6	Bump version to 2.0.4	2017-06-29 22:22:59 -04:00
Gilbert Chen	b7b54478fc	Don't compute file hashes when DUPLICACY_SKIP_FILE_HASH is set; handle vertical backup-style hashes in restore	2017-06-29 22:19:41 -04:00
Gilbert Chen	8d06fa491a	Merge branch 'master' of https://github.com/gilbertchen/duplicacy	2017-06-29 13:19:53 -04:00
Gilbert Chen	42a6ab9140	In fixed-size chunking, create a new chunk after returning the old one	2017-06-29 13:11:28 -04:00
gilbertchen	bad990e702	Merge pull request #81 from chbmuc/master Move error parsing behind status code handling	2017-06-26 11:08:38 -04:00
gilbertchen	d27335ad8d	Update README.md	2017-06-23 22:02:15 -04:00
Gilbert Chen	a584828e1b	Merge branch 'master' of https://github.com/gilbertchen/duplicacy	2017-06-22 22:53:42 -04:00
Gilbert Chen	d0c376f593	Implement fast resume; refactor GetDuplicacyPreferencePath()	2017-06-22 22:53:33 -04:00
gilbertchen	a54029cf2b	Update GUIDE.md	2017-06-22 13:11:04 -04:00
Gilbert Chen	839be6094f	Remove unused import	2017-06-20 16:37:47 -04:00
Gilbert Chen	84a4c86ca7	Bump version to 2.0.3	2017-06-20 14:39:04 -04:00
Gilbert Chen	651d82e511	Check directory existence again when failing to create it to avoid erroring out on race condition	2017-06-20 14:38:09 -04:00
Christian Brunner	6a73a62591	Move error parsing behind status code handling Otherwise request throttling won't work and you will get errors like this: PUT https://api.onedrive.com/v1.0/drive/root:/dup/chunks/91xxx08:/content Failed to upload the chunk 91xxx08: 503 Unexpected response	2017-06-16 14:06:14 +02:00
gilbertchen	169d6db544	Create README.md	2017-06-15 16:22:09 -04:00
gilbertchen	25684942b3	Merge pull request #78 from stefandz/patch-2 another tiny typo	2017-06-15 10:49:59 -04:00
gilbertchen	746431d5e0	Merge pull request #77 from stefandz/patch-1 Update GUIDE.md	2017-06-15 10:49:17 -04:00
Gilbert Chen	28da4d15e2	Fixed #76 : must create a new chunk for uploading in the copy operation	2017-06-15 10:48:24 -04:00
stefandz	d36e80a5eb	another tiny typo	2017-06-15 15:40:28 +01:00
stefandz	fe1de10f22	Update GUIDE.md Tiny typo	2017-06-15 11:29:52 +01:00
gilbertchen	112d5b22e5	Replace goamz with aws-sdk-g	2017-06-13 15:16:24 -04:00
gilbertchen	3da8830592	Fix typos	2017-06-13 13:29:01 -04:00
gilbertchen	04b01fa87d	Merge pull request #73 from sdaros/master Fix typo	2017-06-13 13:17:05 -04:00
Stefano Da Ros	4b60859054	Fix typo The file should be titled "known_hosts" instead.	2017-06-13 18:53:59 +02:00
Gilbert Chen	7e5fc0972d	Make LICENSE a Markdown file for better viewing	2017-06-13 12:37:05 -04:00
Gilbert Chen	c9951d6036	Move LICENSE to the top directory	2017-06-13 12:36:02 -04:00
Gilbert Chen	92b3594e89	Add a LICENSE file	2017-06-13 12:35:06 -04:00
Gilbert Chen	2424a2eeed	Switch from goamz to aws-sdk-go for the S3 storage backend	2017-06-13 12:27:01 -04:00
gilbertchen	2ace6c74e1	Merge pull request #71 from ech1965/pref-dir add -pref-dir command line option for init subcommand	2017-06-13 11:58:07 -04:00
Etienne Charlier	2fcc4d44b9	Merge branch 'master' of https://github.com/gilbertchen/duplicacy into pref-dir	2017-06-12 19:28:52 +02:00
gilbertchen	3f45b0a15a	Update README.md	2017-06-11 14:09:39 -04:00
gilbertchen	2d69f64c20	Create README.md	2017-06-11 14:08:13 -04:00
Gilbert Chen	7a1a541c98	Rename main directory for better support of go get	2017-06-11 14:02:43 -04:00
Etienne Charlier	7aa0eca47c	Fix typo	2017-06-11 14:10:14 +02:00
Etienne Charlier	aa909c0c15	Update documentation	2017-06-11 13:48:11 +02:00
Etienne Charlier	9e1740c1d6	Fix merge error	2017-06-10 17:14:58 +02:00
Etienne Charlier	ae34347741	merge version 2.0.2	2017-06-10 17:12:44 +02:00
Etienne Charlier	1361b553ac	Remove logging statement; refactor test scripts	2017-06-08 22:21:57 +02:00
Etienne Charlier	c688c501d3	Refactor variable names and revert shadow copy path computation	2017-06-07 21:02:55 +02:00
Etienne Charlier	c88e148d59	First steps -pref-dir	2017-06-05 23:16:11 +02:00