Commit Graph

78 Commits

Author SHA1 Message Date
Gilbert Chen
bec3a0edcd Fixed a bug that caused a fresh restore to fail without the -overwrite option
When restoring a file that doesn't exit locally, if the file is large (>100M)
Duplicacy will create an empty sparse file.  But this newly created file will
be mistaken for a local copy and hence the restore will fail with a message
suggesting the -overwrite option.
2020-11-03 10:57:47 -05:00
Gilbert Chen
ace1ba5848 Remove runtime OS check for excluding by attributes 2020-09-25 22:37:54 -04:00
gilbertchen
1fedfd1b1a Merge branch 'master' into mac-exclude 2020-09-25 20:13:43 -04:00
gilbertchen
3fd3f6b267 Merge pull request #606 from gilbertchen/erasure_coding
Implement Erasure Coding
2020-09-25 14:30:39 -04:00
Gilbert Chen
3f29ec2ffb File metedata must be restored even if the content is unchanged. 2020-09-24 22:44:43 -04:00
Gilbert Chen
947006411b Improvements for the copy command
* Added a `-download-threads` option for specifying the number of downloading
  threads
* Show progress log messages during copy
2020-09-24 14:56:19 -04:00
Gilbert Chen
73ae3f809e Revert "Add a -max-list-rate option to backup to slow down the listing"
This reverts commit 67a3103467.
2020-09-22 22:08:43 -04:00
Gilbert Chen
67a3103467 Add a -max-list-rate option to backup to slow down the listing
This option sets the maximum number of files that can be listed in one
second.
2020-09-22 08:27:09 -04:00
Gilbert Chen
16d2c14c5a Follow-up changes for the -persist PR
* Restore/check should report an error instead of a success at the end if there
  were any errors and -persist is specified
* Don't compute the file hash before passing the file to the chunk maker; this is
  redundant as the chunk maker will produce the file hash
* Add a LOG_WERROR function to switch between LOG_WARN and LOG_ERROR dynamically
2020-09-18 11:23:35 -04:00
gilbertchen
ca4d004aca Merge branch 'master' into add_persist_pr 2020-09-09 15:42:01 -04:00
Gilbert Chen
ce472fe375 Show erasure coding/rsa encryption if enabled for backup and copy 2020-09-04 10:43:37 -04:00
Gilbert Chen
09e3cdfebf Ignore 0-byte chunks passed in by the chunk reader.
The fixed-size chunk reader may create 0-byte chunks from empty files.  This
may cause validation errors when preparing the snapshot file as the last step
of a backup.
2020-06-08 10:53:01 -04:00
Tet Woo Lee
4ae16dec7f add -persist in check and restore mode (for PR) 2020-05-06 18:39:52 +12:00
Gilbert Chen
810303ce25 Fail the backup if the repository can't be accessed or there are no files
This is mainly to avoid creating an empty snapshot when a drive/share is
not mounted which causes the subsequent backup to scan all files again.
2020-03-30 17:44:53 -04:00
Gilbert Chen
6699e2f440 Fixed a bug that disabled RSA when copying from a non RSA-encrypted storage.
When copying to an RSA-encrypted storage, we relied on the RSA encryption
version to determine if a chunk is a snapshot chunk or a file chunk.  This is
wrong when the source storage is not encrypted or not RSA-encrypted.  There
is a more reliable to determine if a chunk is a snapshot chunk or not.
2020-03-13 20:13:27 -04:00
Gilbert Chen
cc88abd547 Fixed a bug that caused all copied chunks to be RSA encrypted
The field encryptionVersion in the Chunk struct is supposed to pass the status
of RSA encrytpion from a source chunk to a destination chunk in a copy command.
This field needs to be a 3-state boolean in order to pass the status correctly.
2020-02-13 14:03:07 -05:00
Gilbert Chen
d43fe1a282 Release the list of chunk hashes after processing each snapshot.
The chunk hash list isn't needed any more after being consolidated.
Releasing it immediately after use helps reduce the memory usage.
2019-12-09 22:45:16 -05:00
Gilbert Chen
a99f059b52 Allow a custom location for the filters file
You can now add a key 'filters' in the preferences file that points to the
path of the filters file.  If this key is not found in the preferences,
the default location '.duplicacy/filters' is used.

There is a new option '-filters' for the set command that set this key in
the preferences, but you can also edit the file directly.
2019-11-23 15:23:26 -05:00
Gilbert Chen
9a0d60ca84 Store the public key in the config to ensure one key policy.
Also make sure that RSA encrpytion works with the copy command.
2019-09-23 12:53:43 -04:00
Gilbert Chen
90833f9d86 Implement RSA encryption
This is to support public key encryption in the backup operation.  You can use
the -key option to supply the public key to the backup command, and then the
same option to supply the private key when restoring a previous revision.

The storage must be encrypted for this to work.
2019-09-20 14:19:18 -04:00
Gilbert Chen
47c4c25d8b Fixed a bug that restoring files doesn't work due to missing parent directory
The root cause was path.Dir can't handle Windows paths that use \ as the
separator.
2019-06-04 21:57:10 -04:00
Gilbert Chen
6e7d45caac Add a TRACE message when skipping a file to be restored 2019-05-22 12:03:21 -04:00
Michael Cook
0762c448c4 gofmt -s 2018-12-29 13:20:10 +01:00
Michael Cook
741644b575 spelling 2018-12-29 13:04:40 +01:00
Gilbert Chen
bfb4b44c0a Optimizating restore to avoid reading newly created sparse file 2018-10-21 22:43:24 -04:00
Patrick Seal
a1efbe3b73 Add exclude_by_attribute preference 2018-09-21 21:35:40 -07:00
Gilbert Chen
a7d2a941be Restore UID and GID of symlinks 2018-08-29 17:10:35 -04:00
Gilbert Chen
39d71a3256 Fixed a divide by zero bug when the repository has only zero-byte files 2018-08-10 12:17:40 -04:00
Gilbert Chen
7e021f26d3 Don't show the nil error when failing to replace a file with a directory during restore 2018-05-30 13:23:35 -04:00
Gilbert Chen
aadd2aa390 Add an -enum-only option to the backup command to enumerate the repository only 2018-05-24 11:34:46 -04:00
Frank
4dd5c43307 Add nobackup-file preference.
Directories containing a file with this name will not be backed up. I find it easier to drop a .nobackup file in directories I don't want backed up instead of maintaining a file of exclusions. This is also useful for scripts that create data in the repository but don't want it to be backed up.
2018-04-01 12:50:00 -07:00
Gilbert Chen
be2856ebbd Add a -vss-timeout option to set VSS creation timeout 2018-03-21 22:34:14 -04:00
gilbertchen
13fffc2a11 Merge pull request #329 from pdf/prune_memory
Reduce memory consumption for prune operation
2018-03-20 14:32:12 -04:00
Gilbert Chen
52fd553bb9 Fixed 2 bugs in restoring extended attributes 2018-01-29 14:38:48 -05:00
Gilbert Chen
7230ddbef5 Clear the attributes from last snapshot after loading to save memory 2018-01-28 16:54:06 -05:00
Peter Fern
bd5a689b7d Add keepChunkHashes flag to GetSnapshotChunks, allowing reduced memory 2018-01-21 10:11:59 +11:00
Gilbert Chen
8e8a116028 Fixed a bug that caused -hash to have no effect 2018-01-11 21:43:42 -05:00
Gilbert Chen
f6ef9094bc Add debugging messages in incomplete snapshot handling 2017-12-15 08:23:56 -05:00
Gilbert Chen
a6d071e1b5 Fixed a bug in splitting the existing file that caused all chunks to be redownloaded 2017-11-10 13:59:12 -05:00
Gilbert Chen
8600803ba0 Remove commented-out call to RestoreMetadata 2017-11-10 13:52:41 -05:00
gilbertchen
669d5ed3f4 Merge pull request #259 from gilbertchen/fixed_nesting
Implement new chunk directory structure
2017-11-10 13:30:17 -05:00
Gilbert Chen
86767b3df6 Implement new chunk directory structure 2017-11-07 12:05:39 -05:00
Mark Lowne
57edf5823d allow skip setting uid/gid on restored files 2017-10-30 20:16:28 +01:00
Gilbert Chen
923cd0aa63 Run goimports on all source files 2017-09-20 23:07:43 -04:00
gilbertchen
fab9cc77c6 Merge pull request #180 from niknah/dry_run
Added backup --dry-run option.
2017-09-18 22:32:21 -04:00
niknah
c63621cb8c Tab spacing fixes 2017-09-18 18:43:44 +10:00
niknah
f25783d59d Comments from @gilbertchen, thanks. 2017-09-18 01:38:51 +10:00
Jeff Thompson
b56d7dedba add regex matching to include/exclude filters 2017-09-14 08:49:24 -05:00
niknah
491252e3e4 Didn't merge --dry-run with the latest updates properly. 2017-09-14 14:13:09 +10:00
Jeff Thompson
0bf66168fb futher optimization for the copy command 2017-09-11 11:45:57 -05:00