416 Commits

Author SHA1 Message Date
Gilbert Chen
16d2c14c5a Follow-up changes for the -persist PR
* Restore/check should report an error instead of a success at the end if there
  were any errors and -persist is specified
* Don't compute the file hash before passing the file to the chunk maker; this is
  redundant as the chunk maker will produce the file hash
* Add a LOG_WERROR function to switch between LOG_WARN and LOG_ERROR dynamically
2020-09-18 11:23:35 -04:00
Gilbert Chen
eecbb8fa99 Fix OneDrive Business and improve retry mechanism.
* Switch to the upload-by-session api for OneDrive Bussiness as their servers
may keep incomplete files when an upload is aborted when the simple upload API
is used

* Use the delay value in the http Retry-After header if there is one

* Decorate the https traffic in the hope of less rate limiting.
2020-09-18 10:19:03 -04:00
Gilbert Chen
97bae5f1a3 Close the response body when 301 is returned in the WebDAV backend 2020-09-14 11:36:22 -04:00
Gilbert Chen
40243fb043 Fixed a compile error in previous checkin 2020-09-12 11:39:46 -04:00
Gilbert Chen
403df1fd06 Added a call to discard http response where it was missed in previous fixes.
Also added a missing import "io/ioutil".
2020-09-09 21:47:01 -04:00
gilbertchen
4369bcfc0b Merge pull request #549 from Jos635/master
Improve WebDAV performance
2020-09-09 16:02:36 -04:00
gilbertchen
d2b08aebee Merge pull request #594 from alecuyer/fix/swift
Call Authenticate() before using Swift storage
2020-09-09 15:48:36 -04:00
gilbertchen
ca4d004aca Merge branch 'master' into add_persist_pr 2020-09-09 15:42:01 -04:00
Gilbert Chen
ce472fe375 Show erasure coding/rsa encryption if enabled for backup and copy 2020-09-04 10:43:37 -04:00
Gilbert Chen
923a6fbc5b Implement Erasure Coding 2020-09-03 12:54:48 -04:00
Gilbert Chen
fd469bae9e Check the returned value of Close() when uploading a chunk file via SFTP. 2020-08-29 22:22:51 -04:00
Gilbert Chen
1eb1fb14a8 Don't throw an error on 0-byte chunk files with suffix '.tmp'. 2020-07-07 23:25:09 -04:00
Gilbert Chen
1da7e2b536 Fix a crash when a username is not specified with the WebDAV backend 2020-07-03 12:29:53 -04:00
Gilbert Chen
ed8b4393be Add a new backend for StorageMadeEasy's File Fabric storage.
The storage url is fabric://username@storagemadeeasy.com/path/to/storage.
2020-07-03 12:07:33 -04:00
Gilbert Chen
f2f07a120d Retry on "unexpected EOF" errors for the webdav backend. 2020-07-03 11:48:30 -04:00
Gilbert Chen
153f6a2d20 Use multiple threads to list the chunks directory for Google Drive 2020-06-15 12:49:13 -04:00
Gilbert Chen
5d45999077 Clear the loaded content after a snapshot has been verified
The snapshot content is loaded before verifying the snapshot, but after that
it isn't used anymore so it should be released to save memory.
2020-06-10 10:08:53 -04:00
Gilbert Chen
1adcf56890 Add an SFTP backend that supports more ciphers and kex algorithms.
"sftpc://" supports all algorithms implemented in golang.org/x/crypto/ssh,
especially including those weak ones that are excluded from the defaults.
2020-06-08 11:24:20 -04:00
Gilbert Chen
09e3cdfebf Ignore 0-byte chunks passed in by the chunk reader.
The fixed-size chunk reader may create 0-byte chunks from empty files.  This
may cause validation errors when preparing the snapshot file as the last step
of a backup.
2020-06-08 10:53:01 -04:00
Gilbert Chen
fe854d469d Error out in the check command if there are 0-size chunks. 2020-06-02 11:37:12 -04:00
Gilbert Chen
9c3122b814 Fixed a bug causing the OneDrive Business token file path to be not saved
The file path was saved under the name `one_token` in Keychain/keyring, rather
than the correct name 'odb_token'.
2020-05-10 20:22:49 -04:00
Gilbert Chen
6ca8b8dff0 Disable snapshot cache when checking chunks
Otherwise every chunk will be stored to the snapshot cache when the `-chunks`
option is specified.
2020-05-10 00:26:47 -04:00
Tet Woo Lee
4ae16dec7f add -persist in check and restore mode (for PR) 2020-05-06 18:39:52 +12:00
Alexandre Lécuyer
dae040681d Call Authenticate() before using Swift storage
Failing to call Authenticate() before using the swift connection will
cause a panic in recent versions of the swift library.
2020-05-05 23:09:28 +02:00
Gilbert Chen
835af11334 Fixed a bug in ssh login with encrypted private key
Check the type of the returned error instead of the error message to
determine if the private key file is encrypted by a passphrase.
2020-04-17 15:55:30 -04:00
gilbertchen
8c80470c29 Merge pull request #593 from freaksdotcom/readall
Call ReadAll() on the body io.ReadCloser to allow the http keepalive connection to be reused.
2020-04-09 21:12:41 -04:00
Gilbert Chen
bcb889272d Add test for Google Shared Drive 2020-04-09 00:04:30 -04:00
Gilbert Chen
79d8654a12 Allow the name of Google Shared Drive to be used in the storage url
The previous PR only accepts the id of the shared drive, which is not very
memorable.  This commit makes it able to specify the drive by id or by name.
2020-04-08 23:59:15 -04:00
Brandon High
6bf0d2265c Call ReadAll() on the body io.ReadCloser to allow the http keepalive connection to be reused. 2020-04-07 22:07:41 -07:00
Gilbert Chen
749db78a1f Implemented a global option to suppress logs by ids
You can now use -suppress LOGID or -s LOGID to not print logs with the given
ids.  This is a global option which means it is applicable to all commands.
It can be specified more than once.
2020-04-07 23:22:17 -04:00
Gilbert Chen
0a51bd8d1a Make log.Printf print to Duplicacy's logging system 2020-04-07 13:52:54 -04:00
Gilbert Chen
7208adbce2 Access Google Drive via service account.
Our GCS backend already supports service account.  This just copies relevant
code from there.
2020-04-06 23:25:47 -04:00
gilbertchen
e827662869 Merge pull request #579 from rsanger/master
Add support for Shared Google Drives
2020-04-06 22:55:46 -04:00
gilbertchen
57dd5ba927 Merge pull request #590 from fbarthez/macos_sync_error
Fix "Failed to upload the chunk ... sync ...: operation not supported"
2020-04-06 22:54:44 -04:00
Gilbert Chen
57cd20bb84 Fixed the condition to show 'chunks are encrypted' messages
'File/Metadata chunks are encrypted' were always shown even if the storage
wasn't encrypted.
2020-04-06 12:22:47 -04:00
Gilbert Chen
0e970da222 Fixed build errors in tests caused by snapshotManager.CheckSnapshots 2020-04-06 12:19:42 -04:00
Gilbert Chen
e880636502 Fixed test build errors caused by the prototype change in CheckSnapshots() 2020-03-30 17:49:26 -04:00
Gilbert Chen
810303ce25 Fail the backup if the repository can't be accessed or there are no files
This is mainly to avoid creating an empty snapshot when a drive/share is
not mounted which causes the subsequent backup to scan all files again.
2020-03-30 17:44:53 -04:00
Gilbert Chen
ffac83dd80 Assume the signed certificate of a ssh key file has the suffix '-cert.pub'.
So if the ssh key file is 'mykey' then Duplicacy will check if the signed
certificate can be loaded from the file 'mykey-cert.pub'.  This avoids the
use of another preference variable 'ssh_cert_file'.
2020-03-25 23:50:35 -04:00
gilbertchen
05674871fe Merge pull request #547 from philband/ssh_signed_certificate
Add option to use a ssh key signed with a certificate to authenticate
2020-03-25 23:14:30 -04:00
Gilbert Chen
22d6f3abfc Add a -chunks option to the check command to verify the integrity of chunks
This option will download and verify every chunk.  Unlike the -files option,
this option only downloads each chunk once.  There is also a new -threads
option to use multiple threads to download chunks.
2020-03-24 20:58:45 -04:00
Gilbert Chen
d26ffe2cff Add support for OneDrive for Business
The new storage prefix for OneDrive for Business is odb://

The token file can be downloaded from https://duplicacy.com/odb_start

OneDrive for Business requires basically the same set of API calls with
different endpoints.  However, one major difference is that for files larger
than 4MB, an upload session must be created first which is then used to upload
the file content.  Other than that, there are a few minor differences such as
creating an existing directory, or moving files to a non-existent directory.
2020-03-19 14:59:26 -04:00
Fabian Peters
a35f6c27be Fix "Failed to upload the chunk ... sync ...: operation not supported" issue when using SMB on MacOS. This is done by inspecting the error type and returning the error only if its operation is "sync" and the type is "operation not supported".
Note: this change is my first ever foray into go and based simply on the information provided in: https://forum.duplicacy.com/t/failed-to-upload-the-chunk-operation-not-supported/2875/11
2020-03-16 15:56:31 +01:00
Gilbert Chen
6699e2f440 Fixed a bug that disabled RSA when copying from a non RSA-encrypted storage.
When copying to an RSA-encrypted storage, we relied on the RSA encryption
version to determine if a chunk is a snapshot chunk or a file chunk.  This is
wrong when the source storage is not encrypted or not RSA-encrypted.  There
is a more reliable to determine if a chunk is a snapshot chunk or not.
2020-03-13 20:13:27 -04:00
Gilbert Chen
733b68be2c Do not take an RSA private key if the storage wasn't RSA encrypted. 2020-03-11 23:14:01 -04:00
gilbertchen
a0a07d18cc Merge pull request #589 from fracai/b2_download_url
support downloading from a custom URL pointed at B2
2020-03-05 22:00:53 -05:00
Gilbert Chen
a6ce64e715 Fixed handling of repository ids with spaces in the b2 backend
Usually a repository id should not contain spaces or other non-alphanum
characters, but if it does we should be able to handle it correctly.  This
commit fixes the b2 backend to convert file names in a proper way.
2020-03-05 14:45:09 -05:00
Arno Hautala
499b612a0d moving download url config from a key to the storage url pattern 2020-02-25 20:53:19 -05:00
Arno Hautala
46ce0ba1fb support downloading from a custom URL pointed at B2 2020-02-22 22:12:36 -05:00
Gilbert Chen
cc88abd547 Fixed a bug that caused all copied chunks to be RSA encrypted
The field encryptionVersion in the Chunk struct is supposed to pass the status
of RSA encrytpion from a source chunk to a destination chunk in a copy command.
This field needs to be a 3-state boolean in order to pass the status correctly.
2020-02-13 14:03:07 -05:00