We were using duplicity 0.8.21-ppa202111091602~ubuntu1 from the duplicity PPA probably until June 5, which is when my box automatically updated to 0.8.23-ppa202205151528~ubuntu18.04.1. Starting with that version, two changes broke backups:
* The default s3 backend was changed to boto3. But boto3 depends on the AWS SDK which does not support Ubuntu 18.04, so we can't install it. Instead, we map s3: backup target URLs to the boto+s3 scheme which tells duplicity to use legacy boto. This should be reverted when we can switch to boto3.
* Contrary to the documentation, the s3 target no longer accepts a S3 hostname in the URL. It now reads the bucket from the hostname part of the URL. So we now drop the hostname from our target URL before passing it to duplicity and we pass the endpoint URL in a separate command-line argument. (The boto backend was dropped from duplicity's "uses_netloc" in 74d4cf44b1 (f5a07610d36bd242c3e5b98f8348879a468b866a_37_34), but other changes may be related.)
The change of target URL (due to both changes) seems to also cause duplicity to store cached data in a different directory within $STORAGE_ROOT/backup/cache, so on the next backup it will re-download cached manifest/signature files. Since the cache directory will still hold the prior data which is no longer needed, it might be a good idea to clear out the cache directory to save space. A system status checks message is added about that.
Fixes#2123
* Installing b2sdk for b2 support
* Added Duplicity PPA so the most recent version is used
* Implemented list_target_files for b2
* Implemented b2 in frontend
* removed python2 boto package
removed explicit call of the system python, cause the file has a shebang with the mail-in-a-box shipped python.
for me the system python complaint, that it is missing some modules
The resulting command had nested single quotes which doesn't work
I think this fixes all/most of the issues in #1627. I am getting a full backup, then the next time it's run I get an incremental. running from the CLI with --status looks good, --verify looks good, and --list looks good.
The elif needed to check to see if the string was in the listing of results of the shell command. As it was the conditional was just the string which always evaluates to true and was therefore giving a misleading error message.
* drop the ondrej/php PPA since PHP 7.x is available directly from Ubuntu 18.04
* intall PHP 7.2 which is just the "php" package in Ubuntu 18.04
* some package names changed, some unnecessary packages are no longer provided
* update paths
The cryptography package has created all sorts of installation trouble over the last few years, probably because of mismatches between OS-installed packages and pip-installed packages. Using a virtualenv for all Python packages used by the management daemon should make sure everything is consistent.
See #1298, see #1264.
* Fixed issue with relative path for rsync relative names
Actually using the parsed URL `path` part, instead of doing a lousy split().
Renamed the `p` variable into something more sensible (`target`).
Fixes: #1019
* Added more verbose error messages upon rsync failures
fixes#1033
* Added command to test file listing
* Added support for backup to a remote server using rsync
* updated web interface to get data from user
* added way to list files from server
It’s not using the “username” field of the yaml configuration
file to minimise the amount of patches needed. So the username
is actually sorted within the rsync URL.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Added ssh key generation upon installation for root user.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Removed stale blank lines, and fixed typo
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* fix backup-location lines, by switching it from id to class
* Various web UI fixes
- fixed user field being shadowed ;
- fixed settings reading comparaison ;
- fixed forgotten min-age field.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Added SSH Public Key shown on the web interface UI
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* trailing spaces.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* fixed the extraneous environment
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Updated key setup
- made key lower in bits, but stronger (using -a option),
- made ssh-keygen run in background using nohup,
- added independent key file, as id_rsa_miab,
- added ssh-options to all duplicity calls to use the id_rsa_miab keyfile,
- changed path to the public key display
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* added rsync options for ssh identity support
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* removed strict host checking for all backup operations
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Remove nohup from ssh-keygen so errors aren't hidden. Also only generate a key if none exists yet
* Add trailing slash when checking a remote backup. Also check if we actually can read the remote size
* Factorisation of the repeated rsync/ssh options
cf https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478919
* Updated message SSH key creation
https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478886
* going from s3 to file target wasn't working
* use 'local' in the config instead of a file: url, for the local target, so it is not path-specific
* break out the S3 fields since users can't be expected to know how to form a URL
* use boto to generate a list of S3 hosts
* use boto to validate that the user input for s3 is valid
* fix lots of html errors in the backup admin
Backup location and maximum age can now be configured in the admin panel.
For now only S3 is supported, but adding other duplicity supported backends should be straightforward.
Joshua Tauberer
Hilko
Matthias Hähnel
captainwasabi
Pascal Garber
hlxnd
Michael Kroes
Ian Beringer
guyzmo
Marc Schiller
Leo Koppelkamm
StevesMonkey