Commit Graph

103 Commits

Author SHA1 Message Date
Michael Meidlinger 9d7a892355
Merge 32cfd1ed52 into 18b8f9ab4b 2024-03-10 12:51:12 -07:00
Teal Dulcet 6a47133e3f Fixed F811 (redefined-while-unused): Redefinition of unused `sys` from line 10 2024-03-10 07:56:49 -04:00
Teal Dulcet 7f456d8e8b Fixed ISC002 (multi-line-implicit-string-concatenation): Implicitly concatenated string literals over multiple lines 2024-03-10 07:56:49 -04:00
Teal Dulcet e466b9bb53 Fixed RUF005 (collection-literal-concatenation) 2024-03-10 07:56:49 -04:00
Teal Dulcet 0e9193651d Fixed PLW1514 (unspecified-encoding): `open` in text mode without explicit `encoding` argument 2024-03-10 07:56:49 -04:00
Teal Dulcet 3111cf56de Fixed EM102 (f-string-in-exception): Exception must not use an f-string literal, assign to variable first 2024-03-10 07:56:49 -04:00
Teal Dulcet 51dc7615f7 Fixed RSE102 (unnecessary-paren-on-raise-exception): Unnecessary parentheses on raised exception 2024-03-10 07:54:51 -04:00
Teal Dulcet 13b38cc04d Fixed F841 (unused-variable) 2024-03-10 07:54:51 -04:00
Teal Dulcet 2b426851f9 Fixed UP032 (f-string): Use f-string instead of `format` call 2024-03-10 07:54:51 -04:00
Teal Dulcet b7f70b17ac Fixed RET504 (unnecessary-assign) 2024-03-10 07:54:51 -04:00
Teal Dulcet 6bfd1e5140 Fixed W293 (blank-line-with-whitespace): Blank line contains whitespace 2024-03-10 07:54:51 -04:00
Teal Dulcet 555ecc1ebb Fixed PIE810 (multiple-starts-ends-with): Call `startswith` once with a `tuple` 2024-03-10 07:54:51 -04:00
Teal Dulcet dd61844ced Fixed EM101 (raw-string-in-exception): Exception must not use a string literal, assign to variable first 2024-03-10 07:54:51 -04:00
Teal Dulcet 49124cc9ca Fixed PLR6201 (literal-membership): Use a `set` literal when testing for membership 2024-03-10 07:54:51 -04:00
Teal Dulcet cb922ec286 Fixed UP015 (redundant-open-modes): Unnecessary open mode parameters 2024-03-10 07:54:49 -04:00
Teal Dulcet 0ee64f2fe8 Fixed F401 (unused-import) 2024-03-10 07:54:21 -04:00
Joshua Tauberer a966913963
Fix command line arguments for duplicity 2.1 (#2301) 2023-09-02 15:54:16 -04:00
Joshua Tauberer 08defb12be Add a new backup.py command to print the duplicity command to the console to help debugging 2023-09-02 07:49:41 -04:00
Jeff Volkenant 7be687e601 Move source and target positional arguments to the end, required for Duplicity 2.1.0
(Modified by JT.)
2023-09-02 07:28:48 -04:00
Dmytro Kyrychuk 3148c621d2
Fix issue with slash (/) characters in B2 Application Key (#2281)
Urlencode B2 Application Key when saving configuration, urldecode it
back when reading. Duplicity accepts urlencoded target directly, no
decoding is necessary when backup is performed.

Resolve #1964
2023-09-02 07:03:24 -04:00
Darren Sanders c034b0f789 Fix how the value is being passed for the gpg-options parameter
Duplicity v2.1.0 backups are failing with the error:
"... --gpg-options expected one argument".

The issue is that duplicity v2.1.0 began using the argparse Python
library and the parse_known_args function. This function
interprets the argument being passed, "--cipher-algo=AES256",
as an argument name (because of the leading '-') and not as an
argument value. Because of that it exits with an error and
reports that the --gpg-options arg is missing its value.

Adding an extra set of quotes around this string causes
parse_known_args to interpret the string as an argument
value.
2023-08-30 16:34:17 -07:00
Michael Meidlinger 32cfd1ed52 Change how backup.py script deals with S3 backups.
In case that no static AWS credentials are specified, we try to create the boto3 client without explicitly passing static credentials. This way, we can benedit from dynamic credentials in AWS environments (e.g. using EC2 instance roles)
2023-05-19 13:25:18 +02:00
Joshua Tauberer 51ed030917 Allow setting the S3 region name in backup settings to pass to duplicity
It's stuffed inside the username portion of the target URL. We already mangle the target before passing it to duplicity so there wasn't a need for a new field.

Fixes the issue raised in #2200, #2216.
2023-05-13 07:00:29 -04:00
Hugh Secker-Walker 4408cb1fba
fix(rsync-backup): Provide default port 22 for rsync usage in backup.py (#2226)
Co-authored-by: Hugh Secker-Walker <hsw+miac@hodain.net>
2023-01-28 11:04:46 -05:00
Hugh Secker-Walker a2565227f2
feat(rsync-port): Add support for non-standard ssh port for rsync backup (#2208) 2023-01-15 10:03:05 -05:00
Hugh Secker-Walker 820a39b865
chore(python open): Refactor open and gzip.open to use context manager (#2203)
Co-authored-by: Hugh Secker-Walker <hsw+miac@hodain.net>
2023-01-15 08:28:43 -05:00
jvolkenant b8feb77ef4
Move postgrey database under $STORAGE_ROOT (#2077) 2022-09-24 13:17:55 -04:00
Steve Hay 7cda439c80
Port boto to boto3 and fix asyncio issue in the management daemon (#2156)
Co-authored-by: Steve Hay <hay.steve@gmail.com>
2022-09-17 07:57:12 -04:00
Joshua Tauberer 78d71498fa Upgrade from PHP 7.2 to 8.0 for Ubuntu 22.04
* Add the PHP PPA.
* Specify the version when invoking the php CLI.
* Specify the version in package names.
* Update paths to 8.0 (using a variable in the setup scripts).
* Update z-push's php-xsl dependency to php8.0-xml.
* php-json is now built-into PHP.

Although PHP 8.1 is the stock version in Ubuntu 22.04, it's not supported by Nextcloud yet, and it likely will never be supported by the the version of Nextcloud that succeeds the last version of Nextcloud that supports PHP 7.2, and we have to install the next version so that an upgrade is permitted, so skipping to PHP 8.1 may not be easily possible.
2022-07-28 14:02:46 -04:00
Joshua Tauberer b41a0ad80e Drop some hacks that we needed for Ubuntu 18.04
* certbot's PPA is no longer needed because a recent version is now included in the Ubuntu respository.
* Un-pin b2sdk (reverts 69d8fdef99 and d829d74048).
* Revert boto+s3 workaround for duplicity (partial revert of 99474b348f).
* Revert old "fix boto 2 conflict on Google Compute Engine instances" (cf33be4596) which is probably no longer needed.
2022-07-28 14:02:46 -04:00
Joshua Tauberer 99474b348f Update backup to be compatible with duplicity 0.8.23
We were using duplicity 0.8.21-ppa202111091602~ubuntu1 from the duplicity PPA probably until June 5, which is when my box automatically updated to 0.8.23-ppa202205151528~ubuntu18.04.1. Starting with that version, two changes broke backups:

* The default s3 backend was changed to boto3. But boto3 depends on the AWS SDK which does not support Ubuntu 18.04, so we can't install it. Instead, we map s3: backup target URLs to the boto+s3 scheme which tells duplicity to use legacy boto. This should be reverted when we can switch to boto3.
* Contrary to the documentation, the s3 target no longer accepts a S3 hostname in the URL. It now reads the bucket from the hostname part of the URL. So we now drop the hostname from our target URL before passing it to duplicity and we pass the endpoint URL in a separate command-line argument. (The boto backend was dropped from duplicity's "uses_netloc" in 74d4cf44b1 (f5a07610d36bd242c3e5b98f8348879a468b866a_37_34), but other changes may be related.)

The change of target URL (due to both changes) seems to also cause duplicity to store cached data in a different directory within $STORAGE_ROOT/backup/cache, so on the next backup it will re-download cached manifest/signature files. Since the cache directory will still hold the prior data which is no longer needed, it might be a good idea to clear out the cache directory to save space. A system status checks message is added about that.

Fixes #2123
2022-06-12 08:17:48 -04:00
Joshua Tauberer 8bebaf6a48 Simplify duplicity command line by omitting rsync options if the backup target type is not rsync 2022-06-11 15:12:31 -04:00
Hilko 8664afa997
Implement Backblaze for Backup (#1812)
* Installing b2sdk for b2 support
* Added Duplicity PPA so the most recent version is used
* Implemented list_target_files for b2
* Implemented b2 in frontend
* removed python2 boto package
2020-11-26 07:13:31 -05:00
Matthias Hähnel cd62fd9826 Update usage hint in backup.py (#1662)
removed explicit call of the system python, cause the file has a shebang with the mail-in-a-box shipped python. 
for me the system python complaint, that it is missing some modules
2019-11-23 08:04:22 -05:00
captainwasabi c4cb828f65 Fix rsync backup options string: extraneous single quotes causing problems (#1629)
The resulting command had nested single quotes which doesn't work

I think this fixes all/most of the issues in #1627.  I am getting a full backup, then the next time it's run I get an incremental.  running from the CLI with --status looks good, --verify looks good, and --list looks good.
2019-08-13 05:57:05 -04:00
captainwasabi 0657f9e875 add proper check for DNS error in list_target_files (#1625)
The elif needed to check to see if the string was in the listing of results of the shell command.  As it was the conditional was just the string which always evaluates to true and was therefore giving a misleading error message.
2019-08-13 05:47:11 -04:00
Pascal Garber 77b2246010 Backup Amazon S3: Added support for custom endpoints (#1427) 2019-05-12 05:09:30 -07:00
Joshua Tauberer 86e2cfb6c8 remove old duplicity migration code from 2015, see 42322455 2018-11-30 10:46:54 -05:00
Joshua Tauberer f739662392 duplicity started creating signature files with invalid filenames, fixes #1431 2018-10-13 16:16:30 -04:00
Joshua Tauberer bbfa01f33a update to PHP 7.2
* drop the ondrej/php PPA since PHP 7.x is available directly from Ubuntu 18.04
* intall PHP 7.2 which is just the "php" package in Ubuntu 18.04
* some package names changed, some unnecessary packages are no longer provided
* update paths
2018-10-03 13:00:15 -04:00
hlxnd f420294819 Use ISO 8601 on backups table dates. 2018-08-05 15:26:45 +02:00
Joshua Tauberer 0088fb4553 install Python 3 packages in a virtualenv
The cryptography package has created all sorts of installation trouble over the last few years, probably because of mismatches between OS-installed packages and pip-installed packages. Using a virtualenv for all Python packages used by the management daemon should make sure everything is consistent.

See #1298, see #1264.
2018-01-15 13:27:04 -05:00
Michael Kroes a16855ecf0 Backup script should now stop php7.0-fpm instead of php5-fpm (#1206) 2017-07-17 09:45:40 -04:00
yodax b66f12dd4c Fix rsync backup. The path was not append properly 2017-04-17 07:25:47 -04:00
yodax 6e04eb490f Add check to prevent division by zero during backup status 2017-04-17 07:25:47 -04:00
Ian Beringer 89222d519a Fix date delta display for deltas greater than 1 year (#1099) 2017-02-15 18:24:32 -05:00
Joshua Tauberer a081d04082 move the custom exclusive process code from utils.py into a new python package named exclusiveprocess 2017-01-15 11:02:23 -05:00
guyzmo 34d58fb720 Fix/rsync issues (#1036)
* Fixed issue with relative path for rsync relative names

Actually using the parsed URL `path` part, instead of doing a lousy split().
Renamed the `p` variable into something more sensible (`target`).

Fixes: #1019

* Added more verbose error messages upon rsync failures

fixes #1033

* Added command to test file listing
2016-12-17 09:29:48 -05:00
Joshua Tauberer 96b3a29800 rsync backup broke other things 2016-11-12 09:59:06 -05:00
guyzmo 041b5f883f Support for rsync+ssh backup target (#678)
* Added support for backup to a remote server using rsync

* updated web interface to get data from user
* added way to list files from server

It’s not using the “username” field of the yaml configuration
file to minimise the amount of patches needed. So the username
is actually sorted within the rsync URL.

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* Added ssh key generation upon installation for root user.

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* Removed stale blank lines, and fixed typo

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* fix backup-location lines, by switching it from id to class

* Various web UI fixes

- fixed user field being shadowed ;
- fixed settings reading comparaison ;
- fixed forgotten min-age field.

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* Added SSH Public Key shown on the web interface UI

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* trailing spaces.

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* fixed the extraneous environment

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* Updated key setup

- made key lower in bits, but stronger (using -a option),
- made ssh-keygen run in background using nohup,
- added independent key file, as id_rsa_miab,
- added ssh-options to all duplicity calls to use the id_rsa_miab keyfile,
- changed path to the public key display

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* added rsync options for ssh identity support

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* removed strict host checking for all backup operations

Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>

* Remove nohup from ssh-keygen so errors aren't hidden. Also only generate a key if none exists yet

* Add trailing slash when checking a remote backup. Also check if we actually can read the remote size

* Factorisation of the repeated rsync/ssh options

cf https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478919

* Updated message SSH key creation

https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478886
2016-11-12 09:28:55 -05:00