* Add the PHP PPA.
* Specify the version when invoking the php CLI.
* Specify the version in package names.
* Update paths to 8.0 (using a variable in the setup scripts).
* Update z-push's php-xsl dependency to php8.0-xml.
* php-json is now built-into PHP.
Although PHP 8.1 is the stock version in Ubuntu 22.04, it's not supported by Nextcloud yet, and it likely will never be supported by the the version of Nextcloud that succeeds the last version of Nextcloud that supports PHP 7.2, and we have to install the next version so that an upgrade is permitted, so skipping to PHP 8.1 may not be easily possible.
* certbot's PPA is no longer needed because a recent version is now included in the Ubuntu respository.
* Un-pin b2sdk (reverts 69d8fdef99 and d829d74048).
* Revert boto+s3 workaround for duplicity (partial revert of 99474b348f).
* Revert old "fix boto 2 conflict on Google Compute Engine instances" (cf33be4596) which is probably no longer needed.
The resolve method disables resolving relative names by default. This change probably makes a7710e90 unnecessary. @JoshData added some additional changes from query to resolve.
We were using duplicity 0.8.21-ppa202111091602~ubuntu1 from the duplicity PPA probably until June 5, which is when my box automatically updated to 0.8.23-ppa202205151528~ubuntu18.04.1. Starting with that version, two changes broke backups:
* The default s3 backend was changed to boto3. But boto3 depends on the AWS SDK which does not support Ubuntu 18.04, so we can't install it. Instead, we map s3: backup target URLs to the boto+s3 scheme which tells duplicity to use legacy boto. This should be reverted when we can switch to boto3.
* Contrary to the documentation, the s3 target no longer accepts a S3 hostname in the URL. It now reads the bucket from the hostname part of the URL. So we now drop the hostname from our target URL before passing it to duplicity and we pass the endpoint URL in a separate command-line argument. (The boto backend was dropped from duplicity's "uses_netloc" in 74d4cf44b1 (f5a07610d36bd242c3e5b98f8348879a468b866a_37_34), but other changes may be related.)
The change of target URL (due to both changes) seems to also cause duplicity to store cached data in a different directory within $STORAGE_ROOT/backup/cache, so on the next backup it will re-download cached manifest/signature files. Since the cache directory will still hold the prior data which is no longer needed, it might be a good idea to clear out the cache directory to save space. A system status checks message is added about that.
Fixes#2123
When building the part of the report about the current DS records founded, they are added in the same order as they were received when calling query_dns(), which can differ from run to run. This was making the difflib.SequenceMatcher() method to find the same line removed and added one line later, and sending an Status Checks Change Notice email with the same line added and removed when there was actually no real changes.
DAVdroid has been renamed to DAVx⁵ and price increased from $3.69 to $5.99.
CardDAV-Sync free is no longer in beta.
CalDAV-Sync price increased from $2.89 to $2.99.
By not advertising SMTPUTF8 support at the start, senders may opt to transmit recipient internationalized domain names in IDNA form instead, which will be deliverable.
Incoming mail with internationalized domains was probably working prior to our move to Ubuntu 18.04 when postfix's SMTPUTF8 support became enabled by default.
The previous commit is retained because Mail-in-a-Box users might prefer to keep SMTPUTF8 on for outbound mail, if they are not using internationalized domains for email, in which case the previous commit fixes the 'relay access denied' error even if the emails aren't deliverable.
When an email is received by Postfix using SMTPUTF8 and the recipient domain is a Unicode internationalized domain, it was failing to be delivered (bouncing with 'relay access denied') because our users and aliases tables only store ASCII (IDNA) forms of internationalized domains. In this commit, domain maps are added to the auto_aliases table from the Unicode form of each mail domain to its IDNA form, if those forms are different. The Postfix domains query is updated to look at the auto_aliases table now as well, since it is the only table with Unicode forms of the mail domains.
However, mail delivery is still not working since the Dovecot LMTP server does not support SMTPUTF8, and mail still bounces but with an error that SMTPUTF8 is not supported.
They really should never have been conflated with the user-provided aliases.
Update the postfix alias map to query the automatically generated aliases with lowest priority.
The /admin/munin routes used the same Authorization: header logic as the other API routes, but they are browsed directly in the browser because they are handled as static pages or as a proxy to a CGI script.
This required users to enter their email username/password for HTTP basic authentication in the standard browser auth prompt, which wasn't ideal (and may leak the password in browser storage). It also stopped working when MFA was enabled for user accounts.
A token is now set in a cookie when visiting /admin/munin which is then checked in the routes that proxy the Munin pages. The cookie's lifetime is kept limited to limit the opportunity for any unknown CSRF attacks via the Munin CGI script.
And write MIAB dns zone config into /etc/nsd/nsd.conf.d/zones.conf. Delete lingering old zones.conf file.
Co-authored-by: Joshua Tauberer <jt@occams.info>
* Use "smart invert" for dark mode
Signed-off-by: Elsie Hupp <9206310+elsiehupp@users.noreply.github.com>
* Add more contrast to form controls
Co-authored-by: Joshua Tauberer <jt@occams.info>
* When logged out, no menu items are shown.
* When logged in, Log Out is shown.
* When logged in as an admin, the remaining menu items are also shown.
* When logged in as a non-admin, the mail and contacts/calendar instruction pages are shown.
Fixes#1987
Since the session cache clears keys after a period of time, this fixes#1821.
Based on https://github.com/mail-in-a-box/mailinabox/pull/2012, and so:
Co-Authored-By: NewbieOrange <NewbieOrange@users.noreply.github.com>
Also fixes#2029 by not revealing through the login failure error message whether a user exists or not.
* Rename the 'master' API key to be called the 'system' API key
* Generate the key using the Python secrets module which is meant for this
* Remove some debugging helper code which will be obsoleted by the upcoming changes for session keys
local/domain parts of email address should be standard ASCII or
UTF-8. Some email addresses contain extended ASCII, leading to
decode failure by the UTF-8 codec (and thus failure of the
Usage-Report script)
This change allows maillog parsing to continue over lines
containing such addresses
Port 465 with "implicit" (i.e. always-on) TLS is a more secure approach than port 587 with explicit (i.e. optional and only on with STARTTLS). Although we reject credentials on port 587 without STARTTLS, by that point credentials have already been sent.
All A/AAAA-resolvable domains that don't send or receive mail should have these null records.
This simplifies the handling of domains a bit by handling automatically generated subdomains more like other domains.
* Stop generating RSASHA1-NSEC3-SHA1 keys on new installs since it is no longer recommended, but preserve the key on existing installs so that we continue to sign zones with existing keys to retain the chain of trust with existing DS records.
* Start generating ECDSAP256SHA256 keys during setup, the current best practice (in addition to RSASHA256 which is also ok). See https://www.iana.org/assignments/dns-sec-alg-numbers/dns-sec-alg-numbers.xhtml#dns-sec-alg-numbers-1 and https://www.cloudflare.com/dns/dnssec/ecdsa-and-dnssec/.
* Sign zones using all available keys rather than choosing just one based on the TLD to enable rotation/migration to the new key and to give the user some options since not every registrar/TLD supports every algorithm.
* Allow a user to drop a key from signing specific domains using DOMAINS= in our key configuration file. Signing the zones with extraneous keys may increase the size of DNS responses, which isn't ideal, although I don't know if this is a problem in practice. (Although a user can delete the RSASHA1-NSEC3-SHA1 key file, the other keys will be re-generated on upgrade.)
* When generating zonefiles, add a hash of all of the DNSSEC signing keys so that when the keys change the zone is definitely regenerated and re-signed.
* In status checks, if DNSSEC is not active (or not valid), offer to use all of the keys that have been generated (for RSASHA1-NSEC3-SHA1 on existing installs, RSASHA256, and now ECDSAP256SHA256) with all digest types, since not all registers support everything, but list them in an order that guides users to the best practice.
* In status checks, if the deployed DS record doesn't use a ECDSAP256SHA256 key, prompt the user to update their DS record.
* In status checks, if multiple DS records are set, only fail if none are valid. If some use ECDSAP256SHA256 and some don't, remind the user to delete the DS records that don't.
* Don't fail if the DS record uses the SHA384 digest (by pre-generating a DS record with that digest type) but don't recommend it because it is not in the IANA mandatory list yet (https://www.iana.org/assignments/ds-rr-types/ds-rr-types.xhtml).
See #1953
This reverts commit b1d703a5e7 and adds python3-setuptools per the first version of #1899 which fixes an installation error for the b2sdk Python package.
* Fixed#1894 log date over year change, START_DATE < END_DATE now.
* Corrected mail_log.py argument help and message.
Co-authored-by: Jarek <jarek@box.jurasz.de>
* Installing b2sdk for b2 support
* Added Duplicity PPA so the most recent version is used
* Implemented list_target_files for b2
* Implemented b2 in frontend
* removed python2 boto package
* add user interface for managing 2fa
* update user schema with 2fa columns
* implement two factor check during login
* Use pyotp for validating TOTP codes
* also implements resynchronisation support via `pyotp`'s `valid_window option
* Update API route naming, update setup page
* Rename /two-factor-auth/ => /2fa/
* Nest totp routes under /2fa/totp/
* Update ids and methods in panel to allow for different setup types
* Autofocus otp input when logging in, update layout
* Extract TOTPStrategy class to totp.py
* this decouples `TOTP` validation and storage logic from `auth` and moves it to `totp`
* reduce `pyotp.validate#valid_window` from `2` to `1`
* Update OpenApi docs, rename /2fa/ => /mfa/
* Decouple totp from users table by moving to totp_credentials table
* this allows implementation of other mfa schemes in the future (webauthn)
* also makes key management easier and enforces one totp credentials per user on db-level
* Add sqlite migration
* Rename internal validate_two_factor_secret => validate_two_factor_secret
* conn.close() if mru_token update can't .commit()
* Address review feedback, thanks @hija
* Use hmac.compare_digest() to compare mru_token
* Safeguard against empty mru_token column
* hmac.compare_digest() expects arguments of type string, make sure we don't pass None
* Currently, this cannot happen but we might not want to store `mru_token` during setup
* Do not log failed login attempts for MissingToken errors
* Due to the way that the /login UI works, this persists at least one failed login each time a user logs into the admin panel. This in turn triggers fail2ban at some point.
* Add TOTP secret to user_key hash
thanks @downtownallday
* this invalidates all user_keys after TOTP status is changed for user
* after changing TOTP state, a login is required
* due to the forced login, we can't and don't need to store the code used for setup in `mru_code`
* Typo
* Reorganize the MFA backend methods
* Reorganize MFA front-end and add label column
* Fix handling of bad input when enabling mfa
* Update openAPI docs
* Remove unique key constraint on foreign key user_id in mfa table
* Don't expose mru_token and secret for enabled mfas over HTTP
* Only update mru_token for matched mfa row
* Exclude mru_token in user key hash
* Rename tools/mail.py to management/cli.py
* Add MFA list/disable to the management CLI so admins can restore access if MFA device is lost
Co-authored-by: Joshua Tauberer <jt@occams.info>
* conf/nginx-primaryonly.conf: Use tabs instead of spaces
* management/web_update.py: Includes the tabs so they display with the correct indentation when added to the local.conf
Co-authored-by: 0pis <0pis>
thanks @downtownallday
* this invalidates all user_keys after TOTP status is changed for user
* after changing TOTP state, a login is required
* due to the forced login, we can't and don't need to store the code used for setup in `mru_code`
* Due to the way that the /login UI works, this persists at least one failed login each time a user logs into the admin panel. This in turn triggers fail2ban at some point.
* hmac.compare_digest() expects arguments of type string, make sure we don't pass None
* Currently, this cannot happen but we might not want to store `mru_token` during setup
* this allows implementation of other mfa schemes in the future (webauthn)
* also makes key management easier and enforces one totp credentials per user on db-level
* Only spawn a thread pool when strictly needed
For --check-primary-hostname, the pool is not used.
When exiting, the other processes are left alive and will hang.
* Acquire pools with the 'with' statement
This will make it so that the HSTS header is sent regardless of the request status code (until this point it would only be sent if "the response code equals 200, 201, 206, 301, 302, 303, 307, or 308." - according to thttp://nginx.org/en/docs/http/ngx_http_headers_module.html#add_header)
These subdomains/records are for automatic configuration of mail clients, but if there are no user accounts on a domain, there is no need to publish a DNS record, provision a TLS certificate, or create an nginx server config block.
When the management commands fail, they can print something to the standard error output.
The administrator would never notice, because it wouldn't be send to him with the usual emails.
Fixes#1763
* Create the mta_sts A/AAAA records even if there is no valid TLS certificate because we can't get a TLS certificate if we don't set up the domains.
* Make the policy id in the TXT record stable by using a hash of the policy file so that the DNS record doesn't change every day, which means no nightly notification and also it allows for longer caching by sending MTAs.
Folks didn't want certificates exposing all of the domains hosted by the server (although this can already be found on the internet).
Additionally, if one domain fails (usually because of a misconfiguration), it would be nice if not everything fails. So grouping them helps with that.
Fixes#690.
with this nginx will keep on proxying requests and serve static content
instead of passing this responsibility to proxied server
Without this the one needs to run an additional server to server static
content on the proxied url
removed explicit call of the system python, cause the file has a shebang with the mail-in-a-box shipped python.
for me the system python complaint, that it is missing some modules
Previously the notification email sent when a box's SSL certificate
is automatically updated said, "Error Provisioning TLS Certificate"
even when there was no error. This changes the subject line to "TLS
Certificate Provisioning Results", which is more accurate.
it is sometimes needed to be able to set axfr to more than just one ip address. This can be done with multiple xfr: in the secondary dns input but if you need to add an entire subnet segment (xxx.xxx.xxx.0/yy) then it will not work.
With this patch it is now possible to use a subnet as input for xfr the same way as if it was an ip address.
The resulting command had nested single quotes which doesn't work
I think this fixes all/most of the issues in #1627. I am getting a full backup, then the next time it's run I get an incremental. running from the CLI with --status looks good, --verify looks good, and --list looks good.
The elif needed to check to see if the string was in the listing of results of the shell command. As it was the conditional was just the string which always evaluates to true and was therefore giving a misleading error message.
* drop the ondrej/php PPA since PHP 7.x is available directly from Ubuntu 18.04
* intall PHP 7.2 which is just the "php" package in Ubuntu 18.04
* some package names changed, some unnecessary packages are no longer provided
* update paths
@joshdata squashed pull request #1398, removed some comments, and added these notes:
* The old init.d script for the management daemon is replaced with a systemd service.
* A systemd service configuration is added to configure permissions for munin on startup.
* nginx SSL settings are updated because nginx's options and defaults have changed, and we now enable http2.
* Automatic SSHFP record generation is updated to know that 22 is the default SSH daemon port, since it is no longer explicit in sshd_config.
* The dovecot-lucene package is dropped because the Mail-in-a-Box PPA where we built the package has not been updated for Ubuntu 18.04.
* The stock postgrey package is installed instead of the one from our PPA (which we no longer support), which loses the automatic whitelisting of DNSWL.org-whitelisted senders.
* Drop memcached and the status check for memcached, which we used to use with ownCloud long ago but are no longer installing.
* Other minor changes.
Passwords must be eight characters long; when passwords are changed via the users page the dialog states that passwords need to be at least four characters but only eight or more are acceptable.
The cryptography package has created all sorts of installation trouble over the last few years, probably because of mismatches between OS-installed packages and pip-installed packages. Using a virtualenv for all Python packages used by the management daemon should make sure everything is consistent.
See #1298, see #1264.
