This fixes an issue where unless compression=auto (or compression=gzip) is specified, using wget to download jquery.min.js by setup/management.sh fails with max redirects exceeded.
The man page says --compression is an experimental option, but it doesn't appear to have any problems.
Note that downloading jquery.min.js with curl works fine when wget without --compression fails.
1. Autodetect encryption-at-rest and run the correct setup in boostrap and /usr/local/bin/mailinabox
2. Fix bug where directories in the local mod folder are improperly executed causing setup to fail
3. Add a remote_nextcloud.sh setup mod cleanup function for cleaner removal of the mod
a deep copy of a dictionary with values of type `dict_keys`, required by the underlying ldap3 python3 library, is no longer allowed. they must be a `list` type
actual error:
Traceback (most recent call last):
File "/mailinabox/setup/migrate.py", line 461, in <module>
run_miabldap_migrations()
File "/mailinabox/setup/migrate.py", line 440, in run_miabldap_migrations
raise e
File "/mailinabox/setup/migrate.py", line 431, in run_miabldap_migrations
migration_func(env)
File "/mailinabox/setup/migrate.py", line 249, in migration_miabldap_1
users=m13.create_users(env, conn, ldap, ldap_base, ldap_users_base, ldap_domains_base)
File "/mailinabox/setup/migration_13.py", line 137, in create_users
dn = add_user(env, ldapconn, ldap_base, ldap_users_base, ldap_domains_base, email, password, privs.split("\n"), totp)
File "/mailinabox/setup/migration_13.py", line 90, in add_user
ldapconn.add(dn, objectClasses, attrs)
File "/usr/lib/python3/dist-packages/ldap3/core/connection.py", line 947, in add
_attributes = deepcopy(attributes) # dict could change when adding objectClass values
File "/usr/lib/python3.10/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/usr/lib/python3.10/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/usr/lib/python3.10/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle 'dict_keys' object
Nextcloud:
* The Nextcloud user_external 1.0.0 package for Nextcloud 21.0.7 isn't available from Nextcloud's releases page, but it's not needed in an intermediate upgrade step (hopefully), so we can skip it.
* Nextcloud updgrade steps should not be elifs because multiple intermediate upgrades may be needed.
* Continue if the user_external backend migration fails. Maybe it's not necessary. It gives a scary error message though.
* Remove a line that removes an old file that hasn't been in use since 2019 and the expectation is that Ubuntu 22.04 installations are on fresh machines.
Backups:
* For duplicity, we now need boto3 for AWS.
The first version supporting PHP 8.0 is Nextcloud 21. Therefore we can add migrations only to Nextcloud 21 forward, and so we only support migrating from Nextcloud 20 (Mail-in-a-Box versions v0.51+). Migration steps through Nextcloud 21 and 22 are added.
Also:
* Fix PHP APUc settings to be before Nextcloud tools are run.
* Add the PHP PPA.
* Specify the version when invoking the php CLI.
* Specify the version in package names.
* Update paths to 8.0 (using a variable in the setup scripts).
* Update z-push's php-xsl dependency to php8.0-xml.
* php-json is now built-into PHP.
Although PHP 8.1 is the stock version in Ubuntu 22.04, it's not supported by Nextcloud yet, and it likely will never be supported by the the version of Nextcloud that succeeds the last version of Nextcloud that supports PHP 7.2, and we have to install the next version so that an upgrade is permitted, so skipping to PHP 8.1 may not be easily possible.
* certbot's PPA is no longer needed because a recent version is now included in the Ubuntu respository.
* Un-pin b2sdk (reverts 69d8fdef99 and d829d74048).
* Revert boto+s3 workaround for duplicity (partial revert of 99474b348f).
* Revert old "fix boto 2 conflict on Google Compute Engine instances" (cf33be4596) which is probably no longer needed.
* Fix path to bind9 startup options file in Ubuntu 22.04.
* tinymce has not been a Roundcube requirement recently and is no longer a package in Ubuntu 22.04
* Upgrade Vagrant box to Ubuntu 22.04
The first version supporting PHP 8.0 is Nextcloud 21. Therefore we can add migrations only to Nextcloud 21 forward, and so we only support migrating from Nextcloud 20 (Mail-in-a-Box versions v0.51+). Migration steps through Nextcloud 21 and 22 are added.
* Add the PHP PPA.
* Specify the version when invoking the php CLI.
* Specify the version in package names.
* Update paths to 8.0 (using a variable in the setup scripts).
* Update z-push's php-xsl dependency to php8.0-xml.
* php-json is now built-into PHP.
Although PHP 8.1 is the stock version in Ubuntu 22.04, it's not supported by Nextcloud yet, and it likely will never be supported by the the version of Nextcloud that succeeds the last version of Nextcloud that supports PHP 7.2, and we have to install the next version so that an upgrade is permitted, so skipping to PHP 8.1 may not be easily possible.
* certbot's PPA is no longer needed because a recent version is now included in the Ubuntu respository.
* Un-pin b2sdk (reverts 69d8fdef99 and d829d74048).
* Revert boto+s3 workaround for duplicity (partial revert of 99474b348f).
* Revert old "fix boto 2 conflict on Google Compute Engine instances" (cf33be4596) which is probably no longer needed.
* Fix path to bind9 startup options file in Ubuntu 22.04.
* tinymce has not been a Roundcube requirement recently and is no longer a package in Ubuntu 22.04
* Upgrade Vagrant box to Ubuntu 22.04
We install b2sdk in two places: Once globally for duplicity (see
9d8fdef9915127f016eb6424322a149cdff25d7 for #2125) and once in
a virtualenv used by our control panel. The latter wasn't pinned
when the former was but should be to fix new Python compatibility
issues.
Anyone who updated Python packages recently (so anyone who upgraded
Mail-in-a-Box) started encountering these issues.
Fixes#2131.
See https://discourse.mailinabox.email/t/backblaze-b2-backup-not-working-since-v57/9231.
Update jails.conf to include IPV6 localhost and external ip to ignoreip line. Update system.sh to include IPV6 address in replacement. See mail-in-a-box#2066 for details.
See [mailinabox issue #2088](https://github.com/mail-in-a-box/mailinabox/issues/2088). This also updates the commit hashes to for anyone updating from NextCloud version 17 (as shown in the related issue) since a different hash is used for tags vs releases.
This was tested and verified to work on a setup previously running v0.44 and then updating to the latest version (v56).
1. `systemctl reload postgrey` is broken, so use restart in that case
2. `systemctl restart nsd` succeeds once /var/log is writable by systemd. However, nsd still fails to write to /var/log/nsd.log if the file already exists
3. the default configuration for rsyslogd is to no longer create a /var/run/rsyslogd.pid file, so use /usr/bin/pidof in that case
The first version supporting PHP 8.0 is Nextcloud 21. Therefore we can add migrations only to Nextcloud 21 forward, and so we only support migrating from Nextcloud 20 (Mail-in-a-Box versions v0.51+). Migration steps through Nextcloud 21 and 22 are added.
(Updated by @JoshData from the original commit which was for Ubuntu 20.04 using PHP 7.4. And although 8.1 seems to be available, it's not supported by Nextcloud yet, and it likely will never be supported by the the version of Nextcloud that succeeds the last version of Nextcloud that supports PHP 7.2, and we have to install the next version so that an upgrade is permitted, so skipping to PHP 8.1 may not be easily possible.)
* Update php-xsl to php8.0-xml for zpush.
* Fix path to bind9 startup options file in Ubuntu 22.04.
* tinymce has not been a Roundcube requirement recently and is no longer a package in Ubuntu 22.04
* Upgrade Vagrant box to Ubuntu 22.04
Upstream is adding handling for utf8 domains by creating a domain alias @utf8 -> @idna. I'm deviating from this approach by setting multiple email address (idna and utf8) per user and alias where a domain contains non-ascii characters. The maildrop (mailbox) remains the same - all mail goes to the user's mailbox regardless of which email address was used. This is more in line with how other systems (eg. active directory), handle multiple email addresses for a single user.
# Conflicts:
# README.md
# management/mailconfig.py
# management/templates/index.html
# setup/dns.sh
# setup/mail-users.sh
By not advertising SMTPUTF8 support at the start, senders may opt to transmit recipient internationalized domain names in IDNA form instead, which will be deliverable.
Incoming mail with internationalized domains was probably working prior to our move to Ubuntu 18.04 when postfix's SMTPUTF8 support became enabled by default.
The previous commit is retained because Mail-in-a-Box users might prefer to keep SMTPUTF8 on for outbound mail, if they are not using internationalized domains for email, in which case the previous commit fixes the 'relay access denied' error even if the emails aren't deliverable.
When an email is received by Postfix using SMTPUTF8 and the recipient domain is a Unicode internationalized domain, it was failing to be delivered (bouncing with 'relay access denied') because our users and aliases tables only store ASCII (IDNA) forms of internationalized domains. In this commit, domain maps are added to the auto_aliases table from the Unicode form of each mail domain to its IDNA form, if those forms are different. The Postfix domains query is updated to look at the auto_aliases table now as well, since it is the only table with Unicode forms of the mail domains.
However, mail delivery is still not working since the Dovecot LMTP server does not support SMTPUTF8, and mail still bounces but with an error that SMTPUTF8 is not supported.
They really should never have been conflated with the user-provided aliases.
Update the postfix alias map to query the automatically generated aliases with lowest priority.
And write MIAB dns zone config into /etc/nsd/nsd.conf.d/zones.conf. Delete lingering old zones.conf file.
Co-authored-by: Joshua Tauberer <jt@occams.info>
Since the session cache clears keys after a period of time, this fixes#1821.
Based on https://github.com/mail-in-a-box/mailinabox/pull/2012, and so:
Co-Authored-By: NewbieOrange <NewbieOrange@users.noreply.github.com>
Also fixes#2029 by not revealing through the login failure error message whether a user exists or not.
Port 465 with "implicit" (i.e. always-on) TLS is a more secure approach than port 587 with explicit (i.e. optional and only on with STARTTLS). Although we reject credentials on port 587 without STARTTLS, by that point credentials have already been sent.
* SC2068: Double quote array expansions to avoid re-splitting elements.
* SC2186: tempfile is deprecated. Use mktemp instead.
* SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.
* SC2102: Ranges can only match single chars (mentioned due to duplicates).
* SC2005: Useless echo? Instead of 'echo $(cmd)', just use 'cmd'.
This change will force everyone to be logged out of Roundcube since the encryption key and cipher won't match anyone's already-set cookie, but this happens anyway after every Mail-in-a-Box update since we generate a new key each time already.
Fixes#1968.
* Stop generating RSASHA1-NSEC3-SHA1 keys on new installs since it is no longer recommended, but preserve the key on existing installs so that we continue to sign zones with existing keys to retain the chain of trust with existing DS records.
* Start generating ECDSAP256SHA256 keys during setup, the current best practice (in addition to RSASHA256 which is also ok). See https://www.iana.org/assignments/dns-sec-alg-numbers/dns-sec-alg-numbers.xhtml#dns-sec-alg-numbers-1 and https://www.cloudflare.com/dns/dnssec/ecdsa-and-dnssec/.
* Sign zones using all available keys rather than choosing just one based on the TLD to enable rotation/migration to the new key and to give the user some options since not every registrar/TLD supports every algorithm.
* Allow a user to drop a key from signing specific domains using DOMAINS= in our key configuration file. Signing the zones with extraneous keys may increase the size of DNS responses, which isn't ideal, although I don't know if this is a problem in practice. (Although a user can delete the RSASHA1-NSEC3-SHA1 key file, the other keys will be re-generated on upgrade.)
* When generating zonefiles, add a hash of all of the DNSSEC signing keys so that when the keys change the zone is definitely regenerated and re-signed.
* In status checks, if DNSSEC is not active (or not valid), offer to use all of the keys that have been generated (for RSASHA1-NSEC3-SHA1 on existing installs, RSASHA256, and now ECDSAP256SHA256) with all digest types, since not all registers support everything, but list them in an order that guides users to the best practice.
* In status checks, if the deployed DS record doesn't use a ECDSAP256SHA256 key, prompt the user to update their DS record.
* In status checks, if multiple DS records are set, only fail if none are valid. If some use ECDSAP256SHA256 and some don't, remind the user to delete the DS records that don't.
* Don't fail if the DS record uses the SHA384 digest (by pre-generating a DS record with that digest type) but don't recommend it because it is not in the IANA mandatory list yet (https://www.iana.org/assignments/ds-rr-types/ds-rr-types.xhtml).
See #1953
Configures opendmarc to send failure reports for domains that request them, including when p=none.
The emails are sent as the package default of package name and user@hostname: OpenDMARC Filter <opendmarc@box.example.com>
Note I have been running this for several months with a configuration I did not include in the PR to have reports BCC'd to me (FailureReportsBcc postmaster@example.com). Very low load for my personal server of rarely more than a dozen emails sent out per day.
I am not familiar with editing scripts, so apologies in advance and please feel free to correct me.
roundcube Bug Fixes:
Fix for Cross-Site Scripting (XSS) via HTML messages with malicious CSS content
General Improvements from roundcube's Issue Tracker
This reverts commit b1d703a5e7 and adds python3-setuptools per the first version of #1899 which fixes an installation error for the b2sdk Python package.
This adds a new section to the admin panel called "Activity", that
supplies charts, graphs and details about messages entering and leaving
the host.
A new daemon captures details of system mail activity by monitoring
the /var/log/mail.log file, summarizing it into a sqllite database
that's kept in user-data.
