v60.1

This fixes the monthly view calendar items being in random order.

CHANGELOG.md

@@ -1,6 +1,112 @@
 CHANGELOG
 =========
 
+Version 60.1 (October 30, 2022)
+-------------------------------
+
+* A setup issue where the DNS server nsd isn't running at the end of setup is (hopefully) fixed.
+* Nextcloud is updated to 23.0.10 (contacts to 4.2.2, calendar to 3.5.1).
+
+Version 60 (October 11, 2022)
+-----------------------------
+
+This is the first release for Ubuntu 22.04.
+
+**Before upgrading**, you must **first upgrade your existing Ubuntu 18.04 box to Mail-in-a-Box v0.51 or later**, if you haven't already done so. That may not be possible after Ubuntu 18.04 reaches its end of life in April 2023, so please complete the upgrade well before then. (If you are not using Nextcloud's contacts or calendar, you can migrate to the latest version of Mail-in-a-Box from any previous version.)
+
+For complete upgrade instructions, see:
+
+https://discourse.mailinabox.email/t/version-60-for-ubuntu-22-04-is-about-to-be-released/9558
+
+No major features of Mail-in-a-Box have changed in this release, although some minor fixes were made.
+
+With the newer version of Ubuntu the following software packages we use are updated:
+
+* dovecot is upgraded to 2.3.16, postfix to 3.6.4, opendmark to 1.4 (which adds ARC-Authentication-Results headers), and spampd to 2.53 (alleviating a mail delivery rate limiting bug).
+* Nextcloud is upgraded to 23.0.4 (contacts to 4.2.0, calendar to 3.5.0).
+* Roundcube is upgraded to 1.6.0.
+* certbot is upgraded to 1.21 (via the Ubuntu repository instead of a PPA).
+* fail2ban is upgraded to 0.11.2.
+* nginx is upgraded to 1.18.
+* PHP is upgraded from 7.2 to 8.0.
+
+Also:
+
+* Roundcube's login session cookie was tightened. Existing sessions may require a manual logout.
+* Moved Postgrey's database under $STORAGE_ROOT.
+
+Version 57a (June 19, 2022)
+---------------------------
+
+* The Backblaze backups fix posted in Version 57 was incomplete. It's now fixed.
+
+Version 57 (June 12, 2022)
+--------------------------
+
+Setup:
+
+* Fixed issue upgrading from Mail-in-a-Box v0.40-v0.50 because of a changed URL that Nextcloud is downloaded from.
+
+Backups:
+
+* Fixed S3 backups which broke with duplicity 0.8.23.
+* Fixed Backblaze backups which broke with latest b2sdk package by rolling back its version.
+
+Control panel:
+
+* Fixed spurious changes in system status checks messages by sorting DNSSEC DS records.
+* Fixed fail2ban lockout over IPv6 from excessive loads of the system status checks.
+* Fixed an incorrect IPv6 system status check message.
+
+Version 56 (January 19, 2022)
+-----------------------------
+
+Software updates:
+
+* Roundcube updated to 1.5.2 (from 1.5.0), and the persistent_login and CardDAV (to 4.3.0 from 3.0.3) plugins are updated.
+* Nextcloud updated to 20.0.14 (from 20.0.8), contacts to 4.0.7 (from 3.5.1), and calendar to 3.0.4 (from 2.2.0).
+
+Setup:
+
+* Fixed failed setup if a previous attempt failed while updating Nextcloud.
+
+Control panel:
+
+* Fixed a crash if a custom DNS entry is not under a zone managed by the box.
+* Fix DNSSEC instructions typo.
+
+Other:
+
+* Set systemd journald log retention to 10 days (from no limit) to reduce disk usage.
+* Fixed log processing for submission lines that have a sasl_sender or other extra information.
+* Fix DNS secondary nameserver refesh failure retry period.
+
+Version 55 (October 18, 2021)
+-----------------------------
+
+Mail:
+
+* "SMTPUTF8" is now disabled in Postfix. Because Dovecot still does not support SMTPUTF8, incoming mail to internationalized addresses was bouncing. This fixes incoming mail to internationalized domains (which was probably working prior to v0.40), but it will prevent sending outbound mail to addresses with internationalized local-parts.
+* Upgraded to Roundcube 1.5.
+
+Control panel:
+
+* The control panel menus are now hidden before login, but now non-admins can log in to access the mail and contacts/calendar instruction pages.
+* The login form now disables browser autocomplete in the two-factor authentication code field.
+* After logging in, the default page is now a fast-loading welcome page rather than the slow-loading system status checks page.
+* The backup retention period option now displays for B2 backup targets.
+* The DNSSEC DS record recommendations are cleaned up and now recommend changing records that use SHA1.
+* The Munin monitoring pages no longer require a separate HTTP basic authentication login and can be used if two-factor authentication is turned on.
+* Control panel logins are now tied to a session backend that allows true logouts (rather than an encrypted cookie).
+* Failed logins no longer directly reveal whether the email address corresponds to a user account.
+* Browser dark mode now inverts the color scheme.
+
+Other:
+
+* Fail2ban's IPv6 support is enabled.
+* The mail log tool now doesn't crash if there are email addresess in log messages with invalid UTF-8 characters.
+* Additional nsd.conf files can be placed in /etc/nsd.conf.d.
+
 v0.54 (June 20, 2021)
 ---------------------
 

CONTRIBUTING.md

@@ -20,9 +20,9 @@ _If you're seeing an error message about your *IP address being listed in the Sp
 
 ### Modifying your `hosts` file
 
-After a while, Mail-in-a-Box will be available at `192.168.50.4` (unless you changed that in your `Vagrantfile`). To be able to use the web-based bits, we recommend to add a hostname to your `hosts` file:
+After a while, Mail-in-a-Box will be available at `192.168.56.4` (unless you changed that in your `Vagrantfile`). To be able to use the web-based bits, we recommend to add a hostname to your `hosts` file:
 
-    $ echo "192.168.50.4 mailinabox.lan" | sudo tee -a /etc/hosts
+    $ echo "192.168.56.4 mailinabox.lan" | sudo tee -a /etc/hosts
 
 You should now be able to navigate to https://mailinabox.lan/admin using your browser. There should be an initial admin user with the name `me@mailinabox.lan` and the password `12345678`.
 

README.md

@@ -23,7 +23,7 @@ Additionally, this project has a [Code of Conduct](CODE_OF_CONDUCT.md), which su
 In The Box
 ----------
 
-Mail-in-a-Box turns a fresh Ubuntu 18.04 LTS 64-bit machine into a working mail server by installing and configuring various components.
+Mail-in-a-Box turns a fresh Ubuntu 22.04 LTS 64-bit machine into a working mail server by installing and configuring various components.
 
 It is a one-click email appliance. There are no user-configurable setup options. It "just works."
 
@@ -42,6 +42,8 @@ It also includes system management tools:
 * A control panel for adding/removing mail users, aliases, custom DNS records, configuring backups, etc.
 * An API for all of the actions on the control panel
 
+Internationalized domain names are supported and configured easily (but SMTPUTF8 is not supported, unfortunately).
+
 It also supports static website hosting since the box is serving HTTPS anyway. (To serve a website for your domains elsewhere, just add a custom DNS "A" record in you Mail-in-a-Box's control panel to point domains to another server.)
 
 For more information on how Mail-in-a-Box handles your privacy, see the [security details page](security.md).
@@ -52,13 +54,13 @@ Installation
 
 See the [setup guide](https://mailinabox.email/guide.html) for detailed, user-friendly instructions.
 
-For experts, start with a completely fresh (really, I mean it) Ubuntu 18.04 LTS 64-bit machine. On the machine...
+For experts, start with a completely fresh (really, I mean it) Ubuntu 22.04 LTS 64-bit machine. On the machine...
 
 Clone this repository and checkout the tag corresponding to the most recent release:
 
 	$ git clone https://github.com/mail-in-a-box/mailinabox
 	$ cd mailinabox
-	$ git checkout v0.54
+	$ git checkout v60.1
 
 Begin the installation.
 

Vagrantfile

@@ -2,14 +2,14 @@
 # vi: set ft=ruby :
 
 Vagrant.configure("2") do |config|
-  config.vm.box = "ubuntu/bionic64"
+  config.vm.box = "ubuntu/jammy64"
 
   # Network config: Since it's a mail server, the machine must be connected
   # to the public web. However, we currently don't want to expose SSH since
   # the machine's box will let anyone log into it. So instead we'll put the
   # machine on a private network.
   config.vm.hostname = "mailinabox.lan"
-  config.vm.network "private_network", ip: "192.168.50.4"
+  config.vm.network "private_network", ip: "192.168.56.4"
 
   config.vm.provision :shell, :inline => <<-SH
     # Set environment variables so that the setup script does

api/mailinabox.yml

@@ -54,24 +54,24 @@ tags:
       System operations, which include system status checks, new version checks
       and reboot status.
 paths:
-  /me:
+  /login:
-    get:
+    post:
       tags:
         - User
-      summary: Get user information
+      summary: Exchange a username and password for a session API key.
       description: |
-        Returns user information. Used for user authentication.
+        Returns user information and a session API key.
 
         Authenticate a user by supplying the auth token as a base64 encoded string in
         format `email:password` using basic authentication headers.
 
         If successful, a long-lived `api_key` is returned which can be used for subsequent
-        requests to the API.
+        requests to the API in place of the password.
-      operationId: getMe
+      operationId: login
       x-codeSamples:
         - lang: curl
           source: |
-            curl -X GET "https://{host}/admin/me" \
+            curl -X POST "https://{host}/admin/login" \
               -u "<email>:<password>"
       responses:
         200:
@@ -92,6 +92,26 @@ paths:
                     privileges:
                       - admin
                     status: ok
+  /logout:
+    post:
+      tags:
+        - User
+      summary: Invalidates a session API key.
+      description: |
+        Invalidates a session API key so that it cannot be used after this API call.
+      operationId: logout
+      x-codeSamples:
+        - lang: curl
+          source: |
+            curl -X POST "https://{host}/admin/logout" \
+              -u "<email>:<session_key>"
+      responses:
+        200:
+          description: Successful operation
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/LogoutResponse'
   /system/status:
     post:
       tags:
@@ -1803,7 +1823,7 @@ components:
 
         The `access-token` is comprised of the Base64 encoding of `username:password`.
         The `username` is the mail user's email address, and `password` can either be the mail user's
-        password, or the `api_key` returned from the `getMe` operation.
+        password, or the `api_key` returned from the `login` operation.
 
         When using `curl`, you can supply user credentials using the `-u` or `--user` parameter.
   requestBodies:
@@ -2705,3 +2725,8 @@ components:
           nullable: true
     MfaDisableSuccessResponse:
       type: string
+    LogoutResponse:
+      type: object
+      properties:
+        status:
+          type: string

conf/fail2ban/jails.conf

@@ -5,7 +5,7 @@
 # Whitelist our own IP addresses. 127.0.0.1/8 is the default. But our status checks
 # ping services over the public interface so we should whitelist that address of
 # ours too. The string is substituted during installation.
-ignoreip = 127.0.0.1/8 PUBLIC_IP
+ignoreip = 127.0.0.1/8 PUBLIC_IP ::1 PUBLIC_IPV6
 
 [dovecot]
 enabled = true

conf/mailinabox.service

@@ -4,6 +4,7 @@ After=multi-user.target
 
 [Service]
 Type=idle
+IgnoreSIGPIPE=False
 ExecStart=/usr/local/lib/mailinabox/start
 
 [Install]

conf/nginx-top.conf

@@ -7,6 +7,6 @@
 ##       your own --- please do not ask for help from us.
 
 upstream php-fpm {
-	server unix:/var/run/php/php7.2-fpm.sock;
+	server unix:/var/run/php/php8.0-fpm.sock;
 }
 

management/auth.py

@@ -1,6 +1,7 @@
-import base64, os, os.path, hmac, json
+import base64, os, os.path, hmac, json, secrets
+from datetime import timedelta
 
-from flask import make_response
+from expiringdict import ExpiringDict
 
 import utils
 from mailconfig import get_mail_password, get_mail_user_privileges
@@ -9,150 +10,145 @@ from mfa import get_hash_mfa_state, validate_auth_mfa
 DEFAULT_KEY_PATH   = '/var/lib/mailinabox/api.key'
 DEFAULT_AUTH_REALM = 'Mail-in-a-Box Management Server'
 
-class KeyAuthService:
+class AuthService:
-	"""Generate an API key for authenticating clients
-
-	Clients must read the key from the key file and send the key with all HTTP
-	requests. The key is passed as the username field in the standard HTTP
-	Basic Auth header.
-	"""
 	def __init__(self):
 		self.auth_realm = DEFAULT_AUTH_REALM
-		self.key = self._generate_key()
 		self.key_path = DEFAULT_KEY_PATH
+		self.max_session_duration = timedelta(days=2)
 
-	def write_key(self):
+		self.init_system_api_key()
-		"""Write key to file so authorized clients can get the key
+		self.sessions = ExpiringDict(max_len=64, max_age_seconds=self.max_session_duration.total_seconds())
 
-		The key file is created with mode 0640 so that additional users can be
+	def init_system_api_key(self):
-		authorized to access the API by granting group/ACL read permissions on
+		"""Write an API key to a local file so local processes can use the API"""
-		the key file.
-		"""
-		def create_file_with_mode(path, mode):
-			# Based on answer by A-B-B: http://stackoverflow.com/a/15015748
-			old_umask = os.umask(0)
-			try:
-				return os.fdopen(os.open(path, os.O_WRONLY | os.O_CREAT, mode), 'w')
-			finally:
-				os.umask(old_umask)
 
-		os.makedirs(os.path.dirname(self.key_path), exist_ok=True)
+		with open(self.key_path, 'r') as file:
+			self.key = file.read()
 
-		with create_file_with_mode(self.key_path, 0o640) as key_file:
+	def authenticate(self, request, env, login_only=False, logout=False):
-			key_file.write(self.key + '\n')
+		"""Test if the HTTP Authorization header's username matches the system key, a session key,
-
+		or if the username/password passed in the header matches a local user.
-	def authenticate(self, request, env):
-		"""Test if the client key passed in HTTP Authorization header matches the service key
-		or if the or username/password passed in the header matches an administrator user.
 		Returns a tuple of the user's email address and list of user privileges (e.g.
 		('my@email', []) or ('my@email', ['admin']); raises a ValueError on login failure.
-		If the user used an API key, the user's email is returned as None."""
+		If the user used the system API key, the user's email is returned as None since
+		this key is not associated with a user."""
 
-		def decode(s):
+		def parse_http_authorization_basic(header):
-			return base64.b64decode(s.encode('ascii')).decode('ascii')
+			def decode(s):
-
+				return base64.b64decode(s.encode('ascii')).decode('ascii')
-		def parse_basic_auth(header):
 			if " " not in header:
 				return None, None
 			scheme, credentials = header.split(maxsplit=1)
 			if scheme != 'Basic':
 				return None, None
-
 			credentials = decode(credentials)
 			if ":" not in credentials:
 				return None, None
 			username, password = credentials.split(':', maxsplit=1)
 			return username, password
 
-		header = request.headers.get('Authorization')
+		username, password = parse_http_authorization_basic(request.headers.get('Authorization', ''))
-		if not header:
-			raise ValueError("No authorization header provided.")
-
-		username, password = parse_basic_auth(header)
-
 		if username in (None, ""):
 			raise ValueError("Authorization header invalid.")
-		elif username == self.key:
+
-			# The user passed the master API key which grants administrative privs.
+		if username.strip() == "" and password.strip() == "":
+			raise ValueError("No email address, password, session key, or API key provided.")
+
+		# If user passed the system API key, grant administrative privs. This key
+		# is not associated with a user.
+		if username == self.key and not login_only:
 			return (None, ["admin"])
+
+		# If the password corresponds with a session token for the user, grant access for that user.
+		if self.get_session(username, password, "login", env) and not login_only:
+			sessionid = password
+			session = self.sessions[sessionid]
+			if logout:
+				# Clear the session.
+				del self.sessions[sessionid]
+			else:
+				# Re-up the session so that it does not expire.
+				self.sessions[sessionid] = session
+
+		# If no password was given, but a username was given, we're missing some information.
+		elif password.strip() == "":
+			raise ValueError("Enter a password.")
+
 		else:
-			# The user is trying to log in with a username and either a password
+			# The user is trying to log in with a username and a password
-			# (and possibly a MFA token) or a user-specific API key.
+			# (and possibly a MFA token). On failure, an exception is raised.
-			return (username, self.check_user_auth(username, password, request, env))
+			self.check_user_auth(username, password, request, env)
+
+		# Get privileges for authorization. This call should never fail because by this
+		# point we know the email address is a valid user --- unless the user has been
+		# deleted after the session was granted. On error the call will return a tuple
+		# of an error message and an HTTP status code.
+		privs = get_mail_user_privileges(username, env)
+		if isinstance(privs, tuple): raise ValueError(privs[0])
+
+		# Return the authorization information.
+		return (username, privs)
 
 	def check_user_auth(self, email, pw, request, env):
 		# Validate a user's login email address and password. If MFA is enabled,
 		# check the MFA token in the X-Auth-Token header.
 		#
-		# On success returns a list of privileges (e.g. [] or ['admin']). On login
+		# On login failure, raises a ValueError with a login error message. On
-		# failure, raises a ValueError with a login error message.
+		# success, nothing is returned.
 
-		# Sanity check.
+		# Authenticate.
-		if email == "" or pw == "":
+		try:
-			raise ValueError("Enter an email address and password.")
-
-		# The password might be a user-specific API key. create_user_key raises
-		# a ValueError if the user does not exist.
-		if hmac.compare_digest(self.create_user_key(email, env), pw):
-			# OK.
-			pass
-		else:
 			# Get the hashed password of the user. Raise a ValueError if the
-			# email address does not correspond to a user.
+			# email address does not correspond to a user. But wrap it in the
+			# same exception as if a password fails so we don't easily reveal
+			# if an email address is valid.
 			pw_hash = get_mail_password(email, env)
 
-			# Authenticate.
+			# Use 'doveadm pw' to check credentials. doveadm will return
-			try:
+			# a non-zero exit status if the credentials are no good,
-				# Use 'doveadm pw' to check credentials. doveadm will return
+			# and check_call will raise an exception in that case.
-				# a non-zero exit status if the credentials are no good,
+			utils.shell('check_call', [
-				# and check_call will raise an exception in that case.
+				"/usr/bin/doveadm", "pw",
-				utils.shell('check_call', [
+				"-p", pw,
-					"/usr/bin/doveadm", "pw",
+				"-t", pw_hash,
-					"-p", pw,
+				])
-					"-t", pw_hash,
+		except:
-					])
+			# Login failed.
-			except:
+			raise ValueError("Incorrect email address or password.")
-				# Login failed.
-				raise ValueError("Invalid password.")
 
-			# If MFA is enabled, check that MFA passes.
+		# If MFA is enabled, check that MFA passes.
-			status, hints = validate_auth_mfa(email, request, env)
+		status, hints = validate_auth_mfa(email, request, env)
-			if not status:
+		if not status:
-				# Login valid. Hints may have more info.
+			# Login valid. Hints may have more info.
-				raise ValueError(",".join(hints))
+			raise ValueError(",".join(hints))
 
-		# Get privileges for authorization. This call should never fail because by this
+	def create_user_password_state_token(self, email, env):
-		# point we know the email address is a valid user. But on error the call will
+		# Create a token that changes if the user's password or MFA options change
-		# return a tuple of an error message and an HTTP status code.
+		# so that sessions become invalid if any of that information changes.
-		privs = get_mail_user_privileges(email, env)
+		msg = get_mail_password(email, env).encode("utf8")
-		if isinstance(privs, tuple): raise ValueError(privs[0])
-
-		# Return a list of privileges.
-		return privs
-
-	def create_user_key(self, email, env):
-		# Create a user API key, which is a shared secret that we can re-generate from
-		# static information in our database. The shared secret contains the user's
-		# email address, current hashed password, and current MFA state, so that the
-		# key becomes invalid if any of that information changes.
-		#
-		# Use an HMAC to generate the API key using our master API key as a key,
-		# which also means that the API key becomes invalid when our master API key
-		# changes --- i.e. when this process is restarted.
-		#
-		# Raises ValueError via get_mail_password if the user doesn't exist.
-
-		# Construct the HMAC message from the user's email address and current password.
-		msg = b"AUTH:" + email.encode("utf8") + b" " + get_mail_password(email, env).encode("utf8")
 
 		# Add to the message the current MFA state, which is a list of MFA information.
 		# Turn it into a string stably.
 		msg += b" " + json.dumps(get_hash_mfa_state(email, env), sort_keys=True).encode("utf8")
 
-		# Make the HMAC.
+		# Make a HMAC using the system API key as a hash key.
 		hash_key = self.key.encode('ascii')
 		return hmac.new(hash_key, msg, digestmod="sha256").hexdigest()
 
-	def _generate_key(self):
+	def create_session_key(self, username, env, type=None):
-		raw_key = os.urandom(32)
+		# Create a new session.
-		return base64.b64encode(raw_key).decode('ascii')
+		token = secrets.token_hex(32)
+		self.sessions[token] = {
+			"email": username,
+			"password_token": self.create_user_password_state_token(username, env),
+			"type": type,
+		}
+		return token
+
+	def get_session(self, user_email, session_key, session_type, env):
+		if session_key not in self.sessions: return None
+		session = self.sessions[session_key]
+		if session_type == "login" and session["email"] != user_email: return None
+		if session["type"] != session_type: return None
+		if session["password_token"] != self.create_user_password_state_token(session["email"], env): return None
+		return session

management/backup.py

@@ -12,12 +12,7 @@ import dateutil.parser, dateutil.relativedelta, dateutil.tz
 import rtyaml
 from exclusiveprocess import Lock
 
-from utils import load_environment, shell, wait_for_service, fix_boto
+from utils import load_environment, shell, wait_for_service
-
-rsync_ssh_options = [
-	"--ssh-options= -i /root/.ssh/id_rsa_miab",
-	"--rsync-options= -e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab\"",
-]
 
 def backup_status(env):
 	# If backups are dissbled, return no status.
@@ -64,9 +59,9 @@ def backup_status(env):
 		"--archive-dir", backup_cache_dir,
 		"--gpg-options", "--cipher-algo=AES256",
 		"--log-fd", "1",
-		config["target"],
+		get_duplicity_target_url(config),
-		] + rsync_ssh_options,
+		] + get_duplicity_additional_args(env),
-		get_env(env),
+		get_duplicity_env_vars(env),
 		trap=True)
 	if code != 0:
 		# Command failed. This is likely due to an improperly configured remote
@@ -195,7 +190,43 @@ def get_passphrase(env):
 
 	return passphrase
 
-def get_env(env):
+def get_duplicity_target_url(config):
+	target = config["target"]
+
+	if get_target_type(config) == "s3":
+		from urllib.parse import urlsplit, urlunsplit
+		target = list(urlsplit(target))
+
+		# Although we store the S3 hostname in the target URL,
+		# duplicity no longer accepts it in the target URL. The hostname in
+		# the target URL must be the bucket name. The hostname is passed
+		# via get_duplicity_additional_args. Move the first part of the
+		# path (the bucket name) into the hostname URL component, and leave
+		# the rest for the path.
+		target[1], target[2] = target[2].lstrip('/').split('/', 1)
+
+		target = urlunsplit(target)
+
+	return target
+
+def get_duplicity_additional_args(env):
+	config = get_backup_config(env)
+
+	if get_target_type(config) == 'rsync':
+		return [
+			"--ssh-options= -i /root/.ssh/id_rsa_miab",
+			"--rsync-options= -e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab\"",
+		]
+	elif get_target_type(config) == 's3':
+		# See note about hostname in get_duplicity_target_url.
+		from urllib.parse import urlsplit, urlunsplit
+		target = urlsplit(config["target"])
+		endpoint_url = urlunsplit(("https", target.netloc, '', '', ''))
+		return ["--s3-endpoint-url",  endpoint_url]
+
+	return []
+
+def get_duplicity_env_vars(env):
 	config = get_backup_config(env)
 
 	env = { "PASSPHRASE" : get_passphrase(env) }
@@ -247,9 +278,10 @@ def perform_backup(full_backup):
 			if quit:
 				sys.exit(code)
 
-	service_command("php7.2-fpm", "stop", quit=True)
+	service_command("php8.0-fpm", "stop", quit=True)
 	service_command("postfix", "stop", quit=True)
 	service_command("dovecot", "stop", quit=True)
+	service_command("postgrey", "stop", quit=True)
 
 	# Execute a pre-backup script that copies files outside the homedir.
 	# Run as the STORAGE_USER user, not as root. Pass our settings in
@@ -273,15 +305,16 @@ def perform_backup(full_backup):
 			"--volsize", "250",
 			"--gpg-options", "--cipher-algo=AES256",
 			env["STORAGE_ROOT"],
-			config["target"],
+			get_duplicity_target_url(config),
 			"--allow-source-mismatch"
-			] + rsync_ssh_options,
+			] + get_duplicity_additional_args(env),
-			get_env(env))
+			get_duplicity_env_vars(env))
 	finally:
 		# Start services again.
+		service_command("postgrey", "start", quit=False)
 		service_command("dovecot", "start", quit=False)
 		service_command("postfix", "start", quit=False)
-		service_command("php7.2-fpm", "start", quit=False)
+		service_command("php8.0-fpm", "start", quit=False)
 
 	# Remove old backups. This deletes all backup data no longer needed
 	# from more than 3 days ago.
@@ -292,9 +325,9 @@ def perform_backup(full_backup):
 		"--verbosity", "error",
 		"--archive-dir", backup_cache_dir,
 		"--force",
-		config["target"]
+		get_duplicity_target_url(config)
-		] + rsync_ssh_options,
+		] + get_duplicity_additional_args(env),
-		get_env(env))
+		get_duplicity_env_vars(env))
 
 	# From duplicity's manual:
 	# "This should only be necessary after a duplicity session fails or is
@@ -307,9 +340,9 @@ def perform_backup(full_backup):
 		"--verbosity", "error",
 		"--archive-dir", backup_cache_dir,
 		"--force",
-		config["target"]
+		get_duplicity_target_url(config)
-		] + rsync_ssh_options,
+		] + get_duplicity_additional_args(env),
-		get_env(env))
+		get_duplicity_env_vars(env))
 
 	# Change ownership of backups to the user-data user, so that the after-bcakup
 	# script can access them.
@@ -345,9 +378,9 @@ def run_duplicity_verification():
 		"--compare-data",
 		"--archive-dir", backup_cache_dir,
 		"--exclude", backup_root,
-		config["target"],
+		get_duplicity_target_url(config),
 		env["STORAGE_ROOT"],
-	] + rsync_ssh_options, get_env(env))
+	] + get_duplicity_additional_args(env), get_duplicity_env_vars(env))
 
 def run_duplicity_restore(args):
 	env = load_environment()
@@ -357,9 +390,9 @@ def run_duplicity_restore(args):
 		"/usr/bin/duplicity",
 		"restore",
 		"--archive-dir", backup_cache_dir,
-		config["target"],
+		get_duplicity_target_url(config),
-		] + rsync_ssh_options + args,
+		] + get_duplicity_additional_args(env) + args,
-	get_env(env))
+	get_duplicity_env_vars(env))
 
 def list_target_files(config):
 	import urllib.parse
@@ -415,26 +448,13 @@ def list_target_files(config):
 			raise ValueError("Connection to rsync host failed: {}".format(reason))
 
 	elif target.scheme == "s3":
-		# match to a Region
+		import boto3.s3
-		fix_boto() # must call prior to importing boto
+		from botocore.exceptions import ClientError
-		import boto.s3
+		
-		from boto.exception import BotoServerError
+		# separate bucket from path in target
-		custom_region = False
-		for region in boto.s3.regions():
-			if region.endpoint == target.hostname:
-				break
-		else:
-			# If region is not found this is a custom region
-			custom_region = True
-
 		bucket = target.path[1:].split('/')[0]
 		path = '/'.join(target.path[1:].split('/')[1:]) + '/'
 
-		# Create a custom region with custom endpoint
-		if custom_region:
-			from boto.s3.connection import S3Connection
-			region = boto.s3.S3RegionInfo(name=bucket, endpoint=target.hostname, connection_cls=S3Connection)
-
 		# If no prefix is specified, set the path to '', otherwise boto won't list the files
 		if path == '/':
 			path = ''
@@ -444,18 +464,15 @@ def list_target_files(config):
 
 		# connect to the region & bucket
 		try:
-			conn = region.connect(aws_access_key_id=config["target_user"], aws_secret_access_key=config["target_pass"])
+			s3 = boto3.client('s3', \
-			bucket = conn.get_bucket(bucket)
+				endpoint_url=f'https://{target.hostname}', \
-		except BotoServerError as e:
+				aws_access_key_id=config['target_user'], \
-			if e.status == 403:
+				aws_secret_access_key=config['target_pass'])
-				raise ValueError("Invalid S3 access key or secret access key.")
+			bucket_objects = s3.list_objects_v2(Bucket=bucket, Prefix=path)['Contents']
-			elif e.status == 404:
+			backup_list = [(key['Key'][len(path):], key['Size']) for key in bucket_objects]
-				raise ValueError("Invalid S3 bucket name.")
+		except ClientError as e:
-			elif e.status == 301:
+			raise ValueError(e)
-				raise ValueError("Incorrect region for this bucket.")
+		return backup_list
-			raise ValueError(e.reason)
-
-		return [(key.name[len(path):], key.size) for key in bucket.list(prefix=path)]
 	elif target.scheme == 'b2':
 		from b2sdk.v1 import InMemoryAccountInfo, B2Api
 		from b2sdk.v1.exception import NonExistentBucket

management/daemon.py

@@ -1,5 +1,8 @@
 #!/usr/local/lib/mailinabox/env/bin/python3
 #
+# The API can be accessed on the command line, e.g. use `curl` like so:
+#    curl --user $(</var/lib/mailinabox/api.key): http://localhost:10222/mail/users
+#
 # During development, you can start the Mail-in-a-Box control panel
 # by running this script, e.g.:
 #
@@ -22,7 +25,7 @@ from mfa import get_public_mfa_state, provision_totp, validate_totp_secret, enab
 
 env = utils.load_environment()
 
-auth_service = auth.KeyAuthService()
+auth_service = auth.AuthService()
 
 # We may deploy via a symbolic link, which confuses flask's template finding.
 me = __file__
@@ -53,8 +56,10 @@ def authorized_personnel_only(viewfunc):
 		try:
 			email, privs = auth_service.authenticate(request, env)
 		except ValueError as e:
-			# Write a line in the log recording the failed login
+			# Write a line in the log recording the failed login, unless no authorization header
-			log_failed_login(request)
+			# was given which can happen on an initial request before a 403 response.
+			if "Authorization" in request.headers:
+				log_failed_login(request)
 
 			# Authentication failed.
 			error = str(e)
@@ -116,9 +121,9 @@ def index():
 	no_users_exist = (len(get_mail_users(env)) == 0)
 	no_admins_exist = (len(get_admins(env)) == 0)
 
-	utils.fix_boto() # must call prior to importing boto
+	import boto3.s3
-	import boto.s3
+	backup_s3_hosts = [(r, f"s3.{r}.amazonaws.com") for r in boto3.session.Session().get_available_regions('s3')]
-	backup_s3_hosts = [(r.name, r.endpoint) for r in boto.s3.regions()]
+
 
 	return render_template('index.html',
 		hostname=env['PRIMARY_HOSTNAME'],
@@ -131,11 +136,12 @@ def index():
 		csr_country_codes=csr_country_codes,
 	)
 
-@app.route('/me')
+# Create a session key by checking the username/password in the Authorization header.
-def me():
+@app.route('/login', methods=["POST"])
+def login():
 	# Is the caller authorized?
 	try:
-		email, privs = auth_service.authenticate(request, env)
+		email, privs = auth_service.authenticate(request, env, login_only=True)
 	except ValueError as e:
 		if "missing-totp-token" in str(e):
 			return json_response({
@@ -150,19 +156,29 @@ def me():
 				"reason": str(e),
 			})
 
+	# Return a new session for the user.
 	resp = {
 		"status": "ok",
 		"email": email,
 		"privileges": privs,
+		"api_key": auth_service.create_session_key(email, env, type='login'),
 	}
 
-	# Is authorized as admin? Return an API key for future use.
+	app.logger.info("New login session created for {}".format(email))
-	if "admin" in privs:
-		resp["api_key"] = auth_service.create_user_key(email, env)
 
 	# Return.
 	return json_response(resp)
 
+@app.route('/logout', methods=["POST"])
+def logout():
+	try:
+		email, _ = auth_service.authenticate(request, env, logout=True)
+		app.logger.info("{} logged out".format(email))
+	except ValueError as e:
+		pass
+	finally:
+		return json_response({ "status": "ok" })
+
 # MAIL
 
 @app.route('/mail/users')
@@ -219,7 +235,7 @@ def mail_aliases():
 	if request.args.get("format", "") == "json":
 		return json_response(get_mail_aliases_ex(env))
 	else:
-		return "".join(address+"\t"+receivers+"\t"+(senders or "")+"\n" for address, receivers, senders in get_mail_aliases(env))
+		return "".join(address+"\t"+receivers+"\t"+(senders or "")+"\n" for address, receivers, senders, auto in get_mail_aliases(env))
 
 @app.route('/mail/aliases/add', methods=['POST'])
 @authorized_personnel_only
@@ -314,7 +330,7 @@ def dns_get_records(qname=None, rtype=None):
 		r["sort-order"]["created"] = i
 	domain_sort_order = utils.sort_domains([r["qname"] for r in records], env)
 	for i, r in enumerate(sorted(records, key = lambda r : (
-			zones.index(r["zone"]),
+			zones.index(r["zone"]) if r.get("zone") else 0, # record is not within a zone managed by the box
 			domain_sort_order.index(r["qname"]),
 			r["rtype"]))):
 		r["sort-order"]["qname"] = i
@@ -555,6 +571,8 @@ def system_status():
 	# Create a temporary pool of processes for the status checks
 	with multiprocessing.pool.Pool(processes=5) as pool:
 		run_checks(False, env, output, pool)
+		pool.close()
+		pool.join()
 	return json_response(output.items)
 
 @app.route('/system/updates')
@@ -638,16 +656,42 @@ def privacy_status_set():
 # MUNIN
 
 @app.route('/munin/')
-@app.route('/munin/<path:filename>')
 @authorized_personnel_only
-def munin(filename=""):
+def munin_start():
-	# Checks administrative access (@authorized_personnel_only) and then just proxies
+	# Munin pages, static images, and dynamically generated images are served
-	# the request to static files.
+	# outside of the AJAX API. We'll start with a 'start' API that sets a cookie
+	# that subsequent requests will read for authorization. (We don't use cookies
+	# for the API to avoid CSRF vulnerabilities.)
+	response = make_response("OK")
+	response.set_cookie("session", auth_service.create_session_key(request.user_email, env, type='cookie'),
+	    max_age=60*30, secure=True, httponly=True, samesite="Strict") # 30 minute duration
+	return response
+
+def check_request_cookie_for_admin_access():
+	session = auth_service.get_session(None, request.cookies.get("session", ""), "cookie", env)
+	if not session: return False
+	privs = get_mail_user_privileges(session["email"], env)
+	if not isinstance(privs, list): return False
+	if "admin" not in privs: return False
+	return True
+
+def authorized_personnel_only_via_cookie(f):
+	@wraps(f)
+	def g(*args, **kwargs):
+		if not check_request_cookie_for_admin_access():
+			return Response("Unauthorized", status=403, mimetype='text/plain', headers={})
+		return f(*args, **kwargs)
+	return g
+
+@app.route('/munin/<path:filename>')
+@authorized_personnel_only_via_cookie
+def munin_static_file(filename=""):
+	# Proxy the request to static files.
 	if filename == "": filename = "index.html"
 	return send_from_directory("/var/cache/munin/www", filename)
 
 @app.route('/munin/cgi-graph/<path:filename>')
-@authorized_personnel_only
+@authorized_personnel_only_via_cookie
 def munin_cgi(filename):
 	""" Relay munin cgi dynazoom requests
 	/usr/lib/munin/cgi/munin-cgi-graph is a perl cgi script in the munin package
@@ -724,30 +768,10 @@ if __name__ == '__main__':
 		# Turn on Flask debugging.
 		app.debug = True
 
-		# Use a stable-ish master API key so that login sessions don't restart on each run.
-		# Use /etc/machine-id to seed the key with a stable secret, but add something
-		# and hash it to prevent possibly exposing the machine id, using the time so that
-		# the key is not valid indefinitely.
-		import hashlib
-		with open("/etc/machine-id") as f:
-			api_key = f.read()
-		api_key += "|" + str(int(time.time() / (60*60*2)))
-		hasher = hashlib.sha1()
-		hasher.update(api_key.encode("ascii"))
-		auth_service.key = hasher.hexdigest()
-
-	if "APIKEY" in os.environ: auth_service.key = os.environ["APIKEY"]
-
 	if not app.debug:
 		app.logger.addHandler(utils.create_syslog_handler())
 
-	# For testing on the command line, you can use `curl` like so:
+	#app.logger.info('API key: ' + auth_service.key)
-	#    curl --user $(</var/lib/mailinabox/api.key): http://localhost:10222/mail/users
-	auth_service.write_key()
-
-	# For testing in the browser, you can copy the API key that's output to the
-	# debug console and enter that as the username
-	app.logger.info('API key: ' + auth_service.key)
 
 	# Start the application server. Listens on 127.0.0.1 (IPv4 only).
 	app.run(port=10222)

management/dns_update.py

@@ -96,9 +96,18 @@ def do_dns_update(env, force=False):
 		if len(updated_domains) == 0:
 			updated_domains.append("DNS configuration")
 
-	# Kick nsd if anything changed.
+	# Tell nsd to reload changed zone files.
 	if len(updated_domains) > 0:
-		shell('check_call', ["/usr/sbin/service", "nsd", "restart"])
+		# 'reconfig' is needed if there are added or removed zones, but
+		# it may not reload existing zones, so we call 'reload' too. If
+		# nsd isn't running, nsd-control fails, so in that case revert
+		# to restarting nsd to make sure it is running. Restarting nsd
+		# should also refresh everything.
+		try:
+			shell('check_call', ["/usr/sbin/nsd-control", "reconfig"])
+			shell('check_call', ["/usr/sbin/nsd-control", "reload"])
+		except:
+			shell('check_call', ["/usr/sbin/service", "nsd", "restart"])
 
 	# Write the OpenDKIM configuration tables for all of the mail domains.
 	from mailconfig import get_mail_domains
@@ -298,7 +307,7 @@ def build_zone(domain, domain_properties, additional_records, env, is_zone=True)
 		# Append a DMARC record.
 		# Skip if the user has set a DMARC record already.
 		if not has_rec("_dmarc", "TXT", prefix="v=DMARC1; "):
-			records.append(("_dmarc", "TXT", 'v=DMARC1; p=quarantine', "Recommended. Specifies that mail that does not originate from the box but claims to be from @%s or which does not have a valid DKIM signature is suspect and should be quarantined by the recipient's mail system." % domain))
+			records.append(("_dmarc", "TXT", 'v=DMARC1; p=quarantine;', "Recommended. Specifies that mail that does not originate from the box but claims to be from @%s or which does not have a valid DKIM signature is suspect and should be quarantined by the recipient's mail system." % domain))
 
 	if domain_properties[domain]["user"]:
 		# Add CardDAV/CalDAV SRV records on the non-primary hostname that points to the primary hostname
@@ -363,7 +372,7 @@ def build_zone(domain, domain_properties, additional_records, env, is_zone=True)
 			if not has_rec(qname, "TXT", prefix="v=spf1 "):
 				records.append((qname,  "TXT", 'v=spf1 -all', "Recommended. Prevents use of this domain name for outbound mail by specifying that no servers are valid sources for mail from @%s. If you do send email from this domain name you should either override this record such that the SPF rule does allow the originating server, or, take the recommended approach and have the box handle mail for this domain (simply add any receiving alias at this domain name to make this machine treat the domain name as one of its mail domains)." % d))
 			if not has_rec("_dmarc" + ("."+qname if qname else ""), "TXT", prefix="v=DMARC1; "):
-				records.append(("_dmarc" + ("."+qname if qname else ""), "TXT", 'v=DMARC1; p=reject', "Recommended. Prevents use of this domain name for outbound mail by specifying that the SPF rule should be honoured for mail from @%s." % d))
+				records.append(("_dmarc" + ("."+qname if qname else ""), "TXT", 'v=DMARC1; p=reject;', "Recommended. Prevents use of this domain name for outbound mail by specifying that the SPF rule should be honoured for mail from @%s." % d))
 
 			# And with a null MX record (https://explained-from-first-principles.com/email/#null-mx-record)
 			if not has_rec(qname, "MX"):
@@ -484,7 +493,7 @@ def write_nsd_zone(domain, zonefile, records, env, force):
 	# @ the PRIMARY_HOSTNAME. Hopefully that's legit.
 	#
 	# For the refresh through TTL fields, a good reference is:
-	# http://www.peerwisdom.org/2013/05/15/dns-understanding-the-soa-record/
+	# https://www.ripe.net/publications/docs/ripe-203
 	#
 	# A hash of the available DNSSEC keys are added in a comment so that when
 	# the keys change we force a re-generation of the zone which triggers
@@ -497,7 +506,7 @@ $TTL 86400          ; default time to live
 @ IN SOA ns1.{primary_domain}. hostmaster.{primary_domain}. (
            __SERIAL__     ; serial number
            7200     ; Refresh (secondary nameserver update interval)
-           86400    ; Retry (when refresh fails, how often to try again)
+           3600     ; Retry (when refresh fails, how often to try again, should be lower than the refresh)
            1209600  ; Expire (when refresh fails, how long secondary nameserver will keep records around anyway)
            86400    ; Negative TTL (how long negative responses are cached)
            )
@@ -604,7 +613,7 @@ def get_dns_zonefile(zone, env):
 
 def write_nsd_conf(zonefiles, additional_records, env):
 	# Write the list of zones to a configuration file.
-	nsd_conf_file = "/etc/nsd/zones.conf"
+	nsd_conf_file = "/etc/nsd/nsd.conf.d/zones.conf"
 	nsdconf = ""
 
 	# Append the zones.
@@ -1000,9 +1009,9 @@ def get_secondary_dns(custom_dns, mode=None):
 			# doesn't.
 			if not hostname.startswith("xfr:"):
 				if mode == "xfr":
-					response = dns.resolver.query(hostname+'.', "A", raise_on_no_answer=False)
+					response = dns.resolver.resolve(hostname+'.', "A", raise_on_no_answer=False)
 					values.extend(map(str, response))
-					response = dns.resolver.query(hostname+'.', "AAAA", raise_on_no_answer=False)
+					response = dns.resolver.resolve(hostname+'.', "AAAA", raise_on_no_answer=False)
 					values.extend(map(str, response))
 					continue
 				values.append(hostname)
@@ -1025,10 +1034,10 @@ def set_secondary_dns(hostnames, env):
 			if not item.startswith("xfr:"):
 				# Resolve hostname.
 				try:
-					response = resolver.query(item, "A")
+					response = resolver.resolve(item, "A")
 				except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
 					try:
-						response = resolver.query(item, "AAAA")
+						response = resolver.resolve(item, "AAAA")
 					except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
 						raise ValueError("Could not resolve the IP address of %s." % item)
 			else:

management/mail_log.py

@@ -549,8 +549,9 @@ def scan_postfix_submission_line(date, log, collector):
     """
 
     # Match both the 'plain' and 'login' sasl methods, since both authentication methods are
-    # allowed by Dovecot
+    # allowed by Dovecot. Exclude trailing comma after the username when additional fields 
-    m = re.match("([A-Z0-9]+): client=(\S+), sasl_method=(PLAIN|LOGIN), sasl_username=(\S+)", log)
+	# follow after.
+    m = re.match("([A-Z0-9]+): client=(\S+), sasl_method=(PLAIN|LOGIN), sasl_username=(\S+)(?<!,)", log)
 
     if m:
         _, client, method, user = m.groups()
@@ -586,7 +587,7 @@ def scan_postfix_submission_line(date, log, collector):
 def readline(filename):
     """ A generator that returns the lines of a file
     """
-    with open(filename) as file:
+    with open(filename, errors='replace') as file:
         while True:
           line = file.readline()
           if not line:

management/mailconfig.py

@@ -16,8 +16,8 @@ import idna
 
 def validate_email(email, mode=None):
 	# Checks that an email address is syntactically valid. Returns True/False.
-	# Until Postfix supports SMTPUTF8, an email address may contain ASCII
+	# An email address may contain ASCII characters only because Dovecot's
-	# characters only; IDNs must be IDNA-encoded.
+	# authentication mechanism gets confused with other character encodings.
 	#
 	# When mode=="user", we're checking that this can be a user account name.
 	# Dovecot has tighter restrictions - letters, numbers, underscore, and
@@ -186,9 +186,9 @@ def get_admins(env):
 	return users
 
 def get_mail_aliases(env):
-	# Returns a sorted list of tuples of (address, forward-tos, permitted-senders).
+	# Returns a sorted list of tuples of (address, forward-tos, permitted-senders, auto).
 	c = open_database(env)
-	c.execute('SELECT source, destination, permitted_senders FROM aliases')
+	c.execute('SELECT source, destination, permitted_senders, 0 as auto FROM aliases UNION SELECT source, destination, permitted_senders, 1 as auto FROM auto_aliases')
 	aliases = { row[0]: row for row in c.fetchall() } # make dict
 
 	# put in a canonical order: sort by domain, then by email address lexicographically
@@ -208,7 +208,7 @@ def get_mail_aliases_ex(env):
 	#         address_display: "name@domain.tld", # full Unicode
 	#         forwards_to: ["user1@domain.com", "receiver-only1@domain.com", ...],
 	#         permitted_senders: ["user1@domain.com", "sender-only1@domain.com", ...] OR null,
-	#         required: True|False
+	#         auto: True|False
 	#       },
 	#       ...
 	#     ]
@@ -216,12 +216,13 @@ def get_mail_aliases_ex(env):
 	#   ...
 	# ]
 
-	required_aliases = get_required_aliases(env)
 	domains = {}
-	for address, forwards_to, permitted_senders in get_mail_aliases(env):
+	for address, forwards_to, permitted_senders, auto in get_mail_aliases(env):
+		# skip auto domain maps since these are not informative in the control panel's aliases list
+		if auto and address.startswith("@"): continue
+
 		# get alias info
 		domain = get_domain(address)
-		required = (address in required_aliases)
 
 		# add to list
 		if not domain in domains:
@@ -234,7 +235,7 @@ def get_mail_aliases_ex(env):
 			"address_display": prettify_idn_email_address(address),
 			"forwards_to": [prettify_idn_email_address(r.strip()) for r in forwards_to.split(",")],
 			"permitted_senders": [prettify_idn_email_address(s.strip()) for s in permitted_senders.split(",")] if permitted_senders is not None else None,
-			"required": required,
+			"auto": bool(auto),
 		})
 
 	# Sort domains.
@@ -242,7 +243,7 @@ def get_mail_aliases_ex(env):
 
 	# Sort aliases within each domain first by required-ness then lexicographically by address.
 	for domain in domains:
-		domain["aliases"].sort(key = lambda alias : (alias["required"], alias["address"]))
+		domain["aliases"].sort(key = lambda alias : (alias["auto"], alias["address"]))
 	return domains
 
 def get_domain(emailaddr, as_unicode=True):
@@ -261,11 +262,12 @@ def get_domain(emailaddr, as_unicode=True):
 def get_mail_domains(env, filter_aliases=lambda alias : True, users_only=False):
 	# Returns the domain names (IDNA-encoded) of all of the email addresses
 	# configured on the system. If users_only is True, only return domains
-	# with email addresses that correspond to user accounts.
+	# with email addresses that correspond to user accounts. Exclude Unicode
+	# forms of domain names listed in the automatic aliases table.
 	domains = []
 	domains.extend([get_domain(login, as_unicode=False) for login in get_mail_users(env)])
 	if not users_only:
-		domains.extend([get_domain(address, as_unicode=False) for address, *_ in get_mail_aliases(env) if filter_aliases(address) ])
+		domains.extend([get_domain(address, as_unicode=False) for address, _, _, auto in get_mail_aliases(env) if filter_aliases(address) and not auto ])
 	return set(domains)
 
 def add_mail_user(email, pw, privs, env):
@@ -512,6 +514,13 @@ def remove_mail_alias(address, env, do_kick=True):
 		# Update things in case any domains are removed.
 		return kick(env, "alias removed")
 
+def add_auto_aliases(aliases, env):
+	conn, c = open_database(env, with_connection=True)
+	c.execute("DELETE FROM auto_aliases");
+	for source, destination in aliases.items():
+		c.execute("INSERT INTO auto_aliases (source, destination) VALUES (?, ?)", (source, destination))
+	conn.commit()
+
 def get_system_administrator(env):
 	return "administrator@" + env['PRIMARY_HOSTNAME']
 
@@ -555,39 +564,34 @@ def kick(env, mail_result=None):
 	if mail_result is not None:
 		results.append(mail_result + "\n")
 
-	# Ensure every required alias exists.
+	auto_aliases = { }
 
-	existing_users = get_mail_users(env)
+	# Mape required aliases to the administrator alias (which should be created manually).
-	existing_alias_records = get_mail_aliases(env)
+	administrator = get_system_administrator(env)
-	existing_aliases = set(a for a, *_ in existing_alias_records) # just first entry in tuple
 	required_aliases = get_required_aliases(env)
+	for alias in required_aliases:
+		if alias == administrator: continue # don't make an alias from the administrator to itself --- this alias must be created manually
+		auto_aliases[alias] = administrator
 
-	def ensure_admin_alias_exists(address):
+	# Add domain maps from Unicode forms of IDNA domains to the ASCII forms stored in the alias table.
-		# If a user account exists with that address, we're good.
+	for domain in get_mail_domains(env):
-		if address in existing_users:
+		try:
-			return
+			domain_unicode = idna.decode(domain.encode("ascii"))
+			if domain == domain_unicode: continue # not an IDNA/Unicode domain
+			auto_aliases["@" + domain_unicode] = "@" + domain
+		except (ValueError, UnicodeError, idna.IDNAError):
+			continue
 
-		# If the alias already exists, we're good.
+	add_auto_aliases(auto_aliases, env)
-		if address in existing_aliases:
-			return
 
-		# Doesn't exist.
+	# Remove auto-generated postmaster/admin/abuse alises from the main aliases table.
-		administrator = get_system_administrator(env)
+	# They are now stored in the auto_aliases table.
-		if address == administrator: return # don't make an alias from the administrator to itself --- this alias must be created manually
+	for address, forwards_to, permitted_senders, auto in get_mail_aliases(env):
-		add_mail_alias(address, administrator, "", env, do_kick=False)
-		if administrator not in existing_aliases: return # don't report the alias in output if the administrator alias isn't in yet -- this is a hack to supress confusing output on initial setup
-		results.append("added alias %s (=> %s)\n" % (address, administrator))
-
-	for address in required_aliases:
-		ensure_admin_alias_exists(address)
-
-	# Remove auto-generated postmaster/admin on domains we no
-	# longer have any other email addresses for.
-	for address, forwards_to, *_ in existing_alias_records:
 		user, domain = address.split("@")
 		if user in ("postmaster", "admin", "abuse") \
 			and address not in required_aliases \
-			and forwards_to == get_system_administrator(env):
+			and forwards_to == get_system_administrator(env) \
+			and not auto:
 			remove_mail_alias(address, env, do_kick=False)
 			results.append("removed alias %s (was to %s; domain no longer used for email)\n" % (address, forwards_to))
 

management/ssl_certificates.py

@@ -58,36 +58,33 @@ def get_ssl_certificates(env):
 			# Not a valid PEM format for a PEM type we care about.
 			continue
 
-		# Remember where we got this object.
-		pem._filename = fn
-
 		# Is it a private key?
 		if isinstance(pem, RSAPrivateKey):
-			private_keys[pem.public_key().public_numbers()] = pem
+			private_keys[pem.public_key().public_numbers()] = { "filename": fn, "key": pem }
 
 		# Is it a certificate?
 		if isinstance(pem, Certificate):
-			certificates.append(pem)
+			certificates.append({ "filename": fn, "cert": pem })
 
 	# Process the certificates.
 	domains = { }
 	for cert in certificates:
 		# What domains is this certificate good for?
-		cert_domains, primary_domain = get_certificate_domains(cert)
+		cert_domains, primary_domain = get_certificate_domains(cert["cert"])
-		cert._primary_domain = primary_domain
+		cert["primary_domain"] = primary_domain
 
 		# Is there a private key file for this certificate?
-		private_key = private_keys.get(cert.public_key().public_numbers())
+		private_key = private_keys.get(cert["cert"].public_key().public_numbers())
 		if not private_key:
 			continue
-		cert._private_key = private_key
+		cert["private_key"] = private_key
 
 		# Add this cert to the list of certs usable for the domains.
 		for domain in cert_domains:
 			# The primary hostname can only use a certificate mapped
 			# to the system private key.
 			if domain == env['PRIMARY_HOSTNAME']:
-				if cert._private_key._filename != os.path.join(env['STORAGE_ROOT'], 'ssl', 'ssl_private_key.pem'):
+				if cert["private_key"]["filename"] != os.path.join(env['STORAGE_ROOT'], 'ssl', 'ssl_private_key.pem'):
 					continue
 
 			domains.setdefault(domain, []).append(cert)
@@ -100,10 +97,10 @@ def get_ssl_certificates(env):
 		#for c in cert_list: print(domain, c.not_valid_before, c.not_valid_after, "("+str(now)+")", c.issuer, c.subject, c._filename)
 		cert_list.sort(key = lambda cert : (
 			# must be valid NOW
-			cert.not_valid_before <= now <= cert.not_valid_after,
+			cert["cert"].not_valid_before <= now <= cert["cert"].not_valid_after,
 
 			# prefer one that is not self-signed
-			cert.issuer != cert.subject,
+			cert["cert"].issuer != cert["cert"].subject,
 
 			###########################################################
 			# The above lines ensure that valid certificates are chosen
@@ -113,7 +110,7 @@ def get_ssl_certificates(env):
 
 			# prefer one with the expiration furthest into the future so
 			# that we can easily rotate to new certs as we get them
-			cert.not_valid_after,
+			cert["cert"].not_valid_after,
 
 			###########################################################
 			# We always choose the certificate that is good for the
@@ -128,15 +125,15 @@ def get_ssl_certificates(env):
 
 			# in case a certificate is installed in multiple paths,
 			# prefer the... lexicographically last one?
-			cert._filename,
+			cert["filename"],
 
 		), reverse=True)
 		cert = cert_list.pop(0)
 		ret[domain] = {
-			"private-key": cert._private_key._filename,
+			"private-key": cert["private_key"]["filename"],
-			"certificate": cert._filename,
+			"certificate": cert["filename"],
-			"primary-domain": cert._primary_domain,
+			"primary-domain": cert["primary_domain"],
-			"certificate_object": cert,
+			"certificate_object": cert["cert"],
 			}
 
 	return ret

management/status_checks.py

@@ -135,7 +135,7 @@ def check_service(i, service, env):
 
 			# IPv4 ok but IPv6 failed. Try the PRIVATE_IPV6 address to see if the service is bound to the interface.
 			elif service["port"] != 53 and try_connect(env["PRIVATE_IPV6"]):
-				output.print_error("%s is running (and available over IPv4 and the local IPv6 address), but it is not publicly accessible at %s:%d." % (service['name'], env['PUBLIC_IP'], service['port']))
+				output.print_error("%s is running (and available over IPv4 and the local IPv6 address), but it is not publicly accessible at %s:%d." % (service['name'], env['PUBLIC_IPV6'], service['port']))
 			else:
 				output.print_error("%s is running and available over IPv4 but is not accessible over IPv6 at %s port %d." % (service['name'], env['PUBLIC_IPV6'], service['port']))
 
@@ -253,6 +253,18 @@ def check_free_disk_space(rounded_values, env, output):
 		if rounded_values: disk_msg = "The disk has less than 15% free space."
 		output.print_error(disk_msg)
 
+	# Check that there's only one duplicity cache. If there's more than one,
+	# it's probably no longer in use, and we can recommend clearing the cache
+	# to save space. The cache directory may not exist yet, which is OK.
+	backup_cache_path = os.path.join(env['STORAGE_ROOT'], 'backup/cache')
+	try:
+		backup_cache_count = len(os.listdir(backup_cache_path))
+	except:
+		backup_cache_count = 0
+	if backup_cache_count > 1:
+		output.print_warning("The backup cache directory {} has more than one backup target cache. Consider clearing this directory to save disk space."
+			.format(backup_cache_path))
+
 def check_free_memory(rounded_values, env, output):
 	# Check free memory.
 	percent_free = 100 - psutil.virtual_memory().percent
@@ -612,14 +624,16 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 			#
 			# But it may not be preferred. Only algorithm 13 is preferred. Warn if any of the
 			# matched zones uses a different algorithm.
-			if set(r[1] for r in matched_ds) == { '13' }: # all are alg 13
+			if set(r[1] for r in matched_ds) == { '13' } and set(r[2] for r in matched_ds) <= { '2', '4' }: # all are alg 13 and digest type 2 or 4
 				output.print_ok("DNSSEC 'DS' record is set correctly at registrar.")
 				return
-			elif '13' in set(r[1] for r in matched_ds): # some but not all are alg 13
+			elif len([r for r in matched_ds if r[1] == '13' and r[2] in ( '2', '4' )]) > 0: # some but not all are alg 13
-				output.print_ok("DNSSEC 'DS' record is set correctly at registrar. (Records using algorithm other than ECDSAP256SHA256 should be removed.)")
+				output.print_ok("DNSSEC 'DS' record is set correctly at registrar. (Records using algorithm other than ECDSAP256SHA256 and digest types other than SHA-256/384 should be removed.)")
 				return
 			else: # no record uses alg 13
-				output.print_warning("DNSSEC 'DS' record set at registrar is valid but should be updated to ECDSAP256SHA256 (see below).")
+				output.print_warning("""DNSSEC 'DS' record set at registrar is valid but should be updated to ECDSAP256SHA256 and SHA-256 (see below).
+				IMPORTANT: Do not delete existing DNSSEC 'DS' records for this domain until confirmation that the new DNSSEC 'DS' record
+				for this domain is valid.""")
 		else:
 			if is_checking_primary:
 				output.print_error("""The DNSSEC 'DS' record for %s is incorrect. See further details below.""" % domain)
@@ -630,7 +644,8 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 
 	output.print_line("""Follow the instructions provided by your domain name registrar to set a DS record.
 		Registrars support different sorts of DS records. Use the first option that works:""")
-	preferred_ds_order = [(7, 1), (7, 2), (8, 4), (13, 4), (8, 1), (8, 2), (13, 1), (13, 2)] # low to high
+	preferred_ds_order = [(7, 2), (8, 4), (13, 4), (8, 2), (13, 2)] # low to high, see https://github.com/mail-in-a-box/mailinabox/issues/1998
+
 	def preferred_ds_order_func(ds_suggestion):
 		k = (int(ds_suggestion['alg']), int(ds_suggestion['digalg']))
 		if k in preferred_ds_order:
@@ -638,11 +653,12 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 		return -1 # index before first item
 	output.print_line("")
 	for i, ds_suggestion in enumerate(sorted(expected_ds_records.values(), key=preferred_ds_order_func, reverse=True)):
+		if preferred_ds_order_func(ds_suggestion) == -1: continue # don't offer record types that the RFC says we must not offer
 		output.print_line("")
 		output.print_line("Option " + str(i+1) + ":")
 		output.print_line("----------")
 		output.print_line("Key Tag: " + ds_suggestion['keytag'])
-		output.print_line("Key Flags: KSK")
+		output.print_line("Key Flags: KSK / 257")
 		output.print_line("Algorithm: %s / %s" % (ds_suggestion['alg'], ds_suggestion['alg_name']))
 		output.print_line("Digest Type: %s / %s" % (ds_suggestion['digalg'], ds_suggestion['digalg_name']))
 		output.print_line("Digest: " + ds_suggestion['digest'])
@@ -654,7 +670,7 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 	if len(ds) > 0:
 		output.print_line("")
 		output.print_line("The DS record is currently set to:")
-		for rr in ds:
+		for rr in sorted(ds):
 			output.print_line("Key Tag: {0}, Algorithm: {1}, Digest Type: {2}, Digest: {3}".format(*rr))
 
 def check_mail_domain(domain, env, output):
@@ -699,7 +715,7 @@ def check_mail_domain(domain, env, output):
 		output.print_ok(good_news)
 
 		# Check MTA-STS policy.
-		loop = asyncio.get_event_loop()
+		loop = asyncio.new_event_loop()
 		sts_resolver = postfix_mta_sts_resolver.resolver.STSResolver(loop=loop)
 		valid, policy = loop.run_until_complete(sts_resolver.resolve(domain))
 		if valid == postfix_mta_sts_resolver.resolver.STSFetchResult.VALID:
@@ -781,7 +797,7 @@ def query_dns(qname, rtype, nxdomain='[Not Set]', at=None, as_list=False):
 
 	# Do the query.
 	try:
-		response = resolver.query(qname, rtype)
+		response = resolver.resolve(qname, rtype)
 	except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
 		# Host did not have an answer for this query; not sure what the
 		# difference is between the two exceptions.

management/templates/aliases.html

@@ -1,6 +1,6 @@
 <style>
 #alias_table .actions > * { padding-right: 3px; }
-#alias_table .alias-required .remove { display: none }
+#alias_table .alias-auto .actions > * { display: none }
 </style>
 
 <h2>Aliases</h2>
@@ -163,7 +163,7 @@ function show_aliases() {
           var n = $("#alias-template").clone();
           n.attr('id', '');
 
-          if (alias.required) n.addClass('alias-required');
+          if (alias.auto) n.addClass('alias-auto');
           n.attr('data-address', alias.address_display); // this is decoded from IDNA, but will get re-coded to IDNA on the backend
           n.find('td.address').text(alias.address_display)
           for (var j = 0; j < alias.forwards_to.length; j++)

management/templates/external-dns.html

@@ -38,7 +38,7 @@
 <p class="alert" role="alert">
 	<span class="glyphicon glyphicon-info-sign"></span>
 	You may encounter zone file errors when attempting to create a TXT record with a long string.
-	<a href="http://tools.ietf.org/html/rfc4408#section-3.1.3">RFC 4408</a> states a TXT record is allowed to contain multiple strings, and this technique can be used to construct records that would exceed the 255-byte maximum length.
+	<a href="https://tools.ietf.org/html/rfc4408#section-3.1.3">RFC 4408</a> states a TXT record is allowed to contain multiple strings, and this technique can be used to construct records that would exceed the 255-byte maximum length.
 	You may need to adopt this technique when adding DomainKeys. Use a tool like <code>named-checkzone</code> to validate your zone file.
 </p>
 

management/templates/index.html

@@ -62,6 +62,37 @@
 	    ol li {
 	       margin-bottom: 1em;
 	    }
+          
+            .if-logged-in { display: none; }
+            .if-logged-in-admin { display: none; }
+
+      /* The below only gets used if it is supported */
+      @media (prefers-color-scheme: dark) {
+        /* Invert invert lightness but not hue */
+        html {
+          filter: invert(100%) hue-rotate(180deg);
+        }
+
+        /* Set explicit background color (necessary for Firefox) */
+        html {
+          background-color: #111;
+        }
+          
+        /* Override Boostrap theme here to give more contrast. The black turns to white by the filter. */
+        .form-control {
+          color: black !important;
+        }          
+
+        /* Revert the invert for the navbar */
+        button, div.navbar {
+          filter: invert(100%) hue-rotate(180deg);
+        }
+          
+        /* Revert the revert for the dropdowns */
+        ul.dropdown-menu {
+          filter: invert(100%) hue-rotate(180deg);
+        }
+      }
         </style>
         <link rel="stylesheet" href="/admin/assets/bootstrap/css/bootstrap-theme.min.css">
     </head>
@@ -83,7 +114,7 @@
         </div>
         <div class="navbar-collapse collapse">
           <ul class="nav navbar-nav">
-            <li class="dropdown">
+            <li class="dropdown if-logged-in-admin">
               <a href="#" class="dropdown-toggle" data-toggle="dropdown">System <b class="caret"></b></a>
               <ul class="dropdown-menu">
                 <li><a href="#system_status" onclick="return show_panel(this);">Status Checks</a></li>
@@ -93,10 +124,11 @@
                 <li class="dropdown-header">Advanced Pages</li>
                 <li><a href="#custom_dns" onclick="return show_panel(this);">Custom DNS</a></li>
                 <li><a href="#external_dns" onclick="return show_panel(this);">External DNS</a></li>
-                <li><a href="/admin/munin" target="_blank">Munin Monitoring</a></li>
+                <li><a href="#munin" onclick="return show_panel(this);">Munin Monitoring</a></li>
               </ul>
             </li>
-            <li class="dropdown">
+            <li><a href="#mail-guide" onclick="return show_panel(this);" class="if-logged-in-not-admin">Mail</a></li>
+            <li class="dropdown if-logged-in-admin">
               <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mail &amp; Users <b class="caret"></b></a>
               <ul class="dropdown-menu">
                 <li><a href="#mail-guide" onclick="return show_panel(this);">Instructions</a></li>
@@ -107,17 +139,21 @@
                 <li><a href="#mfa" onclick="return show_panel(this);">Two-Factor Authentication</a></li>
               </ul>
             </li>
-            <li><a href="#sync_guide" onclick="return show_panel(this);">Contacts/Calendar</a></li>
+            <li><a href="#sync_guide" onclick="return show_panel(this);" class="if-logged-in">Contacts/Calendar</a></li>
-            <li><a href="#web" onclick="return show_panel(this);">Web</a></li>
+            <li><a href="#web" onclick="return show_panel(this);" class="if-logged-in-admin">Web</a></li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
-            <li><a href="#" onclick="do_logout(); return false;" style="color: white">Log out</a></li>
+            <li class="if-logged-in"><a href="#" onclick="do_logout(); return false;" style="color: white">Log out</a></li>
           </ul>
         </div><!--/.navbar-collapse -->
       </div>
     </div>
 
     <div class="container">
+      <div id="panel_welcome" class="admin_panel">
+      {% include "welcome.html" %}
+      </div>
+
       <div id="panel_system_status" class="admin_panel">
       {% include "system-status.html" %}
       </div>
@@ -166,6 +202,10 @@
       {% include "ssl.html" %}
       </div>
 
+      <div id="panel_munin" class="admin_panel">
+      {% include "munin.html" %}
+      </div>
+
       <hr>
 
       <footer>
@@ -298,7 +338,7 @@ function ajax_with_indicator(options) {
   return false; // handy when called from onclick
 }
 
-var api_credentials = ["", ""];
+var api_credentials = null;
 function api(url, method, data, callback, callback_error, headers) {
   // from http://www.webtoolkit.info/javascript-base64.html
   function base64encode(input) {
@@ -346,9 +386,10 @@ function api(url, method, data, callback, callback_error, headers) {
       // We don't store user credentials in a cookie to avoid the hassle of CSRF
       // attacks. The Authorization header only gets set in our AJAX calls triggered
       // by user actions.
-      xhr.setRequestHeader(
+      if (api_credentials)
-        'Authorization',
+        xhr.setRequestHeader(
-        'Basic ' + base64encode(api_credentials[0] + ':' + api_credentials[1]));
+          'Authorization',
+          'Basic ' + base64encode(api_credentials.username + ':' + api_credentials.session_key));
     },
     success: callback,
     error: callback_error || default_error,
@@ -367,12 +408,21 @@ var current_panel = null;
 var switch_back_to_panel = null;
 
 function do_logout() {
-  api_credentials = ["", ""];
+  // Clear the session from the backend.
+  api("/logout", "POST");
+
+  // Forget the token.
+  api_credentials = null;
   if (typeof localStorage != 'undefined')
     localStorage.removeItem("miab-cp-credentials");
   if (typeof sessionStorage != 'undefined')
     sessionStorage.removeItem("miab-cp-credentials");
+
+  // Return to the start.
   show_panel('login');
+
+  // Reset menus.
+  show_hide_menus();
 }
 
 function show_panel(panelid) {
@@ -395,14 +445,22 @@ function show_panel(panelid) {
 
 $(function() {
   // Recall saved user credentials.
-  if (typeof sessionStorage != 'undefined' && sessionStorage.getItem("miab-cp-credentials"))
+  try {
-    api_credentials = sessionStorage.getItem("miab-cp-credentials").split(":");
+    if (typeof sessionStorage != 'undefined' && sessionStorage.getItem("miab-cp-credentials"))
-  else if (typeof localStorage != 'undefined' && localStorage.getItem("miab-cp-credentials"))
+      api_credentials = JSON.parse(sessionStorage.getItem("miab-cp-credentials"));
-    api_credentials = localStorage.getItem("miab-cp-credentials").split(":");
+    else if (typeof localStorage != 'undefined' && localStorage.getItem("miab-cp-credentials"))
+      api_credentials = JSON.parse(localStorage.getItem("miab-cp-credentials"));
+  } catch (_) {
+  }
+
+  // Toggle menu state.
+  show_hide_menus();
 
   // Recall what the user was last looking at.
-  if (typeof localStorage != 'undefined' && localStorage.getItem("miab-cp-lastpanel")) {
+  if (api_credentials != null && typeof localStorage != 'undefined' && localStorage.getItem("miab-cp-lastpanel")) {
     show_panel(localStorage.getItem("miab-cp-lastpanel"));
+  } else if (api_credentials != null) {
+    show_panel('welcome');
   } else {
     show_panel('login');
   }

management/templates/login.html

@@ -64,7 +64,7 @@ sudo management/cli.py user make-admin me@{{hostname}}</pre>
     <div class="form-group" id="loginOtp">
       <label for="loginOtpInput" class="col-sm-3 control-label">Code</label>
       <div class="col-sm-9">
-          <input type="text" class="form-control" id="loginOtpInput" placeholder="6-digit code">
+          <input type="text" class="form-control" id="loginOtpInput" placeholder="6-digit code" autocomplete="off">
           <div class="help-block" style="margin-top: 5px; font-size: 90%">Enter the six-digit code generated by your two factor authentication app.</div>
       </div>
     </div>
@@ -102,11 +102,11 @@ function do_login() {
   }
 
   // Exchange the email address & password for an API key.
-  api_credentials = [$('#loginEmail').val(), $('#loginPassword').val()]
+  api_credentials = { username: $('#loginEmail').val(), session_key: $('#loginPassword').val() }
 
   api(
-  "/me",
+  "/login",
-  "GET",
+  "POST",
   {},
   function(response) {
     // This API call always succeeds. It returns a JSON object indicating
@@ -141,7 +141,9 @@ function do_login() {
       // Login succeeded.
 
       // Save the new credentials.
-      api_credentials = [response.email, response.api_key];
+      api_credentials = { username: response.email,
+                          session_key: response.api_key,
+                          privileges: response.privileges };
 
       // Try to wipe the username/password information.
       $('#loginEmail').val('');
@@ -152,18 +154,21 @@ function do_login() {
       // Remember the credentials.
       if (typeof localStorage != 'undefined' && typeof sessionStorage != 'undefined') {
         if ($('#loginRemember').val()) {
-          localStorage.setItem("miab-cp-credentials", api_credentials.join(":"));
+          localStorage.setItem("miab-cp-credentials", JSON.stringify(api_credentials));
           sessionStorage.removeItem("miab-cp-credentials");
         } else {
           localStorage.removeItem("miab-cp-credentials");
-          sessionStorage.setItem("miab-cp-credentials", api_credentials.join(":"));
+          sessionStorage.setItem("miab-cp-credentials", JSON.stringify(api_credentials));
         }
       }
 
+      // Toggle menus.
+      show_hide_menus();
+
       // Open the next panel the user wants to go to. Do this after the XHR response
       // is over so that we don't start a new XHR request while this one is finishing,
       // which confuses the loading indicator.
-      setTimeout(function() { show_panel(!switch_back_to_panel || switch_back_to_panel == "login" ? 'system_status' : switch_back_to_panel) }, 300);
+      setTimeout(function() { show_panel(!switch_back_to_panel || switch_back_to_panel == "login" ? 'welcome' : switch_back_to_panel) }, 300);
     }
   },
   undefined,
@@ -183,4 +188,19 @@ function show_login() {
     }
   });
 }
+
+function show_hide_menus() {
+  var is_logged_in = (api_credentials != null);
+  var privs = api_credentials ? api_credentials.privileges : [];
+  $('.if-logged-in').toggle(is_logged_in);
+  $('.if-logged-in-admin, .if-logged-in-not-admin').toggle(false);
+  if (is_logged_in) {
+    $('.if-logged-in-not-admin').toggle(true);
+    privs.forEach(function(priv) {
+      $('.if-logged-in-' + priv).toggle(true);
+      $('.if-logged-in-not-' + priv).toggle(false);
+    });
+  }
+  $('.if-not-logged-in').toggle(!is_logged_in);
+}
 </script>

management/templates/munin.html

@@ -0,0 +1,20 @@
+<h2>Munin Monitoring</h2>
+
+<style>
+</style>
+
+<p>Opening munin in a new tab... You may need to allow pop-ups for this site.</p>
+
+<script>
+function show_munin() {
+  // Set the cookie.
+  api(
+    "/munin",
+    "GET",
+    { },
+    function(r) {
+      // Redirect.
+      window.open("/admin/munin/index.html", "_blank");
+    });
+}
+</script>

management/templates/sync-guide.html

@@ -30,9 +30,9 @@
 
 			<table class="table">
 			<thead><tr><th>For...</th> <th>Use...</th></tr></thead>
-			<tr><td>Contacts and Calendar</td> <td><a href="https://play.google.com/store/apps/details?id=at.bitfire.davdroid">DAVdroid</a> ($3.69; free <a href="https://f-droid.org/packages/at.bitfire.davdroid/">here</a>)</td></tr>
+			<tr><td>Contacts and Calendar</td> <td><a href="https://play.google.com/store/apps/details?id=at.bitfire.davdroid">DAVx⁵</a> ($5.99; free <a href="https://f-droid.org/packages/at.bitfire.davdroid/">here</a>)</td></tr>
-			<tr><td>Only Contacts</td> <td><a href="https://play.google.com/store/apps/details?id=org.dmfs.carddav.sync">CardDAV-Sync free beta</a> (free)</td></tr>
+			<tr><td>Only Contacts</td> <td><a href="https://play.google.com/store/apps/details?id=org.dmfs.carddav.sync">CardDAV-Sync free</a> (free)</td></tr>
-			<tr><td>Only Calendar</td> <td><a href="https://play.google.com/store/apps/details?id=org.dmfs.caldav.lib">CalDAV-Sync</a> ($2.89)</td></tr>
+			<tr><td>Only Calendar</td> <td><a href="https://play.google.com/store/apps/details?id=org.dmfs.caldav.lib">CalDAV-Sync</a> ($2.99)</td></tr>
 			</table>
 
 			<p>Use the following settings:</p>

management/templates/system-backup.html

@@ -5,7 +5,7 @@
 
 <h2>Backup Status</h2>
 
-<p>The box makes an incremental backup each night. By default the backup is stored on the machine itself, but you can also store in on S3-compatible services like Amazon Web Services (AWS).</p>
+<p>The box makes an incremental backup each night. By default the backup is stored on the machine itself, but you can also store it on S3-compatible services like Amazon Web Services (AWS).</p>
 
 <h3>Configuration</h3>
 
@@ -138,7 +138,7 @@
       </div>
   </div>
   <!-- Common -->
-  <div class="form-group backup-target-local backup-target-rsync backup-target-s3">
+  <div class="form-group backup-target-local backup-target-rsync backup-target-s3 backup-target-b2">
     <label for="min-age" class="col-sm-2 control-label">Retention Days:</label>
     <div class="col-sm-8">
       <input type="number" class="form-control" rows="1" id="min-age">
@@ -269,6 +269,7 @@ function show_custom_backup() {
           $("#backup-target-type").val("s3");
           var hostpath = r.target.substring(5).split('/');
           var host = hostpath.shift();
+          $("#backup-target-s3-host-select").val(host);
           $("#backup-target-s3-host").val(host);
           $("#backup-target-s3-path").val(hostpath.join('/'));
         } else if (r.target.substring(0, 5) == "b2://") {

management/templates/users.html

@@ -203,7 +203,7 @@ function users_set_password(elem) {
   var email = $(elem).parents('tr').attr('data-email');
 
   var yourpw = "";
-  if (api_credentials != null && email == api_credentials[0])
+  if (api_credentials != null && email == api_credentials.username)
     yourpw = "<p class='text-danger'>If you change your own password, you will be logged out of this control panel and will need to log in again.</p>";
 
   show_modal_confirm(
@@ -232,7 +232,7 @@ function users_remove(elem) {
   var email = $(elem).parents('tr').attr('data-email');
 
   // can't remove yourself
-  if (api_credentials != null && email == api_credentials[0]) {
+  if (api_credentials != null && email == api_credentials.username) {
     show_modal_error("Archive User", "You cannot archive your own account.");
     return;
   }
@@ -264,7 +264,7 @@ function mod_priv(elem, add_remove) {
   var priv = $(elem).parents('td').find('.name').text();
 
   // can't remove your own admin access
-  if (priv == "admin" && add_remove == "remove" && api_credentials != null && email == api_credentials[0]) {
+  if (priv == "admin" && add_remove == "remove" && api_credentials != null && email == api_credentials.username) {
     show_modal_error("Modify Privileges", "You cannot remove the admin privilege from yourself.");
     return;
   }

management/templates/welcome.html

@@ -0,0 +1,16 @@
+<style>
+  .title {
+    margin: 1em;
+    text-align: center;
+  }
+
+  .subtitle {
+    margin: 2em;
+    text-align: center;
+  }
+</style>
+
+<h1 class="title">{{hostname}}</h1>
+
+<p class="subtitle">Welcome to your Mail-in-a-Box control panel.</p>
+

management/utils.py

@@ -175,14 +175,6 @@ def wait_for_service(port, public, env, timeout):
 				return False
 		time.sleep(min(timeout/4, 1))
 
-def fix_boto():
-	# Google Compute Engine instances install some Python-2-only boto plugins that
-	# conflict with boto running under Python 3. Disable boto's default configuration
-	# file prior to importing boto so that GCE's plugin is not loaded:
-	import os
-	os.environ["BOTO_CONFIG"] = "/etc/boto3.cfg"
-
-
 if __name__ == "__main__":
 	from web_update import get_web_domains
 	env = load_environment()

management/wsgi.py

@@ -0,0 +1,7 @@
+from daemon import app
+import auth, utils
+
+app.logger.addHandler(utils.create_syslog_handler())
+
+if __name__ == "__main__":
+    app.run(port=10222)

security.md

@@ -3,7 +3,12 @@ Mail-in-a-Box Security Guide
 
 Mail-in-a-Box turns a fresh Ubuntu 18.04 LTS 64-bit machine into a mail server appliance by installing and configuring various components.
 
-This page documents the security features of Mail-in-a-Box. The term “box” is used below to mean a configured Mail-in-a-Box.
+This page documents the security posture of Mail-in-a-Box. The term “box” is used below to mean a configured Mail-in-a-Box.
+
+Reporting Security Vulnerabilities
+----------------------------------
+
+Security vulnerabilities should be reported to the [project's maintainer](https://joshdata.me) via email.
 
 Threat Model
 ------------
@@ -49,9 +54,7 @@ Additionally:
 
 ### Password Storage
 
-The passwords for mail users are stored on disk using the [SHA512-CRYPT](http://man7.org/linux/man-pages/man3/crypt.3.html) hashing scheme. ([source](management/mailconfig.py))
+The passwords for mail users are stored on disk using the [SHA512-CRYPT](http://man7.org/linux/man-pages/man3/crypt.3.html) hashing scheme. ([source](management/mailconfig.py)) Password changes (as well as changes to control panel two-factor authentication settings) expire any control panel login sessions.
-
-When using the web-based administrative control panel, after logging in an API key is placed in the browser's local storage (rather than, say, the user's actual password). The API key is an HMAC based on the user's email address and current password, and it is keyed by a secret known only to the control panel service. By resetting an administrator's password, any HMACs previously generated for that user will expire.
 
 ### Console access
 
@@ -65,12 +68,10 @@ If DNSSEC is enabled at the box's domain name's registrar, the SSHFP record that
 
 `fail2ban` provides some protection from brute-force login attacks (repeated logins that guess account passwords) by blocking offending IP addresses at the network level.
 
-The following services are protected: SSH, IMAP (dovecot), SMTP submission (postfix), webmail (roundcube), Nextcloud/CalDAV/CardDAV (over HTTP), and the Mail-in-a-Box control panel & munin (over HTTP).
+The following services are protected: SSH, IMAP (dovecot), SMTP submission (postfix), webmail (roundcube), Nextcloud/CalDAV/CardDAV (over HTTP), and the Mail-in-a-Box control panel (over HTTP).
 
 Some other services running on the box may be missing fail2ban filters.
 
-`fail2ban` only blocks IPv4 addresses, however. If the box has a public IPv6 address, it is not protected from these attacks.
-
 Outbound Mail
 -------------
 

setup/bootstrap.sh

@@ -9,32 +9,37 @@
 if [ -z "$TAG" ]; then
 	# If a version to install isn't explicitly given as an environment
 	# variable, then install the latest version. But the latest version
-	# depends on the operating system. Existing Ubuntu 14.04 users need
+	# depends on the machine's version of Ubuntu. Existing users need to
-	# to be able to upgrade to the latest version supporting Ubuntu 14.04,
+	# be able to upgrade to the latest version available for that version
-	# in part because an upgrade is required before jumping to Ubuntu 18.04.
+	# of Ubuntu to satisfy the migration requirements.
-	# New users on Ubuntu 18.04 need to get the latest version number too.
 	#
 	# Also, the system status checks read this script for TAG = (without the
 	# space, but if we put it in a comment it would confuse the status checks!)
 	# to get the latest version, so the first such line must be the one that we
 	# want to display in status checks.
-	if [ "$(lsb_release -d | sed 's/.*:\s*//' | sed 's/18\.04\.[0-9]/18.04/' )" == "Ubuntu 18.04 LTS" ]; then
+	#
-		# This machine is running Ubuntu 18.04.
+	# Allow point-release versions of the major releases, e.g. 22.04.1 is OK.
-		TAG=v0.54
+	UBUNTU_VERSION=$( lsb_release -d | sed 's/.*:\s*//' | sed 's/\([0-9]*\.[0-9]*\)\.[0-9]/\1/' )
-
+	if [ "$UBUNTU_VERSION" == "Ubuntu 22.04 LTS" ]; then
-	elif [ "$(lsb_release -d | sed 's/.*:\s*//' | sed 's/14\.04\.[0-9]/14.04/' )" == "Ubuntu 14.04 LTS" ]; then
+		# This machine is running Ubuntu 22.04, which is supported by
-		# This machine is running Ubuntu 14.04.
+		# Mail-in-a-Box versions 60 and later.
-		echo "You are installing the last version of Mail-in-a-Box that will"
+		TAG=v60.1
-		echo "support Ubuntu 14.04. If this is a new installation of Mail-in-a-Box,"
+	elif [ "$UBUNTU_VERSION" == "Ubuntu 18.04 LTS" ]; then
-		echo "stop now and switch to a machine running Ubuntu 18.04. If you are"
+		# This machine is running Ubuntu 18.04, which is supported by
-		echo "upgrading an existing Mail-in-a-Box --- great. After upgrading this"
+		# Mail-in-a-Box versions 0.40 through 5x.
-		echo "box, please visit https://mailinabox.email for notes on how to upgrade"
+		echo "Support is ending for Ubuntu 18.04."
-		echo "to Ubuntu 18.04."
+		echo "Please immediately begin to migrate your data to"
-		echo ""
+		echo "a new machine running Ubuntu 22.04. See:"
+		echo "https://mailinabox.email/maintenance.html#upgrade"
+		TAG=v57a
+	elif [ "$UBUNTU_VERSION" == "Ubuntu 14.04 LTS" ]; then
+		# This machine is running Ubuntu 14.04, which is supported by
+		# Mail-in-a-Box versions 1 through v0.30.
+		echo "Ubuntu 14.04 is no longer supported."
+		echo "The last version of Mail-in-a-Box supporting Ubuntu 14.04 will be installed."
 		TAG=v0.30
-
 	else
-		echo "This script must be run on a system running Ubuntu 18.04 or Ubuntu 14.04."
+		echo "This script may be used only on a machine running Ubuntu 14.04, 18.04, or 22.04."
 		exit 1
 	fi
 fi

setup/dns.sh

@@ -10,17 +10,13 @@
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
-# Install the packages.
-#
-# * nsd: The non-recursive nameserver that publishes our DNS records.
-# * ldnsutils: Helper utilities for signing DNSSEC zones.
-# * openssh-client: Provides ssh-keyscan which we use to create SSHFP records.
-echo "Installing nsd (DNS server)..."
-apt_install nsd ldnsutils openssh-client
-
 # Prepare nsd's configuration.
-
+# We configure nsd before installation as we only want it to bind to some addresses
+# and it otherwise will have port / bind conflicts with bind9 used as the local resolver
 mkdir -p /var/run/nsd
+mkdir -p /etc/nsd
+mkdir -p /etc/nsd/zones
+touch /etc/nsd/zones.conf
 
 cat > /etc/nsd/nsd.conf << EOF;
 # Do not edit. Overwritten by Mail-in-a-Box setup.
@@ -42,6 +38,22 @@ server:
 
 EOF
 
+# Since we have bind9 listening on localhost for locally-generated
+# DNS queries that require a recursive nameserver, and the system
+# might have other network interfaces for e.g. tunnelling, we have
+# to be specific about the network interfaces that nsd binds to.
+for ip in $PRIVATE_IP $PRIVATE_IPV6; do
+	echo "  ip-address: $ip" >> /etc/nsd/nsd.conf;
+done
+
+# Create a directory for additional configuration directives, including
+# the zones.conf file written out by our management daemon.
+echo "include: /etc/nsd/nsd.conf.d/*.conf" >> /etc/nsd/nsd.conf;
+
+# Remove the old location of zones.conf that we generate. It will
+# now be stored in /etc/nsd/nsd.conf.d.
+rm -f /etc/nsd/zones.conf
+
 # Add log rotation
 cat > /etc/logrotate.d/nsd <<EOF;
 /var/log/nsd.log {
@@ -54,15 +66,13 @@ cat > /etc/logrotate.d/nsd <<EOF;
 }
 EOF
 
-# Since we have bind9 listening on localhost for locally-generated
+# Install the packages.
-# DNS queries that require a recursive nameserver, and the system
+#
-# might have other network interfaces for e.g. tunnelling, we have
+# * nsd: The non-recursive nameserver that publishes our DNS records.
-# to be specific about the network interfaces that nsd binds to.
+# * ldnsutils: Helper utilities for signing DNSSEC zones.
-for ip in $PRIVATE_IP $PRIVATE_IPV6; do
+# * openssh-client: Provides ssh-keyscan which we use to create SSHFP records.
-	echo "  ip-address: $ip" >> /etc/nsd/nsd.conf;
+echo "Installing nsd (DNS server)..."
-done
+apt_install nsd ldnsutils openssh-client
-
-echo "include: /etc/nsd/zones.conf" >> /etc/nsd/nsd.conf;
 
 # Create DNSSEC signing keys.
 

setup/functions.sh

@@ -4,6 +4,8 @@
 # -o pipefail: don't ignore errors in the non-last command in a pipeline
 set -euo pipefail
 
+PHP_VER=8.0
+
 function hide_output {
 	# This function hides the output of a command unless the command fails
 	# and returns a non-zero exit code.

setup/mail-dovecot.sh

@@ -84,10 +84,11 @@ tools/editconf.py /etc/dovecot/conf.d/10-ssl.conf \
 	ssl=required \
 	"ssl_cert=<$STORAGE_ROOT/ssl/ssl_certificate.pem" \
 	"ssl_key=<$STORAGE_ROOT/ssl/ssl_private_key.pem" \
-	"ssl_protocols=TLSv1.2" \
+	"ssl_min_protocol=TLSv1.2" \
 	"ssl_cipher_list=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" \
 	"ssl_prefer_server_ciphers=no" \
-	"ssl_dh_parameters_length=2048"
+	"ssl_dh_parameters_length=2048" \
+	"ssl_dh=<$STORAGE_ROOT/ssl/dh2048.pem"
 
 # Disable in-the-clear IMAP/POP because there is no reason for a user to transmit
 # login credentials outside of an encrypted connection. Only the over-TLS versions

setup/mail-postfix.sh

@@ -13,8 +13,8 @@
 # destinations according to aliases, and passses email on to
 # another service for local mail delivery.
 #
-# The first hop in local mail delivery is to Spamassassin via
+# The first hop in local mail delivery is to spampd via
-# LMTP. Spamassassin then passes mail over to Dovecot for
+# LMTP. spampd then passes mail over to Dovecot for
 # storage in the user's mailbox.
 #
 # Postfix also listens on ports 465/587 (SMTPS, SMTP+STARTLS) for
@@ -193,16 +193,17 @@ tools/editconf.py /etc/postfix/main.cf \
 
 # ### Incoming Mail
 
-# Pass any incoming mail over to a local delivery agent. Spamassassin
+# Pass mail to spampd, which acts as the local delivery agent (LDA),
-# will act as the LDA agent at first. It is listening on port 10025
+# which then passes the mail over to the Dovecot LMTP server after.
-# with LMTP. Spamassassin will pass the mail over to Dovecot after.
+# spampd runs on port 10025 by default.
 #
 # In a basic setup we would pass mail directly to Dovecot by setting
 # virtual_transport to `lmtp:unix:private/dovecot-lmtp`.
 tools/editconf.py /etc/postfix/main.cf "virtual_transport=lmtp:[127.0.0.1]:10025"
-# Because of a spampd bug, limit the number of recipients in each connection.
+# Clear the lmtp_destination_recipient_limit setting which in previous
+# versions of Mail-in-a-Box was set to 1 because of a spampd bug.
 # See https://github.com/mail-in-a-box/mailinabox/issues/1523.
-tools/editconf.py /etc/postfix/main.cf lmtp_destination_recipient_limit=1
+tools/editconf.py /etc/postfix/main.cf  -e lmtp_destination_recipient_limit=
 
 
 # Who can send mail to us? Some basic filters.
@@ -232,11 +233,32 @@ tools/editconf.py /etc/postfix/main.cf \
 # As a matter of fact RFC is not strict about retry timer so postfix and
 # other MTA have their own intervals. To fix the problem of receiving
 # e-mails really latter, delay of greylisting has been set to
-# 180 seconds (default is 300 seconds).
+# 180 seconds (default is 300 seconds). We will move the postgrey database
+# under $STORAGE_ROOT. This prevents a "warming up" that would have occured
+# previously with a migrated or reinstalled OS.  We will specify this new path
+# with the --dbdir=... option. Arguments within POSTGREY_OPTS can not have spaces,
+# including dbdir. This is due to the way the init script sources the
+# /etc/default/postgrey file. --dbdir=... either needs to be a path without spaces
+# (luckily $STORAGE_ROOT does not currently work with spaces), or it needs to be a
+# symlink without spaces that can point to a folder with spaces).  We'll just assume
+# $STORAGE_ROOT won't have spaces to simplify things.
 tools/editconf.py /etc/default/postgrey \
-	POSTGREY_OPTS=\"'--inet=127.0.0.1:10023 --delay=180'\"
+	POSTGREY_OPTS=\""--inet=127.0.0.1:10023 --delay=180 --dbdir=$STORAGE_ROOT/mail/postgrey/db"\"
 
 
+# If the $STORAGE_ROOT/mail/postgrey is empty, copy the postgrey database over from the old location
+if [ ! -d $STORAGE_ROOT/mail/postgrey/db ]; then
+	# Stop the service
+	service postgrey stop
+	# Ensure the new paths for postgrey db exists
+	mkdir -p $STORAGE_ROOT/mail/postgrey/db
+	# Move over database files
+	mv /var/lib/postgrey/* $STORAGE_ROOT/mail/postgrey/db/ || true
+fi
+# Ensure permissions are set
+chown -R postgrey:postgrey $STORAGE_ROOT/mail/postgrey/
+chmod 700 $STORAGE_ROOT/mail/postgrey/{,db}
+
 # We are going to setup a newer whitelist for postgrey, the version included in the distribution is old
 cat > /etc/cron.daily/mailinabox-postgrey-whitelist << EOF;
 #!/bin/bash

setup/mail-users.sh

@@ -23,6 +23,7 @@ if [ ! -f $db_path ]; then
 	echo "CREATE TABLE users (id INTEGER PRIMARY KEY AUTOINCREMENT, email TEXT NOT NULL UNIQUE, password TEXT NOT NULL, extra, privileges TEXT NOT NULL DEFAULT '');" | sqlite3 $db_path;
 	echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 $db_path;
 	echo "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);" | sqlite3 $db_path;
+	echo "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);" | sqlite3 $db_path;
 fi
 
 # ### User Authentication
@@ -100,8 +101,12 @@ EOF
 # ### Destination Validation
 
 # Use a Sqlite3 database to check whether a destination email address exists,
-# and to perform any email alias rewrites in Postfix.
+# and to perform any email alias rewrites in Postfix. Additionally, we disable
+# SMTPUTF8 because Dovecot's LMTP server that delivers mail to inboxes does
+# not support it, and if a message is received with the SMTPUTF8 flag it will
+# bounce.
 tools/editconf.py /etc/postfix/main.cf \
+	smtputf8_enable=no \
 	virtual_mailbox_domains=sqlite:/etc/postfix/virtual-mailbox-domains.cf \
 	virtual_mailbox_maps=sqlite:/etc/postfix/virtual-mailbox-maps.cf \
 	virtual_alias_maps=sqlite:/etc/postfix/virtual-alias-maps.cf \
@@ -110,7 +115,7 @@ tools/editconf.py /etc/postfix/main.cf \
 # SQL statement to check if we handle incoming mail for a domain, either for users or aliases.
 cat > /etc/postfix/virtual-mailbox-domains.cf << EOF;
 dbpath=$db_path
-query = SELECT 1 FROM users WHERE email LIKE '%%@%s' UNION SELECT 1 FROM aliases WHERE source LIKE '%%@%s'
+query = SELECT 1 FROM users WHERE email LIKE '%%@%s' UNION SELECT 1 FROM aliases WHERE source LIKE '%%@%s' UNION SELECT 1 FROM auto_aliases WHERE source LIKE '%%@%s'
 EOF
 
 # SQL statement to check if we handle incoming mail for a user.
@@ -145,7 +150,7 @@ EOF
 # empty destination here so that other lower priority rules might match.
 cat > /etc/postfix/virtual-alias-maps.cf << EOF;
 dbpath=$db_path
-query = SELECT destination from (SELECT destination, 0 as priority FROM aliases WHERE source='%s' AND destination<>'' UNION SELECT email as destination, 1 as priority FROM users WHERE email='%s') ORDER BY priority LIMIT 1;
+query = SELECT destination from (SELECT destination, 0 as priority FROM aliases WHERE source='%s' AND destination<>'' UNION SELECT email as destination, 1 as priority FROM users WHERE email='%s' UNION SELECT destination, 2 as priority FROM auto_aliases WHERE source='%s' AND destination<>'') ORDER BY priority LIMIT 1;
 EOF
 
 # Restart Services

setup/management.sh

@@ -1,23 +1,12 @@
 #!/bin/bash
 
 source setup/functions.sh
+source /etc/mailinabox.conf # load global vars
 
 echo "Installing Mail-in-a-Box system management daemon..."
 
 # DEPENDENCIES
 
-# We used to install management daemon-related Python packages
-# directly to /usr/local/lib. We moved to a virtualenv because
-# these packages might conflict with apt-installed packages.
-# We may have a lingering version of acme that conflcits with
-# certbot, which we're about to install below, so remove it
-# first. Once acme is installed by an apt package, this might
-# break the package version and `apt-get install --reinstall python3-acme`
-# might be needed in that case.
-while [ -d /usr/local/lib/python3.4/dist-packages/acme ]; do
-	pip3 uninstall -y acme;
-done
-
 # duplicity is used to make backups of user data.
 #
 # virtualenv is used to isolate the Python 3 packages we
@@ -25,12 +14,12 @@ done
 #
 # certbot installs EFF's certbot which we use to
 # provision free TLS certificates.
-apt_install duplicity python-pip virtualenv certbot
+apt_install duplicity python3-pip virtualenv certbot rsync
 
 # b2sdk is used for backblaze backups.
-# boto is used for amazon aws backups.
+# boto3 is used for amazon aws backups.
 # Both are installed outside the pipenv, so they can be used by duplicity
-hide_output pip3 install --upgrade b2sdk boto
+hide_output pip3 install --upgrade b2sdk boto3
 
 # Create a virtualenv for the installation of Python 3 packages
 # used by the management daemon.
@@ -49,9 +38,10 @@ hide_output $venv/bin/pip install --upgrade pip
 # NOTE: email_validator is repeated in setup/questions.sh, so please keep the versions synced.
 hide_output $venv/bin/pip install --upgrade \
 	rtyaml "email_validator>=1.0.0" "exclusiveprocess" \
-	flask dnspython python-dateutil \
+	flask dnspython python-dateutil expiringdict gunicorn \
-  qrcode[pil] pyotp \
+	qrcode[pil] pyotp \
-	"idna>=2.0.0" "cryptography==2.2.2" boto psutil postfix-mta-sts-resolver b2sdk
+	"idna>=2.0.0" "cryptography==37.0.2" psutil postfix-mta-sts-resolver \
+	b2sdk boto3
 
 # CONFIGURATION
 
@@ -88,6 +78,9 @@ rm -f /tmp/bootstrap.zip
 
 # Create an init script to start the management daemon and keep it
 # running after a reboot.
+# Set a long timeout since some commands take a while to run, matching
+# the timeout we set for PHP (fastcgi_read_timeout in the nginx confs).
+# Note: Authentication currently breaks with more than 1 gunicorn worker.
 cat > $inst_dir/start <<EOF;
 #!/bin/bash
 # Set character encoding flags to ensure that any non-ASCII don't cause problems.
@@ -96,8 +89,13 @@ export LC_ALL=en_US.UTF-8
 export LANG=en_US.UTF-8
 export LC_TYPE=en_US.UTF-8
 
+mkdir -p /var/lib/mailinabox
+tr -cd '[:xdigit:]' < /dev/urandom | head -c 32 > /var/lib/mailinabox/api.key
+chmod 640 /var/lib/mailinabox/api.key
+
 source $venv/bin/activate
-exec python $(pwd)/management/daemon.py
+export PYTHONPATH=$(pwd)/management
+exec gunicorn -b localhost:10222 -w 1 --timeout 630 wsgi:app
 EOF
 chmod +x $inst_dir/start
 cp --remove-destination conf/mailinabox.service /lib/systemd/system/mailinabox.service # target was previously a symlink so remove it first

setup/migrate.py

@@ -186,6 +186,11 @@ def migration_13(env):
 	db = os.path.join(env["STORAGE_ROOT"], 'mail/users.sqlite')
 	shell("check_call", ["sqlite3", db, "CREATE TABLE mfa (id INTEGER PRIMARY KEY AUTOINCREMENT, user_id INTEGER NOT NULL, type TEXT NOT NULL, secret TEXT NOT NULL, mru_token TEXT, label TEXT, FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE);"])
 
+def migration_14(env):
+	# Add the "auto_aliases" table.
+	db = os.path.join(env["STORAGE_ROOT"], 'mail/users.sqlite')
+	shell("check_call", ["sqlite3", db, "CREATE TABLE auto_aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL, permitted_senders TEXT);"])
+
 ###########################################################
 
 def get_current_migration():

setup/nextcloud.sh

@@ -9,12 +9,50 @@ source /etc/mailinabox.conf # load global vars
 
 echo "Installing Nextcloud (contacts/calendar)..."
 
+# Nextcloud core and app (plugin) versions to install.
+# With each version we store a hash to ensure we install what we expect.
+
+# Nextcloud core
+# --------------
+# * See https://nextcloud.com/changelog for the latest version.
+# * Check https://docs.nextcloud.com/server/latest/admin_manual/installation/system_requirements.html
+#   for whether it supports the version of PHP available on this machine.
+# * Since Nextcloud only supports upgrades from consecutive major versions,
+#   we automatically install intermediate versions as needed.
+# * The hash is the SHA1 hash of the ZIP package, which you can find by just running this script and
+#   copying it from the error message when it doesn't match what is below.
+nextcloud_ver=23.0.10
+nextcloud_hash=8831c7862e39460fbb789bacac8729fab0ba02dd
+
+# Nextcloud apps
+# --------------
+# * Find the most recent tag that is compatible with the Nextcloud version above by
+#   consulting the <dependencies>...<nextcloud> node at:
+#   https://github.com/nextcloud-releases/contacts/blob/main/appinfo/info.xml
+#   https://github.com/nextcloud-releases/calendar/blob/main/appinfo/info.xml
+#   https://github.com/nextcloud/user_external/blob/master/appinfo/info.xml
+# * The hash is the SHA1 hash of the ZIP package, which you can find by just running this script and
+#   copying it from the error message when it doesn't match what is below.
+contacts_ver=4.2.2
+contacts_hash=ca13d608ed8955aa374cb4f31b6026b57ef88887
+calendar_ver=3.5.1
+calendar_hash=c8136a3deb872a3ef73ce1155b58f3ab27ec7110
+user_external_ver=3.0.0
+user_external_hash=0df781b261f55bbde73d8c92da3f99397000972f
+
+# Clear prior packages and install dependencies from apt.
+
 apt-get purge -qq -y owncloud* # we used to use the package manager
 
-apt_install php php-fpm \
+apt_install curl php${PHP_VER} php${PHP_VER}-fpm \
-	php-cli php-sqlite3 php-gd php-imap php-curl php-pear curl \
+	php${PHP_VER}-cli php${PHP_VER}-sqlite3 php${PHP_VER}-gd php${PHP_VER}-imap php${PHP_VER}-curl \
-	php-dev php-gd php-xml php-mbstring php-zip php-apcu php-json \
+	php${PHP_VER}-dev php${PHP_VER}-gd php${PHP_VER}-xml php${PHP_VER}-mbstring php${PHP_VER}-zip php${PHP_VER}-apcu \
-	php-intl php-imagick php-gmp php-bcmath
+	php${PHP_VER}-intl php${PHP_VER}-imagick php${PHP_VER}-gmp php${PHP_VER}-bcmath
+
+# Enable APC before Nextcloud tools are run.
+tools/editconf.py /etc/php/$PHP_VER/mods-available/apcu.ini -c ';' \
+	apc.enabled=1 \
+	apc.enable_cli=1
 
 InstallNextcloud() {
 
@@ -46,18 +84,18 @@ InstallNextcloud() {
 	# their github repositories.
 	mkdir -p /usr/local/lib/owncloud/apps
 
-	wget_verify https://github.com/nextcloud/contacts/releases/download/v$version_contacts/contacts.tar.gz $hash_contacts /tmp/contacts.tgz
+	wget_verify https://github.com/nextcloud-releases/contacts/archive/refs/tags/v$version_contacts.tar.gz $hash_contacts /tmp/contacts.tgz
 	tar xf /tmp/contacts.tgz -C /usr/local/lib/owncloud/apps/
 	rm /tmp/contacts.tgz
 
-	wget_verify https://github.com/nextcloud/calendar/releases/download/v$version_calendar/calendar.tar.gz $hash_calendar /tmp/calendar.tgz
+	wget_verify https://github.com/nextcloud-releases/calendar/archive/refs/tags/v$version_calendar.tar.gz $hash_calendar /tmp/calendar.tgz
 	tar xf /tmp/calendar.tgz -C /usr/local/lib/owncloud/apps/
 	rm /tmp/calendar.tgz
 
 	# Starting with Nextcloud 15, the app user_external is no longer included in Nextcloud core,
 	# we will install from their github repository.
 	if [ -n "$version_user_external" ]; then
-		wget_verify https://github.com/nextcloud/user_external/releases/download/v$version_user_external/user_external-$version_user_external.tar.gz $hash_user_external /tmp/user_external.tgz
+		wget_verify https://github.com/nextcloud-releases/user_external/releases/download/v$version_user_external/user_external-v$version_user_external.tar.gz $hash_user_external /tmp/user_external.tgz
 		tar -xf /tmp/user_external.tgz -C /usr/local/lib/owncloud/apps/
 		rm /tmp/user_external.tgz
 	fi
@@ -79,33 +117,23 @@ InstallNextcloud() {
 	if [ -e $STORAGE_ROOT/owncloud/owncloud.db ]; then
 		# ownCloud 8.1.1 broke upgrades. It may fail on the first attempt, but
 		# that can be OK.
-		sudo -u www-data php /usr/local/lib/owncloud/occ upgrade
+		sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ upgrade
 		if [ \( $? -ne 0 \) -a \( $? -ne 3 \) ]; then
 			echo "Trying ownCloud upgrade again to work around ownCloud upgrade bug..."
-			sudo -u www-data php /usr/local/lib/owncloud/occ upgrade
+			sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ upgrade
 			if [ \( $? -ne 0 \) -a \( $? -ne 3 \) ]; then exit 1; fi
-			sudo -u www-data php /usr/local/lib/owncloud/occ maintenance:mode --off
+			sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ maintenance:mode --off
 			echo "...which seemed to work."
 		fi
 
 		# Add missing indices. NextCloud didn't include this in the normal upgrade because it might take some time.
-		sudo -u www-data php /usr/local/lib/owncloud/occ db:add-missing-indices
+		sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ db:add-missing-indices
 
 		# Run conversion to BigInt identifiers, this process may take some time on large tables.
-		sudo -u www-data php /usr/local/lib/owncloud/occ db:convert-filecache-bigint --no-interaction
+		sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ db:convert-filecache-bigint --no-interaction
 	fi
 }
 
-# Nextcloud Version to install. Checks are done down below to step through intermediate versions.
-nextcloud_ver=20.0.8
-nextcloud_hash=372b0b4bb07c7984c04917aff86b280e68fbe761
-contacts_ver=3.5.1
-contacts_hash=d2ffbccd3ed89fa41da20a1dff149504c3b33b93
-calendar_ver=2.2.0
-calendar_hash=673ad72ca28adb8d0f209015ff2dca52ffad99af
-user_external_ver=1.0.0
-user_external_hash=3bf2609061d7214e7f0f69dd8883e55c4ec8f50a
-
 # Current Nextcloud Version, #1623
 # Checking /usr/local/lib/owncloud/version.php shows version of the Nextcloud application, not the DB
 # $STORAGE_ROOT/owncloud is kept together even during a backup.  It is better to rely on config.php than
@@ -114,7 +142,7 @@ user_external_hash=3bf2609061d7214e7f0f69dd8883e55c4ec8f50a
 
 # If config.php exists, get version number, otherwise CURRENT_NEXTCLOUD_VER is empty.
 if [ -f "$STORAGE_ROOT/owncloud/config.php" ]; then
-	CURRENT_NEXTCLOUD_VER=$(php -r "include(\"$STORAGE_ROOT/owncloud/config.php\"); echo(\$CONFIG['version']);")
+	CURRENT_NEXTCLOUD_VER=$(php$PHP_VER -r "include(\"$STORAGE_ROOT/owncloud/config.php\"); echo(\$CONFIG['version']);")
 else
 	CURRENT_NEXTCLOUD_VER=""
 fi
@@ -123,8 +151,8 @@ fi
 # from the version currently installed, do the install/upgrade
 if [ ! -d /usr/local/lib/owncloud/ ] || [[ ! ${CURRENT_NEXTCLOUD_VER} =~ ^$nextcloud_ver ]]; then
 
-	# Stop php-fpm if running. If theyre not running (which happens on a previously failed install), dont bail.
+	# Stop php-fpm if running. If they are not running (which happens on a previously failed install), dont bail.
-	service php7.2-fpm stop &> /dev/null || /bin/true
+	service php$PHP_VER-fpm stop &> /dev/null || /bin/true
 
 	# Backup the existing ownCloud/Nextcloud.
 	# Create a backup directory to store the current installation and database to
@@ -151,41 +179,21 @@ if [ ! -d /usr/local/lib/owncloud/ ] || [[ ! ${CURRENT_NEXTCLOUD_VER} =~ ^$nextc
 		elif [[ ${CURRENT_NEXTCLOUD_VER} =~ ^1[012] ]]; then
 			echo "Upgrades from Mail-in-a-Box prior to v0.28 (dated July 30, 2018) with Nextcloud < 13.0.6 (you have ownCloud 10, 11 or 12) are not supported. Upgrade to Mail-in-a-Box version v0.30 first. Setup will continue, but skip the Nextcloud migration."
 			return 0
-		elif [[ ${CURRENT_NEXTCLOUD_VER} =~ ^13 ]]; then
+		elif [[ ${CURRENT_NEXTCLOUD_VER} =~ ^1[3456789] ]]; then
-			# If we are running Nextcloud 13, upgrade to Nextcloud 14
+			echo "Upgrades from Mail-in-a-Box prior to v60 with Nextcloud 19 or earlier are not supported. Upgrade to the latest Mail-in-a-Box version supported on your machine first. Setup will continue, but skip the Nextcloud migration."
-			InstallNextcloud 14.0.6 4e43a57340f04c2da306c8eea98e30040399ae5a 3.3.0 e55d0357c6785d3b1f3b5f21780cb6d41d32443a 2.0.3 9d9717b29337613b72c74e9914c69b74b346c466
+			return 0
-			CURRENT_NEXTCLOUD_VER="14.0.6"
 		fi
-		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^14 ]]; then
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^20 ]]; then
-			# During the upgrade from Nextcloud 14 to 15, user_external may cause the upgrade to fail.
+			InstallNextcloud 21.0.7 f5c7079c5b56ce1e301c6a27c0d975d608bb01c9 4.0.7 45e7cf4bfe99cd8d03625cf9e5a1bb2e90549136 3.0.4 d0284b68135777ec9ca713c307216165b294d0fe
-			# We will disable it here before the upgrade and install it again after the upgrade.
+			CURRENT_NEXTCLOUD_VER="21.0.7"
-			hide_output sudo -u www-data php /usr/local/lib/owncloud/console.php app:disable user_external
+		fi
-			InstallNextcloud 15.0.8 4129d8d4021c435f2e86876225fb7f15adf764a3 3.3.0 e55d0357c6785d3b1f3b5f21780cb6d41d32443a 2.0.3 9d9717b29337613b72c74e9914c69b74b346c466 0.7.0 555a94811daaf5bdd336c5e48a78aa8567b86437
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^21 ]]; then
-			CURRENT_NEXTCLOUD_VER="15.0.8"
+			InstallNextcloud 22.2.6 9d39741f051a8da42ff7df46ceef2653a1dc70d9 4.1.0 697f6b4a664e928d72414ea2731cb2c9d1dc3077 3.2.2 ce4030ab57f523f33d5396c6a81396d440756f5f 3.0.0 0df781b261f55bbde73d8c92da3f99397000972f
+			CURRENT_NEXTCLOUD_VER="22.2.6"
 		fi
-		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^15 ]]; then
-                        InstallNextcloud 16.0.6 0bb3098455ec89f5af77a652aad553ad40a88819 3.3.0 e55d0357c6785d3b1f3b5f21780cb6d41d32443a 2.0.3 9d9717b29337613b72c74e9914c69b74b346c466 0.7.0 555a94811daaf5bdd336c5e48a78aa8567b86437
-                        CURRENT_NEXTCLOUD_VER="16.0.6"
-        	fi
-	        if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^16 ]]; then
-                        InstallNextcloud 17.0.6 50b98d2c2f18510b9530e558ced9ab51eb4f11b0 3.3.0 e55d0357c6785d3b1f3b5f21780cb6d41d32443a 2.0.3 9d9717b29337613b72c74e9914c69b74b346c466 0.7.0 555a94811daaf5bdd336c5e48a78aa8567b86437
-                        CURRENT_NEXTCLOUD_VER="17.0.6"
-	        fi
-	        if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^17 ]]; then
-			echo "ALTER TABLE oc_flow_operations ADD COLUMN entity VARCHAR;" | sqlite3 $STORAGE_ROOT/owncloud/owncloud.db
-                        InstallNextcloud 18.0.10 39c0021a8b8477c3f1733fddefacfa5ebf921c68 3.4.1 aee680a75e95f26d9285efd3c1e25cf7f3bfd27e 2.0.3 9d9717b29337613b72c74e9914c69b74b346c466 1.0.0 3bf2609061d7214e7f0f69dd8883e55c4ec8f50a
-                        CURRENT_NEXTCLOUD_VER="18.0.10"
-	        fi
-	        if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^18 ]]; then
-                        InstallNextcloud 19.0.4 01e98791ba12f4860d3d4047b9803f97a1b55c60 3.4.1 aee680a75e95f26d9285efd3c1e25cf7f3bfd27e 2.0.3 9d9717b29337613b72c74e9914c69b74b346c466 1.0.0 3bf2609061d7214e7f0f69dd8883e55c4ec8f50a
-                        CURRENT_NEXTCLOUD_VER="19.0.4"
-	        fi
 	fi
 
 	InstallNextcloud $nextcloud_ver $nextcloud_hash $contacts_ver $contacts_hash $calendar_ver $calendar_hash $user_external_ver $user_external_hash
-
-	# Nextcloud 20 needs to have some optional columns added
-	sudo -u www-data php /usr/local/lib/owncloud/occ db:add-missing-columns
 fi
 
 # ### Configuring Nextcloud
@@ -211,10 +219,10 @@ if [ ! -f $STORAGE_ROOT/owncloud/owncloud.db ]; then
   'overwrite.cli.url' => '/cloud',
   'user_backends' => array(
     array(
-      'class' => 'OC_User_IMAP',
+      'class' => '\OCA\UserExternal\IMAP',
-        'arguments' => array(
+      'arguments' => array(
-          '127.0.0.1', 143, null
+        '127.0.0.1', 143, null, null, false, false
-         ),
+       ),
     ),
   ),
   'memcache.local' => '\OC\Memcache\APCu',
@@ -256,7 +264,7 @@ EOF
 	# Execute Nextcloud's setup step, which creates the Nextcloud sqlite database.
 	# It also wipes it if it exists. And it updates config.php with database
 	# settings and deletes the autoconfig.php file.
-	(cd /usr/local/lib/owncloud; sudo -u www-data php /usr/local/lib/owncloud/index.php;)
+	(cd /usr/local/lib/owncloud; sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/index.php;)
 fi
 
 # Update config.php.
@@ -272,10 +280,12 @@ fi
 # Use PHP to read the settings file, modify it, and write out the new settings array.
 TIMEZONE=$(cat /etc/timezone)
 CONFIG_TEMP=$(/bin/mktemp)
-php <<EOF > $CONFIG_TEMP && mv $CONFIG_TEMP $STORAGE_ROOT/owncloud/config.php;
+php$PHP_VER <<EOF > $CONFIG_TEMP && mv $CONFIG_TEMP $STORAGE_ROOT/owncloud/config.php;
 <?php
 include("$STORAGE_ROOT/owncloud/config.php");
 
+\$CONFIG['config_is_read_only'] = true;
+
 \$CONFIG['trusted_domains'] = array('$PRIMARY_HOSTNAME');
 
 \$CONFIG['memcache.local'] = '\OC\Memcache\APCu';
@@ -287,7 +297,14 @@ include("$STORAGE_ROOT/owncloud/config.php");
 
 \$CONFIG['mail_domain'] = '$PRIMARY_HOSTNAME';
 
-\$CONFIG['user_backends'] = array(array('class' => 'OC_User_IMAP','arguments' => array('127.0.0.1', 143, null),),);
+\$CONFIG['user_backends'] = array(
+  array(
+    'class' => '\OCA\UserExternal\IMAP',
+    'arguments' => array(
+      '127.0.0.1', 143, null, null, false, false
+    ),
+  ),
+);
 
 echo "<?php\n\\\$CONFIG = ";
 var_export(\$CONFIG);
@@ -300,25 +317,25 @@ chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
 # The firstrunwizard gave Josh all sorts of problems, so disabling that.
 # user_external is what allows Nextcloud to use IMAP for login. The contacts
 # and calendar apps are the extensions we really care about here.
-hide_output sudo -u www-data php /usr/local/lib/owncloud/console.php app:disable firstrunwizard
+hide_output sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/console.php app:disable firstrunwizard
-hide_output sudo -u www-data php /usr/local/lib/owncloud/console.php app:enable user_external
+hide_output sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/console.php app:enable user_external
-hide_output sudo -u www-data php /usr/local/lib/owncloud/console.php app:enable contacts
+hide_output sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/console.php app:enable contacts
-hide_output sudo -u www-data php /usr/local/lib/owncloud/console.php app:enable calendar
+hide_output sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/console.php app:enable calendar
 
 # When upgrading, run the upgrade script again now that apps are enabled. It seems like
 # the first upgrade at the top won't work because apps may be disabled during upgrade?
 # Check for success (0=ok, 3=no upgrade needed).
-sudo -u www-data php /usr/local/lib/owncloud/occ upgrade
+sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ upgrade
 if [ \( $? -ne 0 \) -a \( $? -ne 3 \) ]; then exit 1; fi
 
 # Disable default apps that we don't support
 sudo -u www-data \
-	php /usr/local/lib/owncloud/occ app:disable photos dashboard activity \
+	php$PHP_VER /usr/local/lib/owncloud/occ app:disable photos dashboard activity \
 	| (grep -v "No such app enabled" || /bin/true)
 
 # Set PHP FPM values to support large file uploads
 # (semicolon is the comment character in this file, hashes produce deprecation warnings)
-tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
+tools/editconf.py /etc/php/$PHP_VER/fpm/php.ini -c ';' \
 	upload_max_filesize=16G \
 	post_max_size=16G \
 	output_buffering=16384 \
@@ -327,7 +344,7 @@ tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
 	short_open_tag=On
 
 # Set Nextcloud recommended opcache settings
-tools/editconf.py /etc/php/7.2/cli/conf.d/10-opcache.ini -c ';' \
+tools/editconf.py /etc/php/$PHP_VER/cli/conf.d/10-opcache.ini -c ';' \
 	opcache.enable=1 \
 	opcache.enable_cli=1 \
 	opcache.interned_strings_buffer=8 \
@@ -336,23 +353,20 @@ tools/editconf.py /etc/php/7.2/cli/conf.d/10-opcache.ini -c ';' \
 	opcache.save_comments=1 \
 	opcache.revalidate_freq=1
 
-# If apc is explicitly disabled we need to enable it
+# Migrate users_external data from <0.6.0 to version 3.0.0 (see https://github.com/nextcloud/user_external).
-if grep -q apc.enabled=0 /etc/php/7.2/mods-available/apcu.ini; then
+# This version was probably in use in Mail-in-a-Box v0.41 (February 26, 2019) and earlier.
-	tools/editconf.py /etc/php/7.2/mods-available/apcu.ini -c ';' \
+# We moved to v0.6.3 in 193763f8. Ignore errors - maybe there are duplicated users with the
-		apc.enabled=1
+# correct backend already.
-fi
+sqlite3 $STORAGE_ROOT/owncloud/owncloud.db "UPDATE oc_users_external SET backend='127.0.0.1';" || /bin/true
 
 # Set up a cron job for Nextcloud.
 cat > /etc/cron.d/mailinabox-nextcloud << EOF;
 #!/bin/bash
 # Mail-in-a-Box
-*/5 * * * *	root	sudo -u www-data php -f /usr/local/lib/owncloud/cron.php
+*/5 * * * *	root	sudo -u www-data php$PHP_VER -f /usr/local/lib/owncloud/cron.php
 EOF
 chmod +x /etc/cron.d/mailinabox-nextcloud
 
-# Remove previous hourly cronjob
-rm -f /etc/cron.hourly/mailinabox-owncloud
-
 # There's nothing much of interest that a user could do as an admin for Nextcloud,
 # and there's a lot they could mess up, so we don't make any users admins of Nextcloud.
 # But if we wanted to, we would do this:
@@ -363,4 +377,4 @@ rm -f /etc/cron.hourly/mailinabox-owncloud
 # ```
 
 # Enable PHP modules and restart PHP.
-restart_service php7.2-fpm
+restart_service php$PHP_VER-fpm

setup/preflight.sh

@@ -7,11 +7,11 @@ if [[ $EUID -ne 0 ]]; then
 	exit 1
 fi
 
-# Check that we are running on Ubuntu 18.04 LTS (or 18.04.xx).
+# Check that we are running on Ubuntu 20.04 LTS (or 20.04.xx).
-if [ "$(lsb_release -d | sed 's/.*:\s*//' | sed 's/18\.04\.[0-9]/18.04/' )" != "Ubuntu 18.04 LTS" ]; then
+if [ "$( lsb_release --id --short )" != "Ubuntu" ] || [ "$( lsb_release --release --short )" != "22.04" ]; then
-	echo "Mail-in-a-Box only supports being installed on Ubuntu 18.04, sorry. You are running:"
+	echo "Mail-in-a-Box only supports being installed on Ubuntu 22.04, sorry. You are running:"
 	echo
-	lsb_release -d | sed 's/.*:\s*//'
+	lsb_release --description --short
 	echo
 	echo "We can't write scripts that run on every possible setup, sorry."
 	exit 1

setup/start.sh

@@ -67,6 +67,10 @@ fi
 fi
 
 # Create the STORAGE_USER and STORAGE_ROOT directory if they don't already exist.
+#
+# Set the directory and all of its parent directories' permissions to world
+# readable since it holds files owned by different processes.
+#
 # If the STORAGE_ROOT is missing the mailinabox.version file that lists a
 # migration (schema) number for the files stored there, assume this is a fresh
 # installation to that directory and write the file to contain the current
@@ -77,6 +81,8 @@ fi
 if [ ! -d $STORAGE_ROOT ]; then
 	mkdir -p $STORAGE_ROOT
 fi
+f=$STORAGE_ROOT
+while [[ $f != / ]]; do chmod a+rx "$f"; f=$(dirname "$f"); done;
 if [ ! -f $STORAGE_ROOT/mailinabox.version ]; then
 	setup/migrate.py --current > $STORAGE_ROOT/mailinabox.version
 	chown $STORAGE_USER.$STORAGE_USER $STORAGE_ROOT/mailinabox.version

setup/system.sh

@@ -75,6 +75,13 @@ then
 	fi
 fi
 
+# ### Set log retention policy.
+
+# Set the systemd journal log retention from infinite to 10 days,
+# since over time the logs take up a large amount of space.
+# (See https://discourse.mailinabox.email/t/journalctl-reclaim-space-on-small-mailinabox/6728/11.)
+tools/editconf.py /etc/systemd/journald.conf MaxRetentionSec=10day
+
 # ### Add PPAs.
 
 # We install some non-standard Ubuntu packages maintained by other
@@ -90,12 +97,13 @@ fi
 # come from there and minimal Ubuntu installs may have it turned off.
 hide_output add-apt-repository -y universe
 
-# Install the certbot PPA.
-hide_output add-apt-repository -y ppa:certbot/certbot
-
 # Install the duplicity PPA.
 hide_output add-apt-repository -y ppa:duplicity-team/duplicity-release-git
 
+# Stock PHP is now 8.1, but we're transitioning through 8.0 because
+# of Nextcloud.
+hide_output add-apt-repository --y ppa:ondrej/php
+
 # ### Update Packages
 
 # Update system packages to make sure we have the latest upstream versions
@@ -116,9 +124,6 @@ apt_get_quiet autoremove
 
 # Install basic utilities.
 #
-# * haveged: Provides extra entropy to /dev/random so it doesn't stall
-#	         when generating random numbers for private keys (e.g. during
-#	         ldns-keygen).
 # * unattended-upgrades: Apt tool to install security updates automatically.
 # * cron: Runs background processes periodically.
 # * ntp: keeps the system time correct
@@ -132,8 +137,8 @@ apt_get_quiet autoremove
 
 echo Installing system packages...
 apt_install python3 python3-dev python3-pip python3-setuptools \
-	netcat-openbsd wget curl git sudo coreutils bc \
+	netcat-openbsd wget curl git sudo coreutils bc file \
-	haveged pollinate openssh-client unzip \
+	pollinate openssh-client unzip \
 	unattended-upgrades cron ntp fail2ban rsyslog
 
 # ### Suppress Upgrade Prompts
@@ -324,7 +329,7 @@ fi #NODOC
 #  	If more queries than specified are sent, bind9 returns SERVFAIL. After flushing the cache during system checks,
 #	we ran into the limit thus we are increasing it from 75 (default value) to 100.
 apt_install bind9
-tools/editconf.py /etc/default/bind9 \
+tools/editconf.py /etc/default/named \
 	"OPTIONS=\"-u bind -4\""
 if ! grep -q "listen-on " /etc/bind/named.conf.options; then
 	# Add a listen-on directive if it doesn't exist inside the options block.
@@ -356,6 +361,7 @@ systemctl restart systemd-resolved
 rm -f /etc/fail2ban/jail.local # we used to use this file but don't anymore
 rm -f /etc/fail2ban/jail.d/defaults-debian.conf # removes default config so we can manage all of fail2ban rules in one config
 cat conf/fail2ban/jails.conf \
+    | sed "s/PUBLIC_IPV6/$PUBLIC_IPV6/g" \
 	| sed "s/PUBLIC_IP/$PUBLIC_IP/g" \
 	| sed "s#STORAGE_ROOT#$STORAGE_ROOT#" \
 	> /etc/fail2ban/jail.d/mailinabox.conf

setup/web.sh

@@ -19,7 +19,7 @@ fi
 
 echo "Installing Nginx (web server)..."
 
-apt_install nginx php-cli php-fpm idn2
+apt_install nginx php${PHP_VER}-cli php${PHP_VER}-fpm idn2
 
 rm -f /etc/nginx/sites-enabled/default
 
@@ -46,15 +46,15 @@ tools/editconf.py /etc/nginx/nginx.conf -s \
 	ssl_protocols="TLSv1.2 TLSv1.3;"
 
 # Tell PHP not to expose its version number in the X-Powered-By header.
-tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
+tools/editconf.py /etc/php/$PHP_VER/fpm/php.ini -c ';' \
 	expose_php=Off
 
 # Set PHPs default charset to UTF-8, since we use it. See #367.
-tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
+tools/editconf.py /etc/php/$PHP_VER/fpm/php.ini -c ';' \
         default_charset="UTF-8"
 
 # Configure the path environment for php-fpm
-tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+tools/editconf.py /etc/php/$PHP_VER/fpm/pool.d/www.conf -c ';' \
 	env[PATH]=/usr/local/bin:/usr/bin:/bin \
 
 # Configure php-fpm based on the amount of memory the machine has
@@ -64,7 +64,7 @@ tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
 TOTAL_PHYSICAL_MEM=$(head -n 1 /proc/meminfo | awk '{print $2}' || /bin/true)
 if [ $TOTAL_PHYSICAL_MEM -lt 1000000 ]
 then
-        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+        tools/editconf.py /etc/php/$PHP_VER/fpm/pool.d/www.conf -c ';' \
                 pm=ondemand \
                 pm.max_children=8 \
                 pm.start_servers=2 \
@@ -72,7 +72,7 @@ then
                 pm.max_spare_servers=3
 elif [ $TOTAL_PHYSICAL_MEM -lt 2000000 ]
 then
-        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+        tools/editconf.py /etc/php/$PHP_VER/fpm/pool.d/www.conf -c ';' \
                 pm=ondemand \
                 pm.max_children=16 \
                 pm.start_servers=4 \
@@ -80,14 +80,14 @@ then
                 pm.max_spare_servers=6
 elif [ $TOTAL_PHYSICAL_MEM -lt 3000000 ]
 then
-        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+        tools/editconf.py /etc/php/$PHP_VER/fpm/pool.d/www.conf -c ';' \
                 pm=dynamic \
                 pm.max_children=60 \
                 pm.start_servers=6 \
                 pm.min_spare_servers=3 \
                 pm.max_spare_servers=9
 else
-        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+        tools/editconf.py /etc/php/$PHP_VER/fpm/pool.d/www.conf -c ';' \
                 pm=dynamic \
                 pm.max_children=120 \
                 pm.start_servers=12 \
@@ -147,7 +147,7 @@ chown -R $STORAGE_USER $STORAGE_ROOT/www
 
 # Start services.
 restart_service nginx
-restart_service php7.2-fpm
+restart_service php$PHP_VER-fpm
 
 # Open ports.
 ufw_allow http

setup/webmail.sh

@@ -22,19 +22,25 @@ source /etc/mailinabox.conf # load global vars
 echo "Installing Roundcube (webmail)..."
 apt_install \
 	dbconfig-common \
-	php-cli php-sqlite3 php-intl php-json php-common php-curl php-ldap \
+	php${PHP_VER}-cli php${PHP_VER}-sqlite3 php${PHP_VER}-intl php${PHP_VER}-common php${PHP_VER}-curl php${PHP_VER}-imap \
-	php-gd php-pspell tinymce libjs-jquery libjs-jquery-mousewheel libmagic1 php-mbstring
+	php${PHP_VER}-gd php${PHP_VER}-pspell php${PHP_VER}-mbstring libjs-jquery libjs-jquery-mousewheel libmagic1
 
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of plugins to track
 # whether we have the latest version of everything.
-
+# For the latest versions, see:
-VERSION=1.4.11
+#   https://github.com/roundcube/roundcubemail/releases
-HASH=3877f0e70f29e7d0612155632e48c3db1e626be3
+#   https://github.com/mfreiholz/persistent_login/commits/master
-PERSISTENT_LOGIN_VERSION=6b3fc450cae23ccb2f393d0ef67aa319e877e435 # version 5.2.0
+#   https://github.com/stremlau/html5_notifier/commits/master
+#   https://github.com/mstilkerich/rcmcarddav/releases
+# The easiest way to get the package hashes is to run this script and get the hash from
+# the error message.
+VERSION=1.6.0
+HASH=fd84b4fac74419bb73e7a3bcae1978d5589c52de
+PERSISTENT_LOGIN_VERSION=bde7b6840c7d91de627ea14e81cf4133cbb3c07a # version 5.2
 HTML5_NOTIFIER_VERSION=68d9ca194212e15b3c7225eb6085dbcf02fd13d7 # version 0.6.4+
-CARDDAV_VERSION=3.0.3
+CARDDAV_VERSION=4.4.3
-CARDDAV_HASH=d1e3b0d851ffa2c6bd42bf0c04f70d0e1d0d78f8
+CARDDAV_HASH=74f8ba7aee33e78beb9de07f7f44b81f6071b644
 
 UPDATE_KEY=$VERSION:$PERSISTENT_LOGIN_VERSION:$HTML5_NOTIFIER_VERSION:$CARDDAV_VERSION
 
@@ -77,13 +83,13 @@ if [ $needs_update == 1 ]; then
 
 	# download and verify the full release of the carddav plugin
 	wget_verify \
-		https://github.com/blind-coder/rcmcarddav/releases/download/v${CARDDAV_VERSION}/carddav-${CARDDAV_VERSION}.zip \
+		https://github.com/mstilkerich/rcmcarddav/releases/download/v${CARDDAV_VERSION}/carddav-v${CARDDAV_VERSION}.tar.gz \
 		$CARDDAV_HASH \
-		/tmp/carddav.zip
+		/tmp/carddav.tar.gz
 
 	# unzip and cleanup
-	unzip -q /tmp/carddav.zip -d ${RCM_PLUGIN_DIR}
+	tar -C ${RCM_PLUGIN_DIR} -zxf /tmp/carddav.tar.gz
-	rm -f /tmp/carddav.zip
+	rm -f /tmp/carddav.tar.gz
 
 	# record the version we've installed
 	echo $UPDATE_KEY > ${RCM_DIR}/version
@@ -109,8 +115,7 @@ cat > $RCM_CONFIG <<EOF;
 \$config['log_dir'] = '/var/log/roundcubemail/';
 \$config['temp_dir'] = '/var/tmp/roundcubemail/';
 \$config['db_dsnw'] = 'sqlite:///$STORAGE_ROOT/mail/roundcube/roundcube.sqlite?mode=0640';
-\$config['default_host'] = 'ssl://localhost';
+\$config['imap_host'] = 'ssl://localhost:993';
-\$config['default_port'] = 993;
 \$config['imap_conn_options'] = array(
   'ssl'         => array(
      'verify_peer'  => false,
@@ -118,7 +123,7 @@ cat > $RCM_CONFIG <<EOF;
    ),
  );
 \$config['imap_timeout'] = 15;
-\$config['smtp_server'] = 'tls://127.0.0.1';
+\$config['smtp_host'] = 'tls://127.0.0.1';
 \$config['smtp_conn_options'] = array(
   'ssl'         => array(
      'verify_peer'  => false,
@@ -132,8 +137,13 @@ cat > $RCM_CONFIG <<EOF;
 \$config['plugins'] = array('html5_notifier', 'archive', 'zipdownload', 'password', 'managesieve', 'jqueryui', 'persistent_login', 'carddav');
 \$config['skin'] = 'elastic';
 \$config['login_autocomplete'] = 2;
+\$config['login_username_filter'] = 'email';
 \$config['password_charset'] = 'UTF-8';
 \$config['junk_mbox'] = 'Spam';
+/* ensure roudcube session id's aren't leaked to other parts of the server */
+\$config['session_path'] = '/mail/';
+/* prevent CSRF, requires php 7.3+ */
+\$config['session_samesite'] = 'Strict';
 ?>
 EOF
 
@@ -147,7 +157,7 @@ cat > ${RCM_PLUGIN_DIR}/carddav/config.inc.php <<EOF;
 	 'name'         =>  'ownCloud',
 	 'username'     =>  '%u', // login username
 	 'password'     =>  '%p', // login password
-	 'url'          =>  'https://${PRIMARY_HOSTNAME}/cloud/remote.php/carddav/addressbooks/%u/contacts',
+	 'url'          =>  'https://${PRIMARY_HOSTNAME}/cloud/remote.php/dav/addressbooks/users/%u/contacts/',
 	 'active'       =>  true,
 	 'readonly'     =>  false,
 	 'refresh_time' => '02:00:00',
@@ -195,10 +205,10 @@ chown -f -R root.www-data ${RCM_PLUGIN_DIR}/carddav
 chmod -R 774 ${RCM_PLUGIN_DIR}/carddav
 
 # Run Roundcube database migration script (database is created if it does not exist)
-${RCM_DIR}/bin/updatedb.sh --dir ${RCM_DIR}/SQL --package roundcube
+php$PHP_VER ${RCM_DIR}/bin/updatedb.sh --dir ${RCM_DIR}/SQL --package roundcube
 chown www-data:www-data $STORAGE_ROOT/mail/roundcube/roundcube.sqlite
 chmod 664 $STORAGE_ROOT/mail/roundcube/roundcube.sqlite
 
 # Enable PHP modules.
-phpenmod -v php mcrypt imap
+phpenmod -v $PHP_VER imap
-restart_service php7.2-fpm
+restart_service php$PHP_VER-fpm

setup/zpush.sh

@@ -17,9 +17,9 @@ source /etc/mailinabox.conf # load global vars
 
 echo "Installing Z-Push (Exchange/ActiveSync server)..."
 apt_install \
-	php-soap php-imap libawl-php php-xsl
+       php${PHP_VER}-soap php${PHP_VER}-imap libawl-php php$PHP_VER-xml
 
-phpenmod -v php imap
+phpenmod -v $PHP_VER imap
 
 # Copy Z-Push into place.
 VERSION=2.6.2
@@ -42,8 +42,6 @@ if [ $needs_update == 1 ]; then
 	rm -rf /tmp/z-push.zip /tmp/z-push
 
 	rm -f /usr/sbin/z-push-{admin,top}
-	ln -s /usr/local/lib/z-push/z-push-admin.php /usr/sbin/z-push-admin
-	ln -s /usr/local/lib/z-push/z-push-top.php /usr/sbin/z-push-top
 	echo $VERSION > /usr/local/lib/z-push/version
 fi
 
@@ -102,8 +100,8 @@ EOF
 
 # Restart service.
 
-restart_service php7.2-fpm
+restart_service php$PHP_VER-fpm
 
 # Fix states after upgrade
 
-hide_output z-push-admin -a fixstates
+hide_output php$PHP_VER /usr/local/lib/z-push/z-push-admin.php -a fixstates

tests/fail2ban.py

@@ -232,7 +232,7 @@ if __name__ == "__main__":
 	run_test(managesieve_test, [], 20, 30, 4)
 
 	# Mail-in-a-Box control panel
-	run_test(http_test, ["/admin/me", 200], 20, 30, 1)
+	run_test(http_test, ["/admin/login", 200], 20, 30, 1)
 
 	# Munin via the Mail-in-a-Box control panel
 	run_test(http_test, ["/admin/munin/", 401], 20, 30, 1)

tests/test_dns.py

@@ -30,7 +30,7 @@ def test(server, description):
 		(hostname, "TXT", "\"v=spf1 mx -all\""),
 		("mail._domainkey." + hostname, "TXT", "\"v=DKIM1; k=rsa; s=email; \" \"p=__KEY__\""),
 		#("_adsp._domainkey." + hostname, "TXT", "\"dkim=all\""),
-		("_dmarc." + hostname, "TXT", "\"v=DMARC1; p=quarantine\""),
+		("_dmarc." + hostname, "TXT", "\"v=DMARC1; p=quarantine;\""),
 	]
 	return test2(tests, server, description)
 
@@ -48,7 +48,7 @@ def test2(tests, server, description):
 	for qname, rtype, expected_answer in tests:
 		# do the query and format the result as a string
 		try:
-			response = dns.resolver.query(qname, rtype)
+			response = dns.resolver.resolve(qname, rtype)
 		except dns.resolver.NoNameservers:
 			# host did not have an answer for this query
 			print("Could not connect to %s for DNS query." % server)

tests/test_mail.py

@@ -48,7 +48,7 @@ server = smtplib.SMTP_SSL(host)
 ipaddr = socket.gethostbyname(host) # IPv4 only!
 reverse_ip = dns.reversename.from_address(ipaddr) # e.g. "1.0.0.127.in-addr.arpa."
 try:
-	reverse_dns = dns.resolver.query(reverse_ip, 'PTR')[0].target.to_text(omit_final_dot=True) # => hostname
+	reverse_dns = dns.resolver.resolve(reverse_ip, 'PTR')[0].target.to_text(omit_final_dot=True) # => hostname
 except dns.resolver.NXDOMAIN:
 	print("Reverse DNS lookup failed for %s. SMTP EHLO name check skipped." % ipaddr)
 	reverse_dns = None

tools/editconf.py

@@ -14,6 +14,10 @@
 #
 # NAME VALUE
 #
+# If the -e option is given and VALUE is empty, the setting is removed
+# from the configuration file if it is set (i.e. existing occurrences
+# are commented out and no new setting is added).
+#
 # If the -c option is given, then the supplied character becomes the comment character
 #
 # If the -w option is given, then setting lines continue onto following
@@ -35,6 +39,7 @@ settings = sys.argv[2:]
 
 delimiter = "="
 delimiter_re = r"\s*=\s*"
+erase_setting = False
 comment_char = "#"
 folded_lines = False
 testing = False
@@ -44,6 +49,9 @@ while settings[0][0] == "-" and settings[0] != "--":
 		# Space is the delimiter
 		delimiter = " "
 		delimiter_re = r"\s+"
+	elif opt == "-e":
+		# Erase settings that have empty values.
+		erase_setting = True
 	elif opt == "-w":
 		# Line folding is possible in this file.
 		folded_lines = True
@@ -81,7 +89,7 @@ while len(input_lines) > 0:
 
 	# See if this line is for any settings passed on the command line.
 	for i in range(len(settings)):
-		# Check that this line contain this setting from the command-line arguments.
+		# Check if this line contain this setting from the command-line arguments.
 		name, val = settings[i].split("=", 1)
 		m = re.match(
 			   "(\s*)"
@@ -91,8 +99,10 @@ while len(input_lines) > 0:
 		if not m: continue
 		indent, is_comment, existing_val = m.groups()
 
-		# If this is already the setting, do nothing.
+		# If this is already the setting, keep it in the file, except:
-		if is_comment is None and existing_val == val:
+		# * If we've already seen it before, then remove this duplicate line.
+		# * If val is empty and erase_setting is on, then comment it out.
+		if is_comment is None and existing_val == val and not (not val and erase_setting):
 			# It may be that we've already inserted this setting higher
 			# in the file so check for that first.
 			if i in found: break
@@ -107,8 +117,9 @@ while len(input_lines) > 0:
 			# the line is already commented, pass it through
 			buf += line
 		
-		# if this option oddly appears more than once, don't add the setting again
+		# if this option already is set don't add the setting again,
-		if i in found:
+		# or if we're clearing the setting with -e, don't add it
+		if (i in found) or (not val and erase_setting):
 			break
 		
 		# add the new setting
@@ -122,11 +133,13 @@ while len(input_lines) > 0:
 		# If did not match any setting names, pass this line through.
 		buf += line
 		
-# Put any settings we didn't see at the end of the file.
+# Put any settings we didn't see at the end of the file,
+# except settings being cleared.
 for i in range(len(settings)):
 	if i not in found:
 		name, val = settings[i].split("=", 1)
-		buf += name + delimiter + val + "\n"
+		if not (not val and erase_setting):
+			buf += name + delimiter + val + "\n"
 
 if not testing:
 	# Write out the new file.

tools/owncloud-restore.sh

@@ -26,7 +26,7 @@ if [ ! -f $1/config.php ]; then
 fi
 
 echo "Restoring backup from $1"
-service php7.2-fpm stop
+service php8.0-fpm stop
 
 # remove the current ownCloud/Nextcloud installation
 rm -rf /usr/local/lib/owncloud/
@@ -43,7 +43,7 @@ ln -sf $STORAGE_ROOT/owncloud/config.php /usr/local/lib/owncloud/config/config.p
 chown -f -R www-data.www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
 chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
 
-sudo -u www-data php /usr/local/lib/owncloud/occ maintenance:mode --off
+sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ maintenance:mode --off
 
-service php7.2-fpm start
+service php8.0-fpm start
 echo "Done"

tools/owncloud-unlockadmin.sh

@@ -20,4 +20,4 @@ echo
 echo Press enter to continue.
 read
 
-sudo -u www-data php /usr/local/lib/owncloud/occ group:adduser admin $ADMIN && echo Done.
+sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ group:adduser admin $ADMIN && echo Done.