Version 65

Updating to latest release, bugfixes no new features.

CHANGELOG.md

@@ -1,6 +1,84 @@
 CHANGELOG
 =========
 
+Version 65 (October 27, 2023)
+-----------------------------
+
+* Roundcube updated to 1.6.4 fixing a security vulnerability.
+* zpush.sh updated to version 2.7.1.
+* Fixed a typo in the control panel.
+
+Version 64 (September 2, 2023)
+------------------------------
+
+* Fixed broken installation when upgrading from Mail-in-a-Box version 56 (Nextcloud 22) and earlier because of an upstream packaging issue.
+* Fixed backups to work with the latest duplicity package which was not backwards compatible.
+* Fixed setting B2 as a backup target with a slash in the application key.
+* Turned off OpenDMARC diagnostic reports sent in response to incoming mail.
+* Fixed some crashes when using an unrelased version of Mail-in-a-Box.
+* Added z-push administration scripts.
+
+Version 63 (July 27, 2023)
+--------------------------
+
+* Nextcloud updated to 25.0.7.
+
+Version 62 (May 20, 2023)
+-------------------------
+
+Package updates:
+
+* Nextcloud updated to 23.0.12 (and its apps also updated).
+* Roundcube updated to 1.6.1.
+* Z-Push to 2.7.0, which has compatibility for Ubuntu 22.04, so it works again.
+
+Mail:
+
+* Roundcube's password change page is now working again.
+
+Control panel:
+
+* Allow setting the backup location's S3 region name for non-AWS S3-compatible backup hosts.
+* Control panel pages can be opened in a new tab/window and bookmarked and browser history navigation now works.
+* Add a Copy button to put the rsync backup public key on clipboard.
+* Allow secondary DNS xfr: items added in the control panel to be hostnames too.
+* Fixed issue where sshkeygen fails when IPv6 is disabled.
+* Fixed issue opening munin reports.
+* Fixed report formatting in status emails sent to the administrator.
+
+Version 61.1 (January 28, 2023)
+-------------------------------
+
+* Fixed rsync backups not working with the default port.
+* Reverted "Improve error messages in the management tools when external command-line tools are run." because of the possibility of user secrets being included in error messages.
+* Fix for TLS certificate SHA fingerprint not being displayed during setup.
+
+Version 61 (January 21, 2023)
+-----------------------------
+
+System:
+
+* fail2ban didn't start after setup.
+
+Mail:
+
+* Disable Roundcube password plugin since it was corrupting the user database.
+
+Control panel:
+
+* Fix changing existing backup settings when the rsync type is used.
+* Allow setting a custom port for rsync backups.
+* Fixes to DNS lookups during status checks when there are timeouts, enforce timeouts better.
+* A new check is added to ensure fail2ban is running.
+* Fixed a color.
+* Improve error messages in the management tools when external command-line tools are run.
+
+Version 60.1 (October 30, 2022)
+-------------------------------
+
+* A setup issue where the DNS server nsd isn't running at the end of setup is (hopefully) fixed.
+* Nextcloud is updated to 23.0.10 (contacts to 4.2.2, calendar to 3.5.1).
+
 Version 60 (October 11, 2022)
 -----------------------------
 
@@ -17,7 +95,7 @@ No major features of Mail-in-a-Box have changed in this release, although some m
 With the newer version of Ubuntu the following software packages we use are updated:
 
 * dovecot is upgraded to 2.3.16, postfix to 3.6.4, opendmark to 1.4 (which adds ARC-Authentication-Results headers), and spampd to 2.53 (alleviating a mail delivery rate limiting bug).
-* Nextcloud is upgraded to 23.0.4.
+* Nextcloud is upgraded to 23.0.4 (contacts to 4.2.0, calendar to 3.5.0).
 * Roundcube is upgraded to 1.6.0.
 * certbot is upgraded to 1.21 (via the Ubuntu repository instead of a PPA).
 * fail2ban is upgraded to 0.11.2.

README.md

@@ -60,7 +60,7 @@ Clone this repository and checkout the tag corresponding to the most recent rele
 
 	$ git clone https://github.com/mail-in-a-box/mailinabox
 	$ cd mailinabox
-	$ git checkout v60
+	$ git checkout v65
 
 Begin the installation.
 

conf/nginx-primaryonly.conf

@@ -73,4 +73,9 @@
 	rewrite ^/.well-known/carddav /cloud/remote.php/carddav/ redirect;
 	rewrite ^/.well-known/caldav /cloud/remote.php/caldav/ redirect;
 
+	# This addresses those service discovery issues mentioned in:
+	# https://docs.nextcloud.com/server/23/admin_manual/issues/general_troubleshooting.html#service-discovery
+	rewrite ^/.well-known/webfinger /cloud/index.php/.well-known/webfinger redirect;
+	rewrite ^/.well-known/nodeinfo /cloud/index.php/.well-known/nodeinfo redirect;
+
 	# ADDITIONAL DIRECTIVES HERE

management/backup.py

@@ -57,10 +57,11 @@ def backup_status(env):
 		"/usr/bin/duplicity",
 		"collection-status",
 		"--archive-dir", backup_cache_dir,
-		"--gpg-options", "--cipher-algo=AES256",
+		"--gpg-options", "'--cipher-algo=AES256'",
 		"--log-fd", "1",
-		get_duplicity_target_url(config),
+		] + get_duplicity_additional_args(env) + [
-		] + get_duplicity_additional_args(env),
+		get_duplicity_target_url(config)
+		],
 		get_duplicity_env_vars(env),
 		trap=True)
 	if code != 0:
@@ -202,7 +203,9 @@ def get_duplicity_target_url(config):
 		# the target URL must be the bucket name. The hostname is passed
 		# via get_duplicity_additional_args. Move the first part of the
 		# path (the bucket name) into the hostname URL component, and leave
-		# the rest for the path.
+		# the rest for the path. (The S3 region name is also stored in the
+		# hostname part of the URL, in the username portion, which we also
+		# have to drop here).
 		target[1], target[2] = target[2].lstrip('/').split('/', 1)
 
 		target = urlunsplit(target)
@@ -213,16 +216,32 @@ def get_duplicity_additional_args(env):
 	config = get_backup_config(env)
 
 	if get_target_type(config) == 'rsync':
+		# Extract a port number for the ssh transport.  Duplicity accepts the
+		# optional port number syntax in the target, but it doesn't appear to act
+		# on it, so we set the ssh port explicitly via the duplicity options.
+		from urllib.parse import urlsplit
+		try:
+			port = urlsplit(config["target"]).port
+		except ValueError:
+			port = 22
+		if port is None:
+			port = 22
+						
 		return [
-			"--ssh-options= -i /root/.ssh/id_rsa_miab",
+			f"--ssh-options='-i /root/.ssh/id_rsa_miab -p {port}'",
-			"--rsync-options= -e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab\"",
+			f"--rsync-options='-e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p {port} -i /root/.ssh/id_rsa_miab\"'",
 		]
 	elif get_target_type(config) == 's3':
 		# See note about hostname in get_duplicity_target_url.
+		# The region name, which is required by some non-AWS endpoints,
+		# is saved inside the username portion of the URL.
 		from urllib.parse import urlsplit, urlunsplit
 		target = urlsplit(config["target"])
-		endpoint_url = urlunsplit(("https", target.netloc, '', '', ''))
+		endpoint_url = urlunsplit(("https", target.hostname, '', '', ''))
-		return ["--s3-endpoint-url",  endpoint_url]
+		args = ["--s3-endpoint-url", endpoint_url]
+		if target.username: # region name is stuffed here
+			args += ["--s3-region-name", target.username]
+		return args
 
 	return []
 
@@ -303,11 +322,12 @@ def perform_backup(full_backup):
 			"--archive-dir", backup_cache_dir,
 			"--exclude", backup_root,
 			"--volsize", "250",
-			"--gpg-options", "--cipher-algo=AES256",
+			"--gpg-options", "'--cipher-algo=AES256'",
+			"--allow-source-mismatch"
+			] + get_duplicity_additional_args(env) + [
 			env["STORAGE_ROOT"],
 			get_duplicity_target_url(config),
-			"--allow-source-mismatch"
+			],
-			] + get_duplicity_additional_args(env),
 			get_duplicity_env_vars(env))
 	finally:
 		# Start services again.
@@ -325,8 +345,9 @@ def perform_backup(full_backup):
 		"--verbosity", "error",
 		"--archive-dir", backup_cache_dir,
 		"--force",
+		] + get_duplicity_additional_args(env) + [
 		get_duplicity_target_url(config)
-		] + get_duplicity_additional_args(env),
+		],
 		get_duplicity_env_vars(env))
 
 	# From duplicity's manual:
@@ -340,8 +361,9 @@ def perform_backup(full_backup):
 		"--verbosity", "error",
 		"--archive-dir", backup_cache_dir,
 		"--force",
+		] + get_duplicity_additional_args(env) + [
 		get_duplicity_target_url(config)
-		] + get_duplicity_additional_args(env),
+		],
 		get_duplicity_env_vars(env))
 
 	# Change ownership of backups to the user-data user, so that the after-bcakup
@@ -378,9 +400,10 @@ def run_duplicity_verification():
 		"--compare-data",
 		"--archive-dir", backup_cache_dir,
 		"--exclude", backup_root,
+		] + get_duplicity_additional_args(env) + [
 		get_duplicity_target_url(config),
 		env["STORAGE_ROOT"],
-	] + get_duplicity_additional_args(env), get_duplicity_env_vars(env))
+	], get_duplicity_env_vars(env))
 
 def run_duplicity_restore(args):
 	env = load_environment()
@@ -390,9 +413,23 @@ def run_duplicity_restore(args):
 		"/usr/bin/duplicity",
 		"restore",
 		"--archive-dir", backup_cache_dir,
-		get_duplicity_target_url(config),
+		] + get_duplicity_additional_args(env) + [
-		] + get_duplicity_additional_args(env) + args,
+		get_duplicity_target_url(config)
-	get_duplicity_env_vars(env))
+		] + args,
+		get_duplicity_env_vars(env))
+
+def print_duplicity_command():
+	import shlex
+	env = load_environment()
+	config = get_backup_config(env)
+	backup_cache_dir = os.path.join(env["STORAGE_ROOT"], 'backup', 'cache')
+	for k, v in get_duplicity_env_vars(env).items():
+		print(f"export {k}={shlex.quote(v)}")
+	print("duplicity", "{command}", shlex.join([
+		"--archive-dir", backup_cache_dir,
+		] + get_duplicity_additional_args(env) + [
+		get_duplicity_target_url(config)
+		]))
 
 def list_target_files(config):
 	import urllib.parse
@@ -408,6 +445,16 @@ def list_target_files(config):
 		rsync_fn_size_re = re.compile(r'.*    ([^ ]*) [^ ]* [^ ]* (.*)')
 		rsync_target = '{host}:{path}'
 
+		# Strip off any trailing port specifier because it's not valid in rsync's
+		# DEST syntax.  Explicitly set the port number for the ssh transport.
+		user_host, *_ = target.netloc.rsplit(':', 1)
+		try:
+			port = target.port
+		except ValueError:
+			 port = 22
+		if port is None:
+			port = 22
+
 		target_path = target.path
 		if not target_path.endswith('/'):
 			target_path = target_path + '/'
@@ -416,11 +463,11 @@ def list_target_files(config):
 
 		rsync_command = [ 'rsync',
 					'-e',
-					'/usr/bin/ssh -i /root/.ssh/id_rsa_miab -oStrictHostKeyChecking=no -oBatchMode=yes',
+					f'/usr/bin/ssh -i /root/.ssh/id_rsa_miab -oStrictHostKeyChecking=no -oBatchMode=yes -p {port}',
 					'--list-only',
 					'-r',
 					rsync_target.format(
-						host=target.netloc,
+						host=user_host,
 						path=target_path)
 				]
 
@@ -481,7 +528,7 @@ def list_target_files(config):
 		
 		# Extract information from target
 		b2_application_keyid = target.netloc[:target.netloc.index(':')]
-		b2_application_key = target.netloc[target.netloc.index(':')+1:target.netloc.index('@')]
+		b2_application_key = urllib.parse.unquote(target.netloc[target.netloc.index(':')+1:target.netloc.index('@')])
 		b2_bucket = target.netloc[target.netloc.index('@')+1:]
 
 		try:
@@ -531,7 +578,8 @@ def get_backup_config(env, for_save=False, for_ui=False):
 
 	# Merge in anything written to custom.yaml.
 	try:
-		custom_config = rtyaml.load(open(os.path.join(backup_root, 'custom.yaml')))
+		with open(os.path.join(backup_root, 'custom.yaml'), 'r') as f:
+			custom_config = rtyaml.load(f)
 		if not isinstance(custom_config, dict): raise ValueError() # caught below
 		config.update(custom_config)
 	except:
@@ -556,7 +604,8 @@ def get_backup_config(env, for_save=False, for_ui=False):
 		config["target"] = "file://" + config["file_target_directory"]
 	ssh_pub_key = os.path.join('/root', '.ssh', 'id_rsa_miab.pub')
 	if os.path.exists(ssh_pub_key):
-		config["ssh_pub_key"] = open(ssh_pub_key, 'r').read()
+		with open(ssh_pub_key, 'r') as f:
+			config["ssh_pub_key"] = f.read()
 
 	return config
 
@@ -588,6 +637,9 @@ if __name__ == "__main__":
 		# to duplicity. The restore path should be specified.
 		run_duplicity_restore(sys.argv[2:])
 
+	elif sys.argv[-1] == "--duplicity-command":
+		print_duplicity_command()
+
 	else:
 		# Perform a backup. Add --full to force a full backup rather than
 		# possibly performing an incremental backup.

management/cli.py

@@ -47,7 +47,8 @@ def read_password():
     return first
 
 def setup_key_auth(mgmt_uri):
-	key = open('/var/lib/mailinabox/api.key').read().strip()
+	with open('/var/lib/mailinabox/api.key', 'r') as f:
+		key = f.read().strip()
 
 	auth_handler = urllib.request.HTTPBasicAuthHandler()
 	auth_handler.add_password(

management/daemon.py

@@ -709,7 +709,7 @@ def munin_cgi(filename):
 	support infrastructure like spawn-fcgi.
 	"""
 
-	COMMAND = 'su - munin --preserve-environment --shell=/bin/bash -c /usr/lib/munin/cgi/munin-cgi-graph'
+	COMMAND = 'su munin --preserve-environment --shell=/bin/bash -c /usr/lib/munin/cgi/munin-cgi-graph'
 	# su changes user, we use the munin user here
 	# --preserve-environment retains the environment, which is where Popen's `env` data is
 	# --shell=/bin/bash ensures the shell used is bash

management/dns_update.py

@@ -98,7 +98,16 @@ def do_dns_update(env, force=False):
 
 	# Tell nsd to reload changed zone files.
 	if len(updated_domains) > 0:
-		shell('check_call', ["/usr/sbin/nsd-control", "reload"])
+		# 'reconfig' is needed if there are added or removed zones, but
+		# it may not reload existing zones, so we call 'reload' too. If
+		# nsd isn't running, nsd-control fails, so in that case revert
+		# to restarting nsd to make sure it is running. Restarting nsd
+		# should also refresh everything.
+		try:
+			shell('check_call', ["/usr/sbin/nsd-control", "reconfig"])
+			shell('check_call', ["/usr/sbin/nsd-control", "reload"])
+		except:
+			shell('check_call', ["/usr/sbin/service", "nsd", "restart"])
 
 	# Write the OpenDKIM configuration tables for all of the mail domains.
 	from mailconfig import get_mail_domains
@@ -456,7 +465,7 @@ def build_sshfp_records():
 					pass
 				break
 
-	keys = shell("check_output", ["ssh-keyscan", "-t", "rsa,dsa,ecdsa,ed25519", "-p", str(port), "localhost"])
+	keys = shell("check_output", ["ssh-keyscan", "-4", "-t", "rsa,dsa,ecdsa,ed25519", "-p", str(port), "localhost"])
 	keys = sorted(keys.split("\n"))
 
 	for key in keys:
@@ -806,7 +815,8 @@ def write_opendkim_tables(domains, env):
 
 def get_custom_dns_config(env, only_real_records=False):
 	try:
-		custom_dns = rtyaml.load(open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml')))
+		with open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml'), 'r') as f:
+			custom_dns = rtyaml.load(f)
 		if not isinstance(custom_dns, dict): raise ValueError() # caught below
 	except:
 		return [ ]
@@ -983,6 +993,7 @@ def set_custom_dns_record(qname, rtype, value, action, env):
 def get_secondary_dns(custom_dns, mode=None):
 	resolver = dns.resolver.get_default_resolver()
 	resolver.timeout = 10
+	resolver.lifetime = 10
 
 	values = []
 	for qname, rtype, value in custom_dns:
@@ -994,25 +1005,33 @@ def get_secondary_dns(custom_dns, mode=None):
 				values.append(hostname)
 				continue
 
-			# This is a hostname. Before including in zone xfr lines,
+			# If the entry starts with "xfr:" only include it in the zone transfer settings.
-			# resolve to an IP address. Otherwise just return the hostname.
+			if hostname.startswith("xfr:"):
-			# It may not resolve to IPv6, so don't throw an exception if it
+				if mode != "xfr": continue
-			# doesn't.
+				hostname = hostname[4:]
-			if not hostname.startswith("xfr:"):
-				if mode == "xfr":
-					response = dns.resolver.resolve(hostname+'.', "A", raise_on_no_answer=False)
-					values.extend(map(str, response))
-					response = dns.resolver.resolve(hostname+'.', "AAAA", raise_on_no_answer=False)
-					values.extend(map(str, response))
-					continue
-				values.append(hostname)
 
-			# This is a zone-xfer-only IP address. Do not return if
+			# If is a hostname, before including in zone xfr lines,
-			# we're querying for NS record hostnames. Only return if
+			# resolve to an IP address.
-			# we're querying for zone xfer IP addresses - return the
+			# It may not resolve to IPv6, so don't throw an exception if it
-			# IP address.
+			# doesn't. Skip the entry if there is a DNS error.
-			elif mode == "xfr":
+			if mode == "xfr":
-				values.append(hostname[4:])
+				try:
+					ipaddress.ip_interface(hostname) # test if it's an IP address or CIDR notation
+					values.append(hostname)
+				except ValueError:
+					try:
+						response = dns.resolver.resolve(hostname+'.', "A", raise_on_no_answer=False)
+						values.extend(map(str, response))
+					except dns.exception.DNSException:
+						pass
+					try:
+						response = dns.resolver.resolve(hostname+'.', "AAAA", raise_on_no_answer=False)
+						values.extend(map(str, response))
+					except dns.exception.DNSException:
+						pass
+
+			else:
+				values.append(hostname)
 
 	return values
 
@@ -1021,15 +1040,17 @@ def set_secondary_dns(hostnames, env):
 		# Validate that all hostnames are valid and that all zone-xfer IP addresses are valid.
 		resolver = dns.resolver.get_default_resolver()
 		resolver.timeout = 5
+		resolver.lifetime = 5
+		
 		for item in hostnames:
 			if not item.startswith("xfr:"):
 				# Resolve hostname.
 				try:
 					response = resolver.resolve(item, "A")
-				except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
+				except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer, dns.resolver.Timeout):
 					try:
 						response = resolver.resolve(item, "AAAA")
-					except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
+					except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer, dns.resolver.Timeout):
 						raise ValueError("Could not resolve the IP address of %s." % item)
 			else:
 				# Validate IP address.
@@ -1062,7 +1083,7 @@ def get_custom_dns_records(custom_dns, qname, rtype):
 def build_recommended_dns(env):
 	ret = []
 	for (domain, zonefile, records) in build_zones(env):
-		# remove records that we don't dislay
+		# remove records that we don't display
 		records = [r for r in records if r[3] is not False]
 
 		# put Required at the top, then Recommended, then everythiing else

management/email_administrator.py

@@ -29,7 +29,7 @@ content = sys.stdin.read().strip()
 
 # If there's nothing coming in, just exit.
 if content == "":
-	sys.exit(0)
+    sys.exit(0)
 
 # create MIME message
 msg = MIMEMultipart('alternative')
@@ -41,7 +41,7 @@ msg['From'] = "\"%s\" <%s>" % (env['PRIMARY_HOSTNAME'], admin_addr)
 msg['To'] = admin_addr
 msg['Subject'] = "[%s] %s" % (env['PRIMARY_HOSTNAME'], subject)
 
-content_html = "<html><body><pre>{}</pre></body></html>".format(html.escape(content))
+content_html = '<html><body><pre style="overflow-x: scroll; white-space: pre;">{}</pre></body></html>'.format(html.escape(content))
 
 msg.attach(MIMEText(content, 'plain'))
 msg.attach(MIMEText(content_html, 'html'))

management/mail_log.py

@@ -73,7 +73,8 @@ def scan_files(collector):
             continue
         elif fn[-3:] == '.gz':
             tmp_file = tempfile.NamedTemporaryFile()
-            shutil.copyfileobj(gzip.open(fn), tmp_file)
+            with gzip.open(fn, 'rb') as f:
+                shutil.copyfileobj(f, tmp_file)
 
         if VERBOSE:
             print("Processing file", fn, "...")

management/ssl_certificates.py

@@ -535,7 +535,8 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 	# Second, check that the certificate matches the private key.
 	if ssl_private_key is not None:
 		try:
-			priv_key = load_pem(open(ssl_private_key, 'rb').read())
+			with open(ssl_private_key, 'rb') as f:
+				priv_key = load_pem(f.read())
 		except ValueError as e:
 			return ("The private key file %s is not a private key file: %s" % (ssl_private_key, str(e)), None)
 

management/status_checks.py

@@ -95,6 +95,12 @@ def run_services_checks(env, output, pool):
 		fatal = fatal or fatal2
 		output2.playback(output)
 
+	# Check fail2ban.
+	code, ret = shell('check_output', ["fail2ban-client", "status"], capture_stderr=True, trap=True)
+	if code != 0:
+		output.print_error("fail2ban is not running.")
+		all_running = False
+
 	if all_running:
 		output.print_ok("All system services are running.")
 
@@ -207,7 +213,8 @@ def check_ssh_password(env, output):
 	# the configuration file.
 	if not os.path.exists("/etc/ssh/sshd_config"):
 		return
-	sshd = open("/etc/ssh/sshd_config").read()
+	with open("/etc/ssh/sshd_config", "r") as f:
+		sshd = f.read()
 	if re.search("\nPasswordAuthentication\s+yes", sshd) \
 		or not re.search("\nPasswordAuthentication\s+no", sshd):
 		output.print_error("""The SSH server on this machine permits password-based login. A more secure
@@ -308,6 +315,8 @@ def run_network_checks(env, output):
 		output.print_ok("IP address is not blacklisted by zen.spamhaus.org.")
 	elif zen == "[timeout]":
 		output.print_warning("Connection to zen.spamhaus.org timed out. We could not determine whether your server's IP address is blacklisted. Please try again later.")
+	elif zen == "[Not Set]":
+		output.print_warning("Could not connect to zen.spamhaus.org. We could not determine whether your server's IP address is blacklisted. Please try again later.")
 	else:
 		output.print_error("""The IP address of this machine %s is listed in the Spamhaus Block List (code %s),
 			which may prevent recipients from receiving your email. See http://www.spamhaus.org/query/ip/%s."""
@@ -541,7 +550,7 @@ def check_dns_zone(domain, env, output, dns_zonefiles):
 		for ns in custom_secondary_ns:
 			# We must first resolve the nameserver to an IP address so we can query it.
 			ns_ips = query_dns(ns, "A")
-			if not ns_ips:
+			if not ns_ips or ns_ips in {'[Not Set]', '[timeout]'}:
 				output.print_error("Secondary nameserver %s is not valid (it doesn't resolve to an IP address)." % ns)
 				continue
 			# Choose the first IP if nameserver returns multiple
@@ -592,7 +601,8 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 			# record that we suggest using is for the KSK (and that's how the DS records were generated).
 			# We'll also give the nice name for the key algorithm.
 			dnssec_keys = load_env_vars_from_file(os.path.join(env['STORAGE_ROOT'], 'dns/dnssec/%s.conf' % alg_name_map[ds_alg]))
-			dnsssec_pubkey = open(os.path.join(env['STORAGE_ROOT'], 'dns/dnssec/' + dnssec_keys['KSK'] + '.key')).read().split("\t")[3].split(" ")[3]
+			with open(os.path.join(env['STORAGE_ROOT'], 'dns/dnssec/' + dnssec_keys['KSK'] + '.key'), 'r') as f:
+				dnsssec_pubkey = f.read().split("\t")[3].split(" ")[3]
 
 			expected_ds_records[ (ds_keytag, ds_alg, ds_digalg, ds_digest) ] = {
 				"record": rr_ds,
@@ -744,6 +754,8 @@ def check_mail_domain(domain, env, output):
 		output.print_ok("Domain is not blacklisted by dbl.spamhaus.org.")
 	elif dbl == "[timeout]":
 		output.print_warning("Connection to dbl.spamhaus.org timed out. We could not determine whether the domain {} is blacklisted. Please try again later.".format(domain))
+	elif dbl == "[Not Set]":
+		output.print_warning("Could not connect to dbl.spamhaus.org. We could not determine whether the domain {} is blacklisted. Please try again later.".format(domain))
 	else:
 		output.print_error("""This domain is listed in the Spamhaus Domain Block List (code %s),
 			which may prevent recipients from receiving your mail.
@@ -788,12 +800,17 @@ def query_dns(qname, rtype, nxdomain='[Not Set]', at=None, as_list=False):
 	# running bind server), or if the 'at' argument is specified, use that host
 	# as the nameserver.
 	resolver = dns.resolver.get_default_resolver()
-	if at:
+
+	# Make sure at is not a string that cannot be used as a nameserver
+	if at and at not in {'[Not set]', '[timeout]'}:
 		resolver = dns.resolver.Resolver()
 		resolver.nameservers = [at]
 
 	# Set a timeout so that a non-responsive server doesn't hold us back.
 	resolver.timeout = 5
+	# The number of seconds to spend trying to get an answer to the question. If the
+	# lifetime expires a dns.exception.Timeout exception will be raised.
+	resolver.lifetime = 5
 
 	# Do the query.
 	try:
@@ -895,11 +912,11 @@ def list_apt_updates(apt_update=True):
 	return pkgs
 
 def what_version_is_this(env):
-	# This function runs `git describe --abbrev=0` on the Mail-in-a-Box installation directory.
+	# This function runs `git describe --always --abbrev=0` on the Mail-in-a-Box installation directory.
 	# Git may not be installed and Mail-in-a-Box may not have been cloned from github,
 	# so this function may raise all sorts of exceptions.
 	miab_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-	tag = shell("check_output", ["/usr/bin/git", "describe", "--abbrev=0"], env={"GIT_DIR": os.path.join(miab_dir, '.git')}).strip()
+	tag = shell("check_output", ["/usr/bin/git", "describe", "--always", "--abbrev=0"], env={"GIT_DIR": os.path.join(miab_dir, '.git')}).strip()
 	return tag
 
 def get_latest_miab_version():
@@ -947,7 +964,8 @@ def run_and_output_changes(env, pool):
 	# Load previously saved status checks.
 	cache_fn = "/var/cache/mailinabox/status_checks.json"
 	if os.path.exists(cache_fn):
-		prev = json.load(open(cache_fn))
+		with open(cache_fn, 'r') as f:
+			prev = json.load(f)
 
 		# Group the serial output into categories by the headings.
 		def group_by_heading(lines):

management/templates/aliases.html

@@ -7,7 +7,7 @@
 
 <h3>Add a mail alias</h3>
 
-<p>Aliases are email forwarders. An alias can forward email to a <a href="#" onclick="return show_panel('users')">mail user</a> or to any email address.</p>
+<p>Aliases are email forwarders. An alias can forward email to a <a href="#users">mail user</a> or to any email address.</p>
 
 <p>To use an alias or any address besides your own login username in outbound mail, the sending user must be included as a permitted sender for the alias.</p>
 

management/templates/custom-dns.html

@@ -77,7 +77,7 @@
 
 <h3>Using a secondary nameserver</h3>
 
-<p>If your TLD requires you to have two separate nameservers, you can either set up <a href="#" onclick="return show_panel('external_dns')">external DNS</a> and ignore the DNS server on this box entirely, or use the DNS server on this box but add a secondary (aka &ldquo;slave&rdquo;) nameserver.</p>
+<p>If your TLD requires you to have two separate nameservers, you can either set up <a href="#external_dns">external DNS</a> and ignore the DNS server on this box entirely, or use the DNS server on this box but add a secondary (aka &ldquo;slave&rdquo;) nameserver.</p>
 <p>If you choose to use a secondary nameserver, you must find a secondary nameserver service provider. Your domain name registrar or virtual cloud provider may provide this service for you. Once you set up the secondary nameserver service, enter the hostname (not the IP address) of <em>their</em> secondary nameserver in the box below.</p>
 
 <form class="form-horizontal" role="form" onsubmit="do_set_secondary_dns(); return false;">
@@ -96,7 +96,7 @@
     <div class="col-sm-offset-1 col-sm-11">
       <p class="small">
         Multiple secondary servers can be separated with commas or spaces (i.e., <code>ns2.hostingcompany.com ns3.hostingcompany.com</code>).
-        To enable zone transfers to additional servers without listing them as secondary nameservers, add an IP address or subnet using <code>xfr:10.20.30.40</code> or <code>xfr:10.0.0.0/8</code>.
+        To enable zone transfers to additional servers without listing them as secondary nameservers, prefix a hostname, IP address, or subnet with <code>xfr:</code>, e.g. <code>xfr:10.20.30.40</code> or <code>xfr:10.0.0.0/8</code>.
       </p>
       <p id="secondarydns-clear-instructions" style="display: none" class="small">
         Clear the input field above and click Update to use this machine itself as secondary DNS, which is the default/normal setup.

management/templates/index.html

@@ -11,9 +11,9 @@
 
         <link rel="stylesheet" href="/admin/assets/bootstrap/css/bootstrap.min.css">
         <style>
-	    body {
+            body {
-		overflow-y: scroll;
+              overflow-y: scroll;
-                padding-bottom: 20px;
+              padding-bottom: 20px;
             }
 
             p {
@@ -36,20 +36,20 @@
               margin-bottom: 13px;
               margin-top: 30px;
             }
-              .panel-heading h3 {
+            .panel-heading h3 {
-                border: none;
+              border: none;
-                padding: 0;
+              padding: 0;
-                margin: 0;
+              margin: 0;
-              }
+            }
 
             h4 {
               font-size: 110%;
               margin-bottom: 13px;
               margin-top: 18px;
             }
-              h4:first-child {
+            h4:first-child {
-                margin-top: 6px;
+              margin-top: 6px;
-              }
+            }
 
             .admin_panel {
               display: none;
@@ -59,9 +59,9 @@
               margin: 1.5em 0;
             }
 
-	    ol li {
+            ol li {
-	       margin-bottom: 1em;
+              margin-bottom: 1em;
-	    }
+            }
 
             .if-logged-in { display: none; }
             .if-logged-in-admin { display: none; }
@@ -73,11 +73,6 @@
           filter: invert(100%) hue-rotate(180deg);
         }
 
-        /* Set explicit background color (necessary for Firefox) */
-        html {
-          background-color: #111;
-        }
-          
         /* Override Boostrap theme here to give more contrast. The black turns to white by the filter. */
         .form-control {
           color: black !important;
@@ -117,30 +112,30 @@
             <li class="dropdown if-logged-in-admin">
               <a href="#" class="dropdown-toggle" data-toggle="dropdown">System <b class="caret"></b></a>
               <ul class="dropdown-menu">
-                <li><a href="#system_status" onclick="return show_panel(this);">Status Checks</a></li>
+                <li><a href="#system_status">Status Checks</a></li>
-                <li><a href="#tls" onclick="return show_panel(this);">TLS (SSL) Certificates</a></li>
+                <li><a href="#tls">TLS (SSL) Certificates</a></li>
-                <li><a href="#system_backup" onclick="return show_panel(this);">Backup Status</a></li>
+                <li><a href="#system_backup">Backup Status</a></li>
                 <li class="divider"></li>
                 <li class="dropdown-header">Advanced Pages</li>
-                <li><a href="#custom_dns" onclick="return show_panel(this);">Custom DNS</a></li>
+                <li><a href="#custom_dns">Custom DNS</a></li>
-                <li><a href="#external_dns" onclick="return show_panel(this);">External DNS</a></li>
+                <li><a href="#external_dns">External DNS</a></li>
-                <li><a href="#munin" onclick="return show_panel(this);">Munin Monitoring</a></li>
+                <li><a href="#munin">Munin Monitoring</a></li>
               </ul>
             </li>
-            <li><a href="#mail-guide" onclick="return show_panel(this);" class="if-logged-in-not-admin">Mail</a></li>
+            <li><a href="#mail-guide" class="if-logged-in-not-admin">Mail</a></li>
             <li class="dropdown if-logged-in-admin">
               <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mail &amp; Users <b class="caret"></b></a>
               <ul class="dropdown-menu">
-                <li><a href="#mail-guide" onclick="return show_panel(this);">Instructions</a></li>
+                <li><a href="#mail-guide">Instructions</a></li>
-                <li><a href="#users" onclick="return show_panel(this);">Users</a></li>
+                <li><a href="#users">Users</a></li>
-                <li><a href="#aliases" onclick="return show_panel(this);">Aliases</a></li>
+                <li><a href="#aliases">Aliases</a></li>
                 <li class="divider"></li>
                 <li class="dropdown-header">Your Account</li>
-                <li><a href="#mfa" onclick="return show_panel(this);">Two-Factor Authentication</a></li>
+                <li><a href="#mfa">Two-Factor Authentication</a></li>
               </ul>
             </li>
-            <li><a href="#sync_guide" onclick="return show_panel(this);" class="if-logged-in">Contacts/Calendar</a></li>
+            <li><a href="#sync_guide" class="if-logged-in">Contacts/Calendar</a></li>
-            <li><a href="#web" onclick="return show_panel(this);" class="if-logged-in-admin">Web</a></li>
+            <li><a href="#web" class="if-logged-in-admin">Web</a></li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
             <li class="if-logged-in"><a href="#" onclick="do_logout(); return false;" style="color: white">Log out</a></li>
@@ -426,23 +421,25 @@ function do_logout() {
 }
 
 function show_panel(panelid) {
-  if (panelid.getAttribute)
+  if (panelid.getAttribute) {
     // we might be passed an HTMLElement <a>.
     panelid = panelid.getAttribute('href').substring(1);
+  }
 
   $('.admin_panel').hide();
   $('#panel_' + panelid).show();
-  if (typeof localStorage != 'undefined')
-    localStorage.setItem("miab-cp-lastpanel", panelid);
   if (window["show_" + panelid])
     window["show_" + panelid]();
 
   current_panel = panelid;
   switch_back_to_panel = null;
-
-  return false; // when called from onclick, cancel navigation
 }
 
+window.onhashchange = function() {
+  var panelid = window.location.hash.substring(1);
+  show_panel(panelid);
+};
+
 $(function() {
   // Recall saved user credentials.
   try {
@@ -457,8 +454,9 @@ $(function() {
   show_hide_menus();
 
   // Recall what the user was last looking at.
-  if (api_credentials != null && typeof localStorage != 'undefined' && localStorage.getItem("miab-cp-lastpanel")) {
+  if (api_credentials != null && window.location.hash) {
-    show_panel(localStorage.getItem("miab-cp-lastpanel"));
+    var panelid = window.location.hash.substring(1);
+    show_panel(panelid);
   } else if (api_credentials != null) {
     show_panel('welcome');
   } else {

management/templates/login.html

@@ -168,7 +168,18 @@ function do_login() {
       // Open the next panel the user wants to go to. Do this after the XHR response
       // is over so that we don't start a new XHR request while this one is finishing,
       // which confuses the loading indicator.
-      setTimeout(function() { show_panel(!switch_back_to_panel || switch_back_to_panel == "login" ? 'welcome' : switch_back_to_panel) }, 300);
+      setTimeout(function() {
+        if (window.location.hash) {
+          var panelid = window.location.hash.substring(1);
+          show_panel(panelid);
+        } else {
+          show_panel(
+            !switch_back_to_panel || switch_back_to_panel == "login"
+            ? 'welcome'
+            : switch_back_to_panel)
+        }
+      }, 300);
+
     }
   },
   undefined,

management/templates/mail-guide.html

@@ -36,7 +36,7 @@
 					<tr><th>Password:</th> <td>Your mail password.</td></tr>
 					</table>
 
-					<p>In addition to setting up your email, you&rsquo;ll also need to set up <a href="#sync_guide" onclick="return show_panel(this);">contacts and calendar synchronization</a> separately.</p>
+					<p>In addition to setting up your email, you&rsquo;ll also need to set up <a href="#sync_guide">contacts and calendar synchronization</a> separately.</p>
 
 					<p>As an alternative to IMAP you can also use the POP protocol: choose POP as the protocol, port 995, and SSL or TLS security in your mail client. The SMTP settings and usernames and passwords remain the same. However, we recommend you use IMAP instead.</p>
 

management/templates/sync-guide.html

@@ -17,14 +17,14 @@
 			<tr><th>Calendar</td> <td><a href="https://{{hostname}}/cloud/calendar">https://{{hostname}}/cloud/calendar</a></td></tr>
 			</table>
 
-			<p>Log in settings are the same as with <a href="#mail-guide" onclick="return show_panel(this);">mail</a>: your
+			<p>Log in settings are the same as with <a href="#mail-guide">mail</a>: your
 			complete email address and your mail password.</p>
 		</div>
 
 		<div class="col-sm-6">
 			<h4>On your mobile device</h4>
 
-			<p>If you set up your <a href="#mail-guide" onclick="return show_panel(this);">mail</a> using Exchange/ActiveSync,
+			<p>If you set up your <a href="#mail-guide">mail</a> using Exchange/ActiveSync,
 			your contacts and calendar may already appear on your device.</p>
 			<p>Otherwise, here are some apps that can synchronize your contacts and calendar to your Android phone.</p>
 

management/templates/system-backup.html

@@ -5,7 +5,7 @@
 
 <h2>Backup Status</h2>
 
-<p>The box makes an incremental backup each night. By default the backup is stored on the machine itself, but you can also store it on S3-compatible services like Amazon Web Services (AWS).</p>
+<p>The box makes an incremental backup each night. You can store the backup on any Amazon Web Services S3-compatible service, or other options.</p>
 
 <h3>Configuration</h3>
 
@@ -45,6 +45,10 @@
     <label for="backup-target-rsync-host" class="col-sm-2 control-label">Hostname</label>
     <div class="col-sm-8">
       <input type="text" placeholder="hostname.local" class="form-control" rows="1" id="backup-target-rsync-host">
+      <div class="small" style="margin-top: 2px">
+	The hostname at your rsync provider, e.g. <tt>da2327.rsync.net</tt>.  Optionally includes a colon
+	and the provider's non-standard ssh port number, e.g. <tt>u215843.your-storagebox.de:23</tt>. 
+      </div>
     </div>
   </div>
   <div class="form-group backup-target-rsync">
@@ -66,9 +70,12 @@
       <div class="small" style="margin-top: 2px">
         Copy the Public SSH Key above, and paste it within the <tt>~/.ssh/authorized_keys</tt>
         of target user on the backup server specified above. That way you'll enable secure and
-        passwordless authentication from your mail-in-a-box server and your backup server.
+        passwordless authentication from your Mail-in-a-Box server and your backup server.
       </div>
     </div>
+    <div id="copy_pub_key_div" class="col-sm">
+      <button type="button" class="btn btn-small" onclick="copy_pub_key_to_clipboard()">Copy</button>
+    </div> 
   </div>
   <!-- S3 BACKUP -->
   <div class="form-group backup-target-s3">
@@ -91,13 +98,19 @@
   <div class="form-group backup-target-s3">
     <label for="backup-target-s3-host" class="col-sm-2 control-label">S3 Host / Endpoint</label>
     <div class="col-sm-8">
-      <input type="text" placeholder="Endpoint" class="form-control" rows="1" id="backup-target-s3-host">
+      <input type="text" placeholder="https://s3.backuphost.com" class="form-control" rows="1" id="backup-target-s3-host">
     </div>
   </div>
   <div class="form-group backup-target-s3">
-    <label for="backup-target-s3-path" class="col-sm-2 control-label">S3 Path</label>
+    <label for="backup-target-s3-region-name" class="col-sm-2 control-label">S3 Region Name <span style="font-weight: normal">(if required)</span></label>
     <div class="col-sm-8">
-      <input type="text" placeholder="your-bucket-name/backup-directory" class="form-control" rows="1" id="backup-target-s3-path">
+      <input type="text" placeholder="region.name" class="form-control" rows="1" id="backup-target-s3-region-name">
+    </div>
+  </div>
+  <div class="form-group backup-target-s3">
+    <label for="backup-target-s3-path" class="col-sm-2 control-label">S3 Bucket &amp; Path</label>
+    <div class="col-sm-8">
+      <input type="text" placeholder="bucket-name/backup-directory" class="form-control" rows="1" id="backup-target-s3-path">
     </div>
   </div>
   <div class="form-group backup-target-s3">
@@ -259,19 +272,18 @@ function show_custom_backup() {
         } else if (r.target == "off") {
           $("#backup-target-type").val("off");
         } else if (r.target.substring(0, 8) == "rsync://") {
-          $("#backup-target-type").val("rsync");
+          const spec = url_split(r.target);
-          var path = r.target.substring(8).split('//');
+          $("#backup-target-type").val(spec.scheme);
-          var host_parts = path.shift().split('@');
+	  $("#backup-target-rsync-user").val(spec.user);
-          $("#backup-target-rsync-user").val(host_parts[0]);
+	  $("#backup-target-rsync-host").val(spec.host);
-          $("#backup-target-rsync-host").val(host_parts[1]);
+	  $("#backup-target-rsync-path").val(spec.path);
-          $("#backup-target-rsync-path").val('/'+path[0]);
         } else if (r.target.substring(0, 5) == "s3://") {
+          const spec = url_split(r.target);
           $("#backup-target-type").val("s3");
-          var hostpath = r.target.substring(5).split('/');
+          $("#backup-target-s3-host-select").val(spec.host);
-          var host = hostpath.shift();
+          $("#backup-target-s3-host").val(spec.host);
-          $("#backup-target-s3-host-select").val(host);
+          $("#backup-target-s3-region-name").val(spec.user); // stuffing the region name in the username
-          $("#backup-target-s3-host").val(host);
+          $("#backup-target-s3-path").val(spec.path);
-          $("#backup-target-s3-path").val(hostpath.join('/'));
         } else if (r.target.substring(0, 5) == "b2://") {
           $("#backup-target-type").val("b2");
           var targetPath = r.target.substring(5);
@@ -279,7 +291,7 @@ function show_custom_backup() {
           var b2_applicationkey = targetPath.split(':')[1].split('@')[0];
           var b2_bucket = targetPath.split('@')[1];
           $("#backup-target-b2-user").val(b2_application_keyid);
-          $("#backup-target-b2-pass").val(b2_applicationkey);
+          $("#backup-target-b2-pass").val(decodeURIComponent(b2_applicationkey));
           $("#backup-target-b2-bucket").val(b2_bucket);
         }
         toggle_form()
@@ -295,13 +307,16 @@ function set_custom_backup() {
   if (target_type == "local" || target_type == "off")
     target = target_type;
   else if (target_type == "s3")
-    target = "s3://" + $("#backup-target-s3-host").val() + "/" + $("#backup-target-s3-path").val();
+    target = "s3://"
+             + ($("#backup-target-s3-region-name").val() ? ($("#backup-target-s3-region-name").val() + "@") : "")
+             + $("#backup-target-s3-host").val()
+             + "/" + $("#backup-target-s3-path").val();
   else if (target_type == "rsync") {
     target = "rsync://" + $("#backup-target-rsync-user").val() + "@" + $("#backup-target-rsync-host").val()
                                                                 + "/" + $("#backup-target-rsync-path").val();
     target_user = '';
   } else if (target_type == "b2") {
-    target = 'b2://' + $('#backup-target-b2-user').val() + ':' + $('#backup-target-b2-pass').val()
+    target = 'b2://' + $('#backup-target-b2-user').val() + ':' + encodeURIComponent($('#backup-target-b2-pass').val())
         + '@' + $('#backup-target-b2-bucket').val()
     target_user = '';
     target_pass = '';
@@ -344,4 +359,42 @@ function init_inputs(target_type) {
     set_host($('#backup-target-s3-host-select').val());
   }
 }
+
+// Return a two-element array of the substring preceding and the substring following
+// the first occurence of separator in string. Return [undefined, string] if the
+// separator does not appear in string.
+const split1_rest = (string, separator) => {
+    const index = string.indexOf(separator);
+    return (index >= 0) ? [string.substring(0, index), string.substring(index + separator.length)] : [undefined, string];
+};
+
+// Note: The manifest JS URL class does not work in some security-conscious
+// settings, e.g. Brave browser, so we roll our own that handles only what we need.
+//
+// Use greedy separator parsing to get parts of a MIAB backup target url.
+// Note: path will not include a leading forward slash '/'
+const url_split = url => {
+    const [ scheme, scheme_rest ] = split1_rest(url, '://');
+    const [ user, user_rest ] = split1_rest(scheme_rest, '@');
+    const [ host, path ] = split1_rest(user_rest, '/');
+
+    return {
+	scheme,
+	user,
+	host,
+	path,
+    }
+};
+  
+// Hide Copy button if not in a modern clipboard-supporting environment.
+// Using document API because jQuery is not necessarily available in this script scope.
+if (!(navigator && navigator.clipboard && navigator.clipboard.writeText)) {
+    document.getElementById('copy_pub_key_div').hidden = true;
+}
+
+function copy_pub_key_to_clipboard() {
+    const ssh_pub_key = $("#ssh-pub-key").val();
+    navigator.clipboard.writeText(ssh_pub_key);
+}  
+
 </script>

management/templates/system-status.html

@@ -10,13 +10,13 @@
   border-top: none;
   padding-top: 0;
 }
-#system-checks .status-error td {
+#system-checks .status-error td, .summary-error {
   color: #733;
 }
-#system-checks .status-warning td {
+#system-checks .status-warning td, .summary-warning {
   color: #770;
 }
-#system-checks .status-ok td {
+#system-checks .status-ok td, .summary-ok {
   color: #040;
 }
 #system-checks div.extra {
@@ -52,6 +52,9 @@
 	</div> <!-- /col -->
 	<div class="col-md-pull-3 col-md-8">
 
+          <div id="system-checks-summary">
+          </div>
+
 <table id="system-checks" class="table" style="max-width: 60em">
   <thead>
   </thead>
@@ -64,6 +67,9 @@
 
 <script>
 function show_system_status() {
+  const summary = $('#system-checks-summary');
+  summary.html("");
+
   $('#system-checks tbody').html("<tr><td colspan='2' class='text-muted'>Loading...</td></tr>")
 
   api(
@@ -93,6 +99,12 @@ function show_system_status() {
     { },
     function(r) {
       $('#system-checks tbody').html("");
+      const ok_symbol = "✓";
+      const error_symbol = "✖";
+      const warning_symbol = "?";
+
+      let count_by_status = { ok: 0, error: 0, warning: 0 };
+
       for (var i = 0; i < r.length; i++) {
         var n = $("<tr><td class='status'/><td class='message'><p style='margin: 0'/><div class='extra'/><a class='showhide' href='#'/></tr>");
         if (i == 0) n.addClass('first')
@@ -100,9 +112,12 @@ function show_system_status() {
                 n.addClass(r[i].type)
         else
                 n.addClass("status-" + r[i].type)
-        if (r[i].type == "ok") n.find('td.status').text("✓")
+
-        if (r[i].type == "error") n.find('td.status').text("✖")
+        if (r[i].type == "ok") n.find('td.status').text(ok_symbol);
-        if (r[i].type == "warning") n.find('td.status').text("?")
+        if (r[i].type == "error") n.find('td.status').text(error_symbol);
+        if (r[i].type == "warning") n.find('td.status').text(warning_symbol);
+        count_by_status[r[i].type]++;
+
         n.find('td.message p').text(r[i].text)
         $('#system-checks tbody').append(n);
 
@@ -122,8 +137,17 @@ function show_system_status() {
           n.find('> td.message > div').append(m);
         }
       }
-    })
 
+      // Summary counts
+      summary.html("Summary: ");
+      if (count_by_status['error'] + count_by_status['warning'] == 0) {
+        summary.append($('<span class="summary-ok"/>').text(`All ${count_by_status['ok']} ${ok_symbol} OK`));
+      } else {
+        summary.append($('<span class="summary-ok"/>').text(`${count_by_status['ok']} ${ok_symbol} OK, `));
+        summary.append($('<span class="summary-error"/>').text(`${count_by_status['error']} ${error_symbol} Error, `));
+        summary.append($('<span class="summary-warning"/>').text(`${count_by_status['warning']} ${warning_symbol} Warning`));
+      }
+    })
 }
 
 var current_privacy_setting = null;

management/templates/users.html

@@ -31,9 +31,9 @@
 </form>
 <ul style="margin-top: 1em; padding-left: 1.5em; font-size: 90%;">
   <li>Passwords must be at least eight characters consisting of English letters and numbers only. For best results, <a href="#" onclick="return generate_random_password()">generate a random password</a>.</li>
-  <li>Use <a href="#" onclick="return show_panel('aliases')">aliases</a> to create email addresses that forward to existing accounts.</li>
+  <li>Use <a href="#aliases">aliases</a> to create email addresses that forward to existing accounts.</li>
   <li>Administrators get access to this control panel.</li>
-  <li>User accounts cannot contain any international (non-ASCII) characters, but <a href="#" onclick="return show_panel('aliases');">aliases</a> can.</li>
+  <li>User accounts cannot contain any international (non-ASCII) characters, but <a href="#aliases">aliases</a> can.</li>
 </ul>
 
 <h3>Existing mail users</h3>

management/templates/web.html

@@ -10,7 +10,7 @@
 <p>You can replace the default website with your own HTML pages and other static files. This control panel won&rsquo;t help you design a website, but once you have <tt>.html</tt> files you can upload them following these instructions:</p>
 
 <ol>
-<li>Ensure that any domains you are publishing a website for have no problems on the <a href="#system_status" onclick="return show_panel(this);">Status Checks</a> page.</li>
+<li>Ensure that any domains you are publishing a website for have no problems on the <a href="#system_status">Status Checks</a> page.</li>
 
 <li>On your personal computer, install an SSH file transfer program such as <a href="https://filezilla-project.org/">FileZilla</a> or <a href="http://linuxcommand.org/man_pages/scp1.html">scp</a>.</li>
 
@@ -32,7 +32,7 @@
         </tbody>
 </table>
 
-<p>To add a domain to this table, create a dummy <a href="#users" onclick="return show_panel(this);">mail user</a> or <a href="#aliases" onclick="return show_panel(this);">alias</a> on the domain first and see the <a href="https://mailinabox.email/guide.html#domain-name-configuration">setup guide</a> for adding nameserver records to the new domain at your registrar (but <i>not</i> glue records).</p>
+<p>To add a domain to this table, create a dummy <a href="#users">mail user</a> or <a href="#aliases">alias</a> on the domain first and see the <a href="https://mailinabox.email/guide.html#domain-name-configuration">setup guide</a> for adding nameserver records to the new domain at your registrar (but <i>not</i> glue records).</p>
 
 </ol>
 

management/utils.py

@@ -14,7 +14,9 @@ def load_env_vars_from_file(fn):
     # Load settings from a KEY=VALUE file.
     import collections
     env = collections.OrderedDict()
-    for line in open(fn): env.setdefault(*line.strip().split("=", 1))
+    with open(fn, 'r')  as f:
+        for line in f:
+            env.setdefault(*line.strip().split("=", 1))
     return env
 
 def save_environment(env):
@@ -34,7 +36,8 @@ def load_settings(env):
     import rtyaml
     fn = os.path.join(env['STORAGE_ROOT'], 'settings.yaml')
     try:
-        config = rtyaml.load(open(fn, "r"))
+        with open(fn, "r") as f:
+            config = rtyaml.load(f)
         if not isinstance(config, dict): raise ValueError() # caught below
         return config
     except:

management/web_update.py

@@ -63,7 +63,8 @@ def get_web_domains_with_root_overrides(env):
 	root_overrides = { }
 	nginx_conf_custom_fn = os.path.join(env["STORAGE_ROOT"], "www/custom.yaml")
 	if os.path.exists(nginx_conf_custom_fn):
-		custom_settings = rtyaml.load(open(nginx_conf_custom_fn))
+		with open(nginx_conf_custom_fn, 'r') as f:
+			custom_settings = rtyaml.load(f)
 		for domain, settings in custom_settings.items():
 			for type, value in [('redirect', settings.get('redirects', {}).get('/')),
 				('proxy', settings.get('proxies', {}).get('/'))]:
@@ -75,13 +76,18 @@ def do_web_update(env):
 	# Pre-load what SSL certificates we will use for each domain.
 	ssl_certificates = get_ssl_certificates(env)
 
+	# Helper for reading config files and templates
+	def read_conf(conf_fn):
+		with open(os.path.join(os.path.dirname(__file__), "../conf", conf_fn), "r") as f:
+			return f.read()
+
 	# Build an nginx configuration file.
-	nginx_conf = open(os.path.join(os.path.dirname(__file__), "../conf/nginx-top.conf")).read()
+	nginx_conf = read_conf("nginx-top.conf")
 
 	# Load the templates.
-	template0 = open(os.path.join(os.path.dirname(__file__), "../conf/nginx.conf")).read()
+	template0 = read_conf("nginx.conf")
-	template1 = open(os.path.join(os.path.dirname(__file__), "../conf/nginx-alldomains.conf")).read()
+	template1 = read_conf("nginx-alldomains.conf")
-	template2 = open(os.path.join(os.path.dirname(__file__), "../conf/nginx-primaryonly.conf")).read()
+	template2 = read_conf("nginx-primaryonly.conf")
 	template3 = "\trewrite ^(.*) https://$REDIRECT_DOMAIN$1 permanent;\n"
 
 	# Add the PRIMARY_HOST configuration first so it becomes nginx's default server.
@@ -141,11 +147,8 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 	def hashfile(filepath):
 		import hashlib
 		sha1 = hashlib.sha1()
-		f = open(filepath, 'rb')
+		with open(filepath, 'rb') as f:
-		try:
 			sha1.update(f.read())
-		finally:
-			f.close()
 		return sha1.hexdigest()
 	nginx_conf_extra += "\t# ssl files sha1: %s / %s\n" % (hashfile(tls_cert["private-key"]), hashfile(tls_cert["certificate"]))
 
@@ -153,7 +156,8 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 	hsts = "yes"
 	nginx_conf_custom_fn = os.path.join(env["STORAGE_ROOT"], "www/custom.yaml")
 	if os.path.exists(nginx_conf_custom_fn):
-		yaml = rtyaml.load(open(nginx_conf_custom_fn))
+		with open(nginx_conf_custom_fn, 'r') as f:
+			yaml = rtyaml.load(f)
 		if domain in yaml:
 			yaml = yaml[domain]
 

security.md

@@ -1,7 +1,7 @@
 Mail-in-a-Box Security Guide
 ============================
 
-Mail-in-a-Box turns a fresh Ubuntu 18.04 LTS 64-bit machine into a mail server appliance by installing and configuring various components.
+Mail-in-a-Box turns a fresh Ubuntu 22.04 LTS 64-bit machine into a mail server appliance by installing and configuring various components.
 
 This page documents the security posture of Mail-in-a-Box. The term “box” is used below to mean a configured Mail-in-a-Box.
 

setup/bootstrap.sh

@@ -19,11 +19,11 @@ if [ -z "$TAG" ]; then
 	# want to display in status checks.
 	#
 	# Allow point-release versions of the major releases, e.g. 22.04.1 is OK.
-	UBUNTU_VERSION=$( lsb_release -d | sed 's/.*:\s*//' | sed 's/\([0-9]*\.[0-9]*\)\.[0-9]/\1/' )"
+	UBUNTU_VERSION=$( lsb_release -d | sed 's/.*:\s*//' | sed 's/\([0-9]*\.[0-9]*\)\.[0-9]/\1/' )
 	if [ "$UBUNTU_VERSION" == "Ubuntu 22.04 LTS" ]; then
 		# This machine is running Ubuntu 22.04, which is supported by
 		# Mail-in-a-Box versions 60 and later.
-		TAG=v60
+		TAG=v65
 	elif [ "$UBUNTU_VERSION" == "Ubuntu 18.04 LTS" ]; then
 		# This machine is running Ubuntu 18.04, which is supported by
 		# Mail-in-a-Box versions 0.40 through 5x.
@@ -73,7 +73,7 @@ fi
 cd $HOME/mailinabox
 
 # Update it.
-if [ "$TAG" != $(git describe) ]; then
+if [ "$TAG" != $(git describe --always) ]; then
 	echo Updating Mail-in-a-Box to $TAG . . .
 	git fetch --depth 1 --force --prune origin tag $TAG
 	if ! git checkout -q $TAG; then

setup/dkim.sh

@@ -63,7 +63,7 @@ chmod go-rwx $STORAGE_ROOT/mail/dkim
 tools/editconf.py /etc/opendmarc.conf -s \
 	"Syslog=true" \
 	"Socket=inet:8893@[127.0.0.1]" \
-	"FailureReports=true"
+	"FailureReports=false"
 
 # SPFIgnoreResults causes the filter to ignore any SPF results in the header
 # of the message. This is useful if you want the filter to perfrom SPF checks
@@ -82,11 +82,11 @@ tools/editconf.py /etc/opendmarc.conf -s \
 tools/editconf.py /etc/opendmarc.conf -s \
         "SPFSelfValidate=true"
 
-# Enables generation of failure reports for sending domains that publish a
+# Disables generation of failure reports for sending domains that publish a
 # "none" policy.
 
 tools/editconf.py /etc/opendmarc.conf -s \
-        "FailureReportsOnNone=true"
+        "FailureReportsOnNone=false"
 
 # AlwaysAddARHeader Adds an "Authentication-Results:" header field even to
 # unsigned messages from domains with no "signs all" policy. The reported DKIM

setup/mail-dovecot.sh

@@ -202,13 +202,13 @@ chmod -R o-rwx /etc/dovecot
 
 # Ensure mailbox files have a directory that exists and are owned by the mail user.
 mkdir -p $STORAGE_ROOT/mail/mailboxes
-chown -R mail.mail $STORAGE_ROOT/mail/mailboxes
+chown -R mail:mail $STORAGE_ROOT/mail/mailboxes
 
 # Same for the sieve scripts.
 mkdir -p $STORAGE_ROOT/mail/sieve
 mkdir -p $STORAGE_ROOT/mail/sieve/global_before
 mkdir -p $STORAGE_ROOT/mail/sieve/global_after
-chown -R mail.mail $STORAGE_ROOT/mail/sieve
+chown -R mail:mail $STORAGE_ROOT/mail/sieve
 
 # Allow the IMAP/POP ports in the firewall.
 ufw_allow imaps

setup/munin.sh

@@ -34,8 +34,8 @@ contact.admin.always_send warning critical
 EOF
 
 # The Debian installer touches these files and chowns them to www-data:adm for use with spawn-fcgi
-chown munin. /var/log/munin/munin-cgi-html.log
+chown munin /var/log/munin/munin-cgi-html.log
-chown munin. /var/log/munin/munin-cgi-graph.log
+chown munin /var/log/munin/munin-cgi-graph.log
 
 # ensure munin-node knows the name of this machine
 # and reduce logging level to warning

setup/nextcloud.sh

@@ -21,24 +21,28 @@ echo "Installing Nextcloud (contacts/calendar)..."
 #   we automatically install intermediate versions as needed.
 # * The hash is the SHA1 hash of the ZIP package, which you can find by just running this script and
 #   copying it from the error message when it doesn't match what is below.
-nextcloud_ver=23.0.8
+nextcloud_ver=25.0.7
-nextcloud_hash=9d63416a0697eeecf243d09461f5881f8997f50a
+nextcloud_hash=a5a565c916355005c7b408dd41a1e53505e1a080
 
 # Nextcloud apps
 # --------------
 # * Find the most recent tag that is compatible with the Nextcloud version above by
 #   consulting the <dependencies>...<nextcloud> node at:
-#   https://github.com/nextcloud-releases/contacts/blob/master/appinfo/info.xml
+#   https://github.com/nextcloud-releases/contacts/blob/main/appinfo/info.xml
-#   https://github.com/nextcloud-releases/calendar/blob/master/appinfo/info.xml
+#   https://github.com/nextcloud-releases/calendar/blob/main/appinfo/info.xml
 #   https://github.com/nextcloud/user_external/blob/master/appinfo/info.xml
 # * The hash is the SHA1 hash of the ZIP package, which you can find by just running this script and
 #   copying it from the error message when it doesn't match what is below.
-contacts_ver=4.2.0
+contacts_ver=5.3.0
-contacts_hash=79b506574834db5e1b6ab47aadd4041e12ad9a9c
+contacts_hash=4b0a6666374e3b55cfd2ae9b72e1d458b87d4c8c
-calendar_ver=3.5.0
+
-calendar_hash=941381536287a015081669513f8f79f6f262508a
+# Always ensure the versions are supported, see https://apps.nextcloud.com/apps/calendar
-user_external_ver=3.0.0
+calendar_ver=4.4.2
-user_external_hash=0df781b261f55bbde73d8c92da3f99397000972f
+calendar_hash=21a42e15806adc9b2618760ef94f1797ef399e2f
+
+# And https://apps.nextcloud.com/apps/user_external
+user_external_ver=3.2.0
+user_external_hash=a494073dcdecbbbc79a9c77f72524ac9994d2eec
 
 # Clear prior packages and install dependencies from apt.
 
@@ -69,8 +73,8 @@ InstallNextcloud() {
 	echo "Upgrading to Nextcloud version $version"
 	echo
 
-        # Download and verify
+	# Download and verify
-        wget_verify https://download.nextcloud.com/server/releases/nextcloud-$version.zip $hash /tmp/nextcloud.zip
+	wget_verify https://download.nextcloud.com/server/releases/nextcloud-$version.zip $hash /tmp/nextcloud.zip
 
 	# Remove the current owncloud/Nextcloud
 	rm -rf /usr/local/lib/owncloud
@@ -110,7 +114,7 @@ InstallNextcloud() {
 	# Make sure permissions are correct or the upgrade step won't run.
 	# $STORAGE_ROOT/owncloud may not yet exist, so use -f to suppress
 	# that error.
-	chown -f -R www-data.www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud || /bin/true
+	chown -f -R www-data:www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud || /bin/true
 
 	# If this isn't a new installation, immediately run the upgrade script.
 	# Then check for success (0=ok and 3=no upgrade needed, both are success).
@@ -128,6 +132,7 @@ InstallNextcloud() {
 
 		# Add missing indices. NextCloud didn't include this in the normal upgrade because it might take some time.
 		sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ db:add-missing-indices
+		sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ db:add-missing-primary-keys
 
 		# Run conversion to BigInt identifiers, this process may take some time on large tables.
 		sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ db:convert-filecache-bigint --no-interaction
@@ -173,6 +178,12 @@ if [ ! -d /usr/local/lib/owncloud/ ] || [[ ! ${CURRENT_NEXTCLOUD_VER} =~ ^$nextc
 	if [ ! -z ${CURRENT_NEXTCLOUD_VER} ]; then
 		# Database migrations from ownCloud are no longer possible because ownCloud cannot be run under
 		# PHP 7.
+
+		if [ -e $STORAGE_ROOT/owncloud/config.php ]; then
+			# Remove the read-onlyness of the config, which is needed for migrations, especially for v24
+			sed -i -e '/config_is_read_only/d' $STORAGE_ROOT/owncloud/config.php
+		fi
+
 		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^[89] ]]; then
 			echo "Upgrades from Mail-in-a-Box prior to v0.28 (dated July 30, 2018) with Nextcloud < 13.0.6 (you have ownCloud 8 or 9) are not supported. Upgrade to Mail-in-a-Box version v0.30 first. Setup will continue, but skip the Nextcloud migration."
 			return 0
@@ -183,6 +194,7 @@ if [ ! -d /usr/local/lib/owncloud/ ] || [[ ! ${CURRENT_NEXTCLOUD_VER} =~ ^$nextc
 			echo "Upgrades from Mail-in-a-Box prior to v60 with Nextcloud 19 or earlier are not supported. Upgrade to the latest Mail-in-a-Box version supported on your machine first. Setup will continue, but skip the Nextcloud migration."
 			return 0
 		fi
+
 		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^20 ]]; then
 			InstallNextcloud 21.0.7 f5c7079c5b56ce1e301c6a27c0d975d608bb01c9 4.0.7 45e7cf4bfe99cd8d03625cf9e5a1bb2e90549136 3.0.4 d0284b68135777ec9ca713c307216165b294d0fe
 			CURRENT_NEXTCLOUD_VER="21.0.7"
@@ -191,6 +203,14 @@ if [ ! -d /usr/local/lib/owncloud/ ] || [[ ! ${CURRENT_NEXTCLOUD_VER} =~ ^$nextc
 			InstallNextcloud 22.2.6 9d39741f051a8da42ff7df46ceef2653a1dc70d9 4.1.0 697f6b4a664e928d72414ea2731cb2c9d1dc3077 3.2.2 ce4030ab57f523f33d5396c6a81396d440756f5f 3.0.0 0df781b261f55bbde73d8c92da3f99397000972f
 			CURRENT_NEXTCLOUD_VER="22.2.6"
 		fi
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^22 ]]; then
+			InstallNextcloud 23.0.12 d138641b8e7aabebe69bb3ec7c79a714d122f729 4.1.0 697f6b4a664e928d72414ea2731cb2c9d1dc3077 3.2.2 ce4030ab57f523f33d5396c6a81396d440756f5f 3.0.0 0df781b261f55bbde73d8c92da3f99397000972f
+			CURRENT_NEXTCLOUD_VER="23.0.12"
+		fi
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^23 ]]; then
+			InstallNextcloud 24.0.12 7aa5d61632c1ccf4ca3ff00fb6b295d318c05599 4.1.0 697f6b4a664e928d72414ea2731cb2c9d1dc3077 3.2.2 ce4030ab57f523f33d5396c6a81396d440756f5f 3.0.0 0df781b261f55bbde73d8c92da3f99397000972f
+			CURRENT_NEXTCLOUD_VER="24.0.12"
+		fi
 	fi
 
 	InstallNextcloud $nextcloud_ver $nextcloud_hash $contacts_ver $contacts_hash $calendar_ver $calendar_hash $user_external_ver $user_external_hash
@@ -259,7 +279,7 @@ EOF
 EOF
 
 	# Set permissions
-	chown -R www-data.www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
+	chown -R www-data:www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
 
 	# Execute Nextcloud's setup step, which creates the Nextcloud sqlite database.
 	# It also wipes it if it exists. And it updates config.php with database
@@ -284,12 +304,12 @@ php$PHP_VER <<EOF > $CONFIG_TEMP && mv $CONFIG_TEMP $STORAGE_ROOT/owncloud/confi
 <?php
 include("$STORAGE_ROOT/owncloud/config.php");
 
-\$CONFIG['config_is_read_only'] = true;
+\$CONFIG['config_is_read_only'] = false;
 
 \$CONFIG['trusted_domains'] = array('$PRIMARY_HOSTNAME');
 
 \$CONFIG['memcache.local'] = '\OC\Memcache\APCu';
-\$CONFIG['overwrite.cli.url'] = '/cloud';
+\$CONFIG['overwrite.cli.url'] = 'https://${PRIMARY_HOSTNAME}/cloud';
 \$CONFIG['mail_from_address'] = 'administrator'; # just the local part, matches our master administrator address
 
 \$CONFIG['logtimezone'] = '$TIMEZONE';
@@ -311,7 +331,7 @@ var_export(\$CONFIG);
 echo ";";
 ?>
 EOF
-chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
+chown www-data:www-data $STORAGE_ROOT/owncloud/config.php
 
 # Enable/disable apps. Note that this must be done after the Nextcloud setup.
 # The firstrunwizard gave Josh all sorts of problems, so disabling that.
@@ -353,20 +373,45 @@ tools/editconf.py /etc/php/$PHP_VER/cli/conf.d/10-opcache.ini -c ';' \
 	opcache.save_comments=1 \
 	opcache.revalidate_freq=1
 
-# Migrate users_external data from <0.6.0 to version 3.0.0 (see https://github.com/nextcloud/user_external).
+# Migrate users_external data from <0.6.0 to version 3.0.0
+# (see https://github.com/nextcloud/user_external).
 # This version was probably in use in Mail-in-a-Box v0.41 (February 26, 2019) and earlier.
 # We moved to v0.6.3 in 193763f8. Ignore errors - maybe there are duplicated users with the
 # correct backend already.
 sqlite3 $STORAGE_ROOT/owncloud/owncloud.db "UPDATE oc_users_external SET backend='127.0.0.1';" || /bin/true
 
-# Set up a cron job for Nextcloud.
+# Set up a general cron job for Nextcloud.
+# Also add another job for Calendar updates, per advice in the Nextcloud docs
+# https://docs.nextcloud.com/server/24/admin_manual/groupware/calendar.html#background-jobs
 cat > /etc/cron.d/mailinabox-nextcloud << EOF;
 #!/bin/bash
 # Mail-in-a-Box
 */5 * * * *	root	sudo -u www-data php$PHP_VER -f /usr/local/lib/owncloud/cron.php
+*/5 * * * *	root	sudo -u www-data php$PHP_VER -f /usr/local/lib/owncloud/occ dav:send-event-reminders
 EOF
 chmod +x /etc/cron.d/mailinabox-nextcloud
 
+# We also need to change the sending mode from background-job to occ.
+# Or else the reminders will just be sent as soon as possible when the background jobs run.
+hide_output sudo -u www-data php$PHP_VER -f /usr/local/lib/owncloud/occ config:app:set dav sendEventRemindersMode --value occ
+
+# Now set the config to read-only.
+# Do this only at the very bottom when no further occ commands are needed.
+sed -i'' "s/'config_is_read_only'\s*=>\s*false/'config_is_read_only' => true/" $STORAGE_ROOT/owncloud/config.php
+
+# Rotate the nextcloud.log file
+cat > /etc/logrotate.d/nextcloud <<EOF
+# Nextcloud logs
+$STORAGE_ROOT/owncloud/nextcloud.log {
+		size 10M
+		create 640 www-data www-data
+		rotate 30
+		copytruncate
+		missingok
+		compress
+}
+EOF
+
 # There's nothing much of interest that a user could do as an admin for Nextcloud,
 # and there's a lot they could mess up, so we don't make any users admins of Nextcloud.
 # But if we wanted to, we would do this:

setup/questions.sh

@@ -207,6 +207,6 @@ if [ "$PRIVATE_IPV6" != "$PUBLIC_IPV6" ]; then
 	echo "Private IPv6 Address: $PRIVATE_IPV6"
 fi
 if [ -f /usr/bin/git ] && [ -d .git ]; then
-	echo "Mail-in-a-Box Version: " $(git describe)
+	echo "Mail-in-a-Box Version: " $(git describe --always)
 fi
 echo

setup/start.sh

@@ -85,7 +85,7 @@ f=$STORAGE_ROOT
 while [[ $f != / ]]; do chmod a+rx "$f"; f=$(dirname "$f"); done;
 if [ ! -f $STORAGE_ROOT/mailinabox.version ]; then
 	setup/migrate.py --current > $STORAGE_ROOT/mailinabox.version
-	chown $STORAGE_USER.$STORAGE_USER $STORAGE_ROOT/mailinabox.version
+	chown $STORAGE_USER:$STORAGE_USER $STORAGE_ROOT/mailinabox.version
 fi
 
 # Save the global options in /etc/mailinabox.conf so that standalone
@@ -167,7 +167,7 @@ if management/status_checks.py --check-primary-hostname; then
 	echo "If you have a DNS problem put the box's IP address in the URL"
 	echo "(https://$PUBLIC_IP/admin) but then check the TLS fingerprint:"
 	openssl x509 -in $STORAGE_ROOT/ssl/ssl_certificate.pem -noout -fingerprint -sha256\
-        	| sed "s/SHA256 Fingerprint=//"
+        	| sed "s/SHA256 Fingerprint=//i"
 else
 	echo https://$PUBLIC_IP/admin
 	echo
@@ -175,7 +175,7 @@ else
 	echo the certificate fingerprint matches:
 	echo
 	openssl x509 -in $STORAGE_ROOT/ssl/ssl_certificate.pem -noout -fingerprint -sha256\
-        	| sed "s/SHA256 Fingerprint=//"
+        	| sed "s/SHA256 Fingerprint=//i"
 	echo
 	echo Then you can confirm the security exception and continue.
 	echo

setup/system.sh

@@ -373,3 +373,5 @@ cp -f conf/fail2ban/filter.d/* /etc/fail2ban/filter.d/
 # scripts will ensure the files exist and then fail2ban is given another
 # restart at the very end of setup.
 restart_service fail2ban
+
+systemctl enable fail2ban

setup/webmail.sh

@@ -23,7 +23,8 @@ echo "Installing Roundcube (webmail)..."
 apt_install \
 	dbconfig-common \
 	php${PHP_VER}-cli php${PHP_VER}-sqlite3 php${PHP_VER}-intl php${PHP_VER}-common php${PHP_VER}-curl php${PHP_VER}-imap \
-	php${PHP_VER}-gd php${PHP_VER}-pspell php${PHP_VER}-mbstring libjs-jquery libjs-jquery-mousewheel libmagic1
+	php${PHP_VER}-gd php${PHP_VER}-pspell php${PHP_VER}-mbstring libjs-jquery libjs-jquery-mousewheel libmagic1 \
+	sqlite3
 
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of plugins to track
@@ -35,9 +36,9 @@ apt_install \
 #   https://github.com/mstilkerich/rcmcarddav/releases
 # The easiest way to get the package hashes is to run this script and get the hash from
 # the error message.
-VERSION=1.6.0
+VERSION=1.6.4
-HASH=fd84b4fac74419bb73e7a3bcae1978d5589c52de
+HASH=bfc693d6590542d63171e6a3997fc29f0a5f12ca
-PERSISTENT_LOGIN_VERSION=bde7b6840c7d91de627ea14e81cf4133cbb3c07a # version 5.2
+PERSISTENT_LOGIN_VERSION=bde7b6840c7d91de627ea14e81cf4133cbb3c07a # version 5.3
 HTML5_NOTIFIER_VERSION=68d9ca194212e15b3c7225eb6085dbcf02fd13d7 # version 0.6.4+
 CARDDAV_VERSION=4.4.3
 CARDDAV_HASH=74f8ba7aee33e78beb9de07f7f44b81f6071b644
@@ -170,7 +171,7 @@ EOF
 
 # Create writable directories.
 mkdir -p /var/log/roundcubemail /var/tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
-chown -R www-data.www-data /var/log/roundcubemail /var/tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
+chown -R www-data:www-data /var/log/roundcubemail /var/tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
 
 # Ensure the log file monitored by fail2ban exists, or else fail2ban can't start.
 sudo -u www-data touch /var/log/roundcubemail/errors.log
@@ -184,24 +185,23 @@ cp ${RCM_PLUGIN_DIR}/password/config.inc.php.dist \
 tools/editconf.py ${RCM_PLUGIN_DIR}/password/config.inc.php \
 	"\$config['password_minimum_length']=8;" \
 	"\$config['password_db_dsn']='sqlite:///$STORAGE_ROOT/mail/users.sqlite';" \
-	"\$config['password_query']='UPDATE users SET password=%D WHERE email=%u';" \
+	"\$config['password_query']='UPDATE users SET password=%P WHERE email=%u';" \
-	"\$config['password_dovecotpw']='/usr/bin/doveadm pw';" \
+	"\$config['password_algorithm']='sha512-crypt';" \
-	"\$config['password_dovecotpw_method']='SHA512-CRYPT';" \
+	"\$config['password_algorithm_prefix']='{SHA512-CRYPT}';"
-	"\$config['password_dovecotpw_with_method']=true;"
 
 # so PHP can use doveadm, for the password changing plugin
 usermod -a -G dovecot www-data
 
 # set permissions so that PHP can use users.sqlite
 # could use dovecot instead of www-data, but not sure it matters
-chown root.www-data $STORAGE_ROOT/mail
+chown root:www-data $STORAGE_ROOT/mail
 chmod 775 $STORAGE_ROOT/mail
-chown root.www-data $STORAGE_ROOT/mail/users.sqlite
+chown root:www-data $STORAGE_ROOT/mail/users.sqlite
 chmod 664 $STORAGE_ROOT/mail/users.sqlite
 
 # Fix Carddav permissions:
-chown -f -R root.www-data ${RCM_PLUGIN_DIR}/carddav
+chown -f -R root:www-data ${RCM_PLUGIN_DIR}/carddav
-# root.www-data need all permissions, others only read
+# root:www-data need all permissions, others only read
 chmod -R 774 ${RCM_PLUGIN_DIR}/carddav
 
 # Run Roundcube database migration script (database is created if it does not exist)
@@ -209,6 +209,16 @@ php$PHP_VER ${RCM_DIR}/bin/updatedb.sh --dir ${RCM_DIR}/SQL --package roundcube
 chown www-data:www-data $STORAGE_ROOT/mail/roundcube/roundcube.sqlite
 chmod 664 $STORAGE_ROOT/mail/roundcube/roundcube.sqlite
 
+# Patch the Roundcube code to eliminate an issue that causes postfix to reject our sqlite
+# user database (see https://github.com/mail-in-a-box/mailinabox/issues/2185)
+sed -i.miabold 's/^[^#]\+.\+PRAGMA journal_mode = WAL.\+$/#&/' \
+/usr/local/lib/roundcubemail/program/lib/Roundcube/db/sqlite.php
+
+# Because Roundcube wants to set the PRAGMA we just deleted from the source, we apply it here
+# to the roundcube database (see https://github.com/roundcube/roundcubemail/issues/8035)
+# Database should exist, created by migration script
+sqlite3 $STORAGE_ROOT/mail/roundcube/roundcube.sqlite 'PRAGMA journal_mode=WAL;'
+
 # Enable PHP modules.
 phpenmod -v $PHP_VER imap
 restart_service php$PHP_VER-fpm

setup/zpush.sh

@@ -22,8 +22,8 @@ apt_install \
 phpenmod -v $PHP_VER imap
 
 # Copy Z-Push into place.
-VERSION=2.6.2
+VERSION=2.7.1
-TARGETHASH=f0e8091a8030e5b851f5ba1f9f0e1a05b8762d80
+TARGETHASH=f15c566b1ad50de24f3f08f505f0c3d8155c2d0d
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/z-push/version ]; then
 	needs_update=1 #NODOC
@@ -41,7 +41,15 @@ if [ $needs_update == 1 ]; then
 	mv /tmp/z-push/*/src /usr/local/lib/z-push
 	rm -rf /tmp/z-push.zip /tmp/z-push
 
+	# Create admin and top scripts with PHP_VER  
 	rm -f /usr/sbin/z-push-{admin,top}
+    echo '#!/bin/bash' > /usr/sbin/z-push-admin
+    echo php$PHP_VER /usr/local/lib/z-push/z-push-admin.php '"$@"' >> /usr/sbin/z-push-admin
+    chmod 755 /usr/sbin/z-push-admin
+    echo '#!/bin/bash' > /usr/sbin/z-push-top
+    echo php$PHP_VER /usr/local/lib/z-push/z-push-top.php '"$@"' >> /usr/sbin/z-push-top
+    chmod 755 /usr/sbin/z-push-top
+	
 	echo $VERSION > /usr/local/lib/z-push/version
 fi
 

tools/editconf.py

@@ -76,7 +76,8 @@ for setting in settings:
 
 found = set()
 buf = ""
-input_lines = list(open(filename))
+with open(filename, "r") as f:
+        input_lines = list(f)
 
 while len(input_lines) > 0:
 	line = input_lines.pop(0)

tools/owncloud-restore.sh

@@ -40,8 +40,8 @@ cp "$1/owncloud.db" $STORAGE_ROOT/owncloud/
 cp "$1/config.php" $STORAGE_ROOT/owncloud/
 
 ln -sf $STORAGE_ROOT/owncloud/config.php /usr/local/lib/owncloud/config/config.php
-chown -f -R www-data.www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
+chown -f -R www-data:www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
-chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
+chown www-data:www-data $STORAGE_ROOT/owncloud/config.php
 
 sudo -u www-data php$PHP_VER /usr/local/lib/owncloud/occ maintenance:mode --off
 

tools/parse-nginx-log-bootstrap-accesses.py

@@ -17,13 +17,8 @@ accesses = set()
 # Scan the current and rotated access logs.
 for fn in glob.glob("/var/log/nginx/access.log*"):
 	# Gunzip if necessary.
-	if fn.endswith(".gz"):
-		f = gzip.open(fn)
-	else:
-		f = open(fn, "rb")
-
 	# Loop through the lines in the access log.
-	with f:
+	with (gzip.open if fn.endswith(".gz") else open)(fn, "rb") as f:
 		for line in f:
 			# Find lines that are GETs on the bootstrap script by either curl or wget.
 			# (Note that we purposely skip ...?ping=1 requests which is the admin panel querying us for updates.)
@@ -43,7 +38,8 @@ for date, ip in accesses:
 # Since logs are rotated, store the statistics permanently in a JSON file.
 # Load in the stats from an existing file.
 if os.path.exists(outfn):
-	existing_data = json.load(open(outfn))
+	with open(outfn, "r") as f:
+		existing_data = json.load(f)
 	for date, count in existing_data:
 		if date not in by_date:
 			by_date[date] = count

tools/readable_bash.py

@@ -124,13 +124,14 @@ def generate_documentation():
  """)
 
 	parser = Source.parser()
-	for line in open("setup/start.sh"):
+	with open("setup/start.sh", "r") as start_file: 
-		try:
+                for line in start_file:
-			fn = parser.parse_string(line).filename()
+                        try:
-		except:
+                                fn = parser.parse_string(line).filename()
-			continue
+                        except:
-		if fn in ("setup/start.sh", "setup/preflight.sh", "setup/questions.sh", "setup/firstuser.sh", "setup/management.sh"):
+                                continue
-			continue
+                        if fn in ("setup/start.sh", "setup/preflight.sh", "setup/questions.sh", "setup/firstuser.sh", "setup/management.sh"):
+                                continue
 
 		import sys
 		print(fn, file=sys.stderr)
@@ -401,7 +402,8 @@ class BashScript(Grammar):
 	@staticmethod
 	def parse(fn):
 		if fn in ("setup/functions.sh", "/etc/mailinabox.conf"): return ""
-		string = open(fn).read()
+		with open(fn, "r") as f:
+			string = f.read()
 
 		# tokenize
 		string = re.sub(".* #NODOC\n", "", string)