v0.48

Merges #1809

.editorconfig

@@ -13,7 +13,7 @@ trim_trailing_whitespace = true
 insert_final_newline = true
 
 [Makefile]
-indent_style = tabs
+indent_style = tab
 indent_size = 4
 
 [Vagrantfile]
@@ -23,7 +23,7 @@ indent_size = 2
 indent_size = 2
 
 [*.py]
-indent_style = tabs
+indent_style = tab
 
 [*.js]
 indent_size = 2

CHANGELOG.md

@@ -1,9 +1,115 @@
 CHANGELOG
 =========
 
-v0.42 (July 4, 2019)
+v0.48 (August 26, 2020)
+-----------------------
+
+Security fixes:
+
+* Roundcube is updated to version 1.4.8 fixing additional cross-site scripting (XSS) vulnerabilities.
+
+v0.47 (July 29, 2020)
+---------------------
+
+Security fixes:
+
+* Roundcube is updated to version 1.4.7 fixing a cross-site scripting (XSS) vulnerability with HTML messages with malicious svg/namespace (CVE-2020-15562) (https://roundcube.net/news/2020/07/05/security-updates-1.4.7-1.3.14-and-1.2.11).
+* SSH connections are now rate-limited at the firewall level (in addition to fail2ban).
+
+v0.46 (June 11, 2020)
+---------------------
+
+Security fixes:
+
+* Roundcube is updated to version 1.4.6 (https://roundcube.net/news/2020/06/02/security-updates-1.4.5-and-1.3.12).
+
+v0.45 (May 16, 2020)
 --------------------
 
+Security fixes:
+
+* Fix missing brute force login protection for Roundcube logins.
+
+Software updates:
+
+* Upgraded Roundcube from 1.4.2 to 1.4.4.
+* Upgraded Nextcloud from 17.0.2 to 17.0.6 (with Contacts from 3.1.6 to 3.3.0 and Calendar from 1.7.1 to v2.0.3)
+* Upgraded Z-Push to 2.5.2.
+
+System:
+
+* Nightly backups now occur on a random minute in the 3am hour (in the system time zone). The minute is chosen during Mail-in-a-Box installation/upgrade and remains the same until the next upgrade.
+* Fix for mail log statistics report on leap days.
+* Fix Mozilla autoconfig useGlobalPreferredServer setting.
+
+Web:
+
+* Add a new hidden feature to set nginx alias in www/custom.yaml.
+
+Setup:
+
+* Improved error handling.
+
+v0.44 (February 15, 2020)
+-------------------------
+
+System:
+
+* TLS settings have been upgraded following Mozilla's recommendations for servers. TLS1.2 and 1.3 are now the only supported protocols for web, IMAP, and SMTP (submission).
+* Fixed an issue starting services when Mail-in-a-Box isn't on the root filesystem.
+* Changed some performance options affecting Roundcube and Nextcloud.
+
+Software updates:
+
+* Upgraded Nextcloud from 15.0.8 to 17.0.2 (with Contacts from 3.1.1 to 3.1.6 and Calendar from 1.6.5 to 1.7.1)
+* Upgraded Z-Push to 2.5.1.
+* Upgraded Roundcube from 1.3.10 to 1.4.2 and changed the default skin (theme) to Elastic.
+
+Control panel:
+
+* The Custom DNS list of records is now sorted.
+* The emails that report TLS provisioning results now has a less scary subject line.
+
+Mail:
+
+* Fetching of updated whitelist for greylisting was fetching each day instead of every month.
+* OpenDKIM signing has been changed to 'relaxed' mode so that some old mail lists that forward mail can do so.
+
+DNS:
+
+* Automatic autoconfig.* subdomains can now be suppressed with custom DNS records.
+* DNS zone transfer now works with IPv6 addresses.
+
+Setup:
+
+* An Ubuntu package source was missing on systems where it defaults off.
+
+v0.43 (September 1, 2019)
+-------------------------
+
+Security fixes:
+
+* A security issue was discovered in rsync backups. If you have enabled rsync backups, the file `id_rsa_miab` may have been copied to your backup destination. This file can be used to access your backup destination. If the file was copied to your backup destination, we recommend that you delete the file on your backup destination, delete `/root/.ssh/id_rsa_miab` on your Mail-in-a-Box, then re-run Mail-in-a-Box setup, and re-configure your SSH public key at your backup destination according to the instructions in the Mail-in-a-Box control panel.
+* Brute force attack prevention was missing for the managesieve service.
+
+Setup:
+
+* Nextcloud was not upgraded properly after restoring Mail-in-a-Box from a backup from v0.40 or earlier.
+
+Mail:
+
+* Upgraded Roundcube to 1.3.10.
+* Fetch an updated whitelist for greylisting on a monthly basis to reduce the number of delayed incoming emails.
+
+Control panel:
+
+* When using secondary DNS, it is now possible to specify a subnet range with the `xfr:` option.
+* Fixed an issue when the secondary DNS option is used and the secondary DNS hostname resolves to multiple IP addresses.
+* Fix a bug in how a backup configuration error is shown.
+
+v0.42b (August 3, 2019)
+-----------------------
+
 Changes:
 
 * Decreased the minimum supported RAM to 502 Mb.
@@ -19,6 +125,8 @@ Software updates:
 * Upgraded Nextcloud from 14.0.6 to 15.0.8 (with Contacts from 2.1.8 to 3.1.1 and Calendar from 1.6.4 to 1.6.5).
 * Upgraded Z-Push from 2.4.4 to 2.5.0.
 
+Note that v0.42 (July 4, 2019) was pulled shortly after it was released to fix a Nextcloud upgrade issue.
+
 v0.41 (February 26, 2019)
 -------------------------
 

README.md

@@ -13,7 +13,7 @@ Our goals are to:
 
 * Make deploying a good mail server easy.
 * Promote [decentralization](http://redecentralize.org/), innovation, and privacy on the web.
-* Have automated, auditable, and [idempotent](https://sharknet.us/2014/02/01/automated-configuration-management-challenges-with-idempotency/) configuration.
+* Have automated, auditable, and [idempotent](https://web.archive.org/web/20190518072631/https://sharknet.us/2014/02/01/automated-configuration-management-challenges-with-idempotency/) configuration.
 * **Not** make a totally unhackable, NSA-proof server.
 * **Not** make something customizable by power users.
 
@@ -58,7 +58,7 @@ by him:
 	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
 	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported
 
-	$ git verify-tag v0.42
+	$ git verify-tag v0.48
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
@@ -71,7 +71,7 @@ and on his [personal homepage](https://razor.occams.info/). (Of course, if this
 
 Checkout the tag corresponding to the most recent release:
 
-	$ git checkout v0.42
+	$ git checkout v0.48
 
 Begin the installation.
 

conf/fail2ban/filter.d/dovecotimap.conf

@@ -1,4 +1,4 @@
-# Fail2Ban filter Dovecot authentication and pop3/imap server
+# Fail2Ban filter Dovecot authentication and pop3/imap/managesieve server
 # For Mail-in-a-Box
 
 [INCLUDES]
@@ -9,7 +9,7 @@ before = common.conf
 
 _daemon = (auth|dovecot(-auth)?|auth-worker)
 
-failregex = ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((no auth attempts|auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>, lip=(\d{1,3}\.){3}\d{1,3}(, TLS( handshaking)?(: Disconnected)?)?(, session=<\S+>)?\s*$
+failregex = ^%(__prefix_line)s(pop3|imap|managesieve)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((no auth attempts|auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>, lip=(\d{1,3}\.){3}\d{1,3}(, TLS( handshaking)?(: Disconnected)?)?(, session=<\S+>)?\s*$
 
 ignoreregex =
 

conf/fail2ban/jails.conf

@@ -50,7 +50,7 @@ findtime = 30
 enabled  = true
 port     = http,https
 filter   = miab-roundcube
-logpath  = /var/log/roundcubemail/errors
+logpath  = /var/log/roundcubemail/errors.log
 maxretry = 20
 findtime = 30
 

conf/mozilla-autoconfig.xml

@@ -21,7 +21,7 @@
          <username>%EMAILADDRESS%</username>
          <authentication>password-cleartext</authentication>
          <addThisServer>true</addThisServer>
-         <useGlobalPreferredServer>true</useGlobalPreferredServer>
+         <useGlobalPreferredServer>false</useGlobalPreferredServer>
       </outgoingServer>
 
       <documentation url="https://PRIMARY_HOSTNAME/">

conf/nginx-ssl.conf

@@ -1,7 +1,7 @@
 # We track the Mozilla "intermediate" compatibility TLS recommendations.
 # Note that these settings are repeated in the SMTP and IMAP configuration.
 # ssl_protocols has moved to nginx.conf in bionic, check there for enabled protocols.
-ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
+ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
 ssl_dhparam STORAGE_ROOT/ssl/dh2048.pem;
 
 # as recommended by http://nginx.org/en/docs/http/configuring_https_servers.html

management/backup.py

@@ -15,8 +15,8 @@ from exclusiveprocess import Lock
 from utils import load_environment, shell, wait_for_service, fix_boto
 
 rsync_ssh_options = [
-	"--ssh-options='-i /root/.ssh/id_rsa_miab'",
-	"--rsync-options=-e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab\"",
+	"--ssh-options= -i /root/.ssh/id_rsa_miab",
+	"--rsync-options= -e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab\"",
 ]
 
 def backup_status(env):
@@ -406,11 +406,11 @@ def list_target_files(config):
 				reason = "Provided path {} is invalid.".format(target_path)
 			elif 'Network is unreachable' in listing:
 				reason = "The IP address {} is unreachable.".format(target.hostname)
-			elif 'Could not resolve hostname':
+			elif 'Could not resolve hostname' in listing:
 				reason = "The hostname {} cannot be resolved.".format(target.hostname)
 			else:
 				reason = "Unknown error." \
-						"Please check running 'python management/backup.py --verify'" \
+						"Please check running 'management/backup.py --verify'" \
 						"from mailinabox sources to debug the issue."
 			raise ValueError("Connection to rsync host failed: {}".format(reason))
 

management/daily_tasks.sh

@@ -19,7 +19,7 @@ fi
 management/backup.py | management/email_administrator.py "Backup Status"
 
 # Provision any new certificates for new domains or domains with expiring certificates.
-management/ssl_certificates.py -q | management/email_administrator.py "Error Provisioning TLS Certificate"
+management/ssl_certificates.py -q | management/email_administrator.py "TLS Certificate Provisioning Result"
 
 # Run status checks and email the administrator if anything changed.
 management/status_checks.py --show-changes | management/email_administrator.py "Status Checks Change Notice"

management/dns_update.py

@@ -292,10 +292,16 @@ def build_zone(domain, all_domains, additional_records, www_redirect_domains, en
 	# This allows the following clients to automatically configure email addresses in the respective applications.
 	# autodiscover.* - Z-Push ActiveSync Autodiscover
 	# autoconfig.* - Thunderbird Autoconfig
-	if not has_rec("autodiscover", "A"):
-		records.append(("autodiscover", "A", env["PUBLIC_IP"], "Provides email configuration autodiscovery support for Z-Push ActiveSync Autodiscover."))
-	if not has_rec("autoconfig", "A"):
-		records.append(("autoconfig", "A", env["PUBLIC_IP"], "Provides email configuration autodiscovery support for Thunderbird Autoconfig."))
+	autodiscover_records = [
+		("autodiscover", "A", env["PUBLIC_IP"], "Provides email configuration autodiscovery support for Z-Push ActiveSync Autodiscover."),
+		("autodiscover", "AAAA", env["PUBLIC_IPV6"], "Provides email configuration autodiscovery support for Z-Push ActiveSync Autodiscover."),
+		("autoconfig", "A", env["PUBLIC_IP"], "Provides email configuration autodiscovery support for Thunderbird Autoconfig."),
+		("autoconfig", "AAAA", env["PUBLIC_IPV6"], "Provides email configuration autodiscovery support for Thunderbird Autoconfig.")
+	]
+	for qname, rtype, value, explanation in autodiscover_records:
+		if value is None or value.strip() == "": continue # skip IPV6 if not set
+		if not has_rec(qname, rtype):
+			records.append((qname, rtype, value, explanation))
 
 	# Sort the records. The None records *must* go first in the nsd zone file. Otherwise it doesn't matter.
 	records.sort(key = lambda rec : list(reversed(rec[0].split(".")) if rec[0] is not None else ""))
@@ -517,9 +523,11 @@ zone:
 """ % (domain, zonefile)
 
 		# If custom secondary nameservers have been set, allow zone transfers
-		# and notifies to them.
+		# and, if not a subnet, notifies to them.
 		for ipaddr in get_secondary_dns(additional_records, mode="xfr"):
-			nsdconf += "\n\tnotify: %s NOKEY\n\tprovide-xfr: %s NOKEY\n" % (ipaddr, ipaddr)
+			if "/" not in ipaddr:
+				nsdconf += "\n\tnotify: %s NOKEY" % (ipaddr)
+			nsdconf += "\n\tprovide-xfr: %s NOKEY\n" % (ipaddr)
 
 	# Check if the file is changing. If it isn't changing,
 	# return False to flag that no change was made.
@@ -867,10 +875,15 @@ def get_secondary_dns(custom_dns, mode=None):
 
 			# This is a hostname. Before including in zone xfr lines,
 			# resolve to an IP address. Otherwise just return the hostname.
+			# It may not resolve to IPv6, so don't throw an exception if it
+			# doesn't.
 			if not hostname.startswith("xfr:"):
 				if mode == "xfr":
-					response = dns.resolver.query(hostname+'.', "A")
-					hostname = str(response[0])
+					response = dns.resolver.query(hostname+'.', "A", raise_on_no_answer=False)
+					values.extend(map(str, response))
+					response = dns.resolver.query(hostname+'.', "AAAA", raise_on_no_answer=False)
+					values.extend(map(str, response))
+					continue
 				values.append(hostname)
 
 			# This is a zone-xfer-only IP address. Do not return if
@@ -897,10 +910,14 @@ def set_secondary_dns(hostnames, env):
 			else:
 				# Validate IP address.
 				try:
+					if "/" in item[4:]:
+						v = ipaddress.ip_network(item[4:]) # raises a ValueError if there's a problem
+						if not isinstance(v, ipaddress.IPv4Network): raise ValueError("That's an IPv6 subnet.")
+					else:
 						v = ipaddress.ip_address(item[4:]) # raises a ValueError if there's a problem
 						if not isinstance(v, ipaddress.IPv4Address): raise ValueError("That's an IPv6 address.")
 				except ValueError:
-					raise ValueError("'%s' is not an IPv4 address." % item[4:])
+					raise ValueError("'%s' is not an IPv4 address or subnet." % item[4:])
 
 		# Set.
 		set_custom_dns_record("_secondary_nameserver", "A", " ".join(hostnames), "set", env)

management/mail_log.py

@@ -18,13 +18,13 @@ import utils
 
 
 LOG_FILES = (
-    '/var/log/mail.log',
-    '/var/log/mail.log.1',
-    '/var/log/mail.log.2.gz',
-    '/var/log/mail.log.3.gz',
-    '/var/log/mail.log.4.gz',
-    '/var/log/mail.log.5.gz',
     '/var/log/mail.log.6.gz',
+    '/var/log/mail.log.5.gz',
+    '/var/log/mail.log.4.gz',
+    '/var/log/mail.log.3.gz',
+    '/var/log/mail.log.2.gz',
+    '/var/log/mail.log.1',
+    '/var/log/mail.log',
 )
 
 TIME_DELTAS = OrderedDict([
@@ -80,7 +80,7 @@ def scan_files(collector):
             print("Processing file", fn, "...")
         fn = tmp_file.name if tmp_file else fn
 
-        for line in reverse_readline(fn):
+        for line in readline(fn):
             if scan_mail_log_line(line.strip(), collector) is False:
                 if stop_scan:
                     return
@@ -344,16 +344,22 @@ def scan_mail_log_line(line, collector):
 
     # Replaced the dateutil parser for a less clever way of parser that is roughly 4 times faster.
     # date = dateutil.parser.parse(date)
-    date = datetime.datetime.strptime(date, '%b %d %H:%M:%S')
-    date = date.replace(START_DATE.year)
+
+    # date = datetime.datetime.strptime(date, '%b %d %H:%M:%S')
+    # date = date.replace(START_DATE.year)
+
+    # strptime fails on Feb 29 if correct year is not provided. See https://bugs.python.org/issue26460
+    date = datetime.datetime.strptime(str(START_DATE.year) + ' ' + date, '%Y %b %d %H:%M:%S')
+    # print("date:", date)
 
     # Check if the found date is within the time span we are scanning
+    # END_DATE < START_DATE
     if date > START_DATE:
-        # Don't process, but continue
-        return True
-    elif date < END_DATE:
         # Don't process, and halt
         return False
+    elif date < END_DATE:
+        # Don't process, but continue
+        return True
 
     if service == "postfix/submission/smtpd":
         if SCAN_OUT:
@@ -453,9 +459,9 @@ def scan_postfix_smtpd_line(date, log, collector):
                     if m:
                         message = "domain blocked: " + m.group(2)
 
-                if data["latest"] is None:
-                    data["latest"] = date
+                if data["earliest"] is None:
                     data["earliest"] = date
+                data["latest"] = date
                 data["blocked"].append((date, sender, message))
 
                 collector["rejected"][user] = data
@@ -487,9 +493,9 @@ def add_login(user, date, protocol_name, host, collector):
                 }
             )
 
-            if data["latest"] is None:
-                data["latest"] = date
+            if data["earliest"] is None:
                 data["earliest"] = date
+            data["latest"] = date
 
             data["totals_by_protocol"][protocol_name] += 1
             data["totals_by_protocol_and_host"][(protocol_name, host)] += 1
@@ -528,9 +534,9 @@ def scan_postfix_lmtp_line(date, log, collector):
             data["received_count"] += 1
             data["activity-by-hour"][date.hour] += 1
 
-            if data["latest"] is None:
-                data["latest"] = date
+            if data["earliest"] is None:
                 data["earliest"] = date
+            data["latest"] = date
 
             collector["received_mail"][user] = data
 
@@ -567,9 +573,9 @@ def scan_postfix_submission_line(date, log, collector):
             data["hosts"].add(client)
             data["activity-by-hour"][date.hour] += 1
 
-            if data["latest"] is None:
-                data["latest"] = date
+            if data["earliest"] is None:
                 data["earliest"] = date
+            data["latest"] = date
 
             collector["sent_mail"][user] = data
 
@@ -578,42 +584,15 @@ def scan_postfix_submission_line(date, log, collector):
 
 # Utility functions
 
-def reverse_readline(filename, buf_size=8192):
-    """ A generator that returns the lines of a file in reverse order
-
-    http://stackoverflow.com/a/23646049/801870
-
+def readline(filename):
+    """ A generator that returns the lines of a file
     """
-
-    with open(filename) as fh:
-        segment = None
-        offset = 0
-        fh.seek(0, os.SEEK_END)
-        file_size = remaining_size = fh.tell()
-        while remaining_size > 0:
-            offset = min(file_size, offset + buf_size)
-            fh.seek(file_size - offset)
-            buff = fh.read(min(remaining_size, buf_size))
-            remaining_size -= buf_size
-            lines = buff.split('\n')
-            # the first line of the buffer is probably not a complete line so
-            # we'll save it and append it to the last line of the next buffer
-            # we read
-            if segment is not None:
-                # if the previous chunk starts right from the beginning of line
-                # do not concat the segment to the last line of new chunk
-                # instead, yield the segment first
-                if buff[-1] is not '\n':
-                    lines[-1] += segment
-                else:
-                    yield segment
-            segment = lines[0]
-            for index in range(len(lines) - 1, 0, -1):
-                if len(lines[index]):
-                    yield lines[index]
-        # Don't yield None if the file was empty
-        if segment is not None:
-            yield segment
+    with open(filename) as file:
+        while True:
+          line = file.readline()
+          if not line:
+              break
+          yield line
 
 
 def user_match(user):

management/status_checks.py

@@ -486,10 +486,12 @@ def check_dns_zone(domain, env, output, dns_zonefiles):
 	if custom_secondary_ns and not probably_external_dns:
 		for ns in custom_secondary_ns:
 			# We must first resolve the nameserver to an IP address so we can query it.
-			ns_ip = query_dns(ns, "A")
-			if not ns_ip:
+			ns_ips = query_dns(ns, "A")
+			if not ns_ips:
 				output.print_error("Secondary nameserver %s is not valid (it doesn't resolve to an IP address)." % ns)
 				continue
+			# Choose the first IP if nameserver returns multiple
+			ns_ip = ns_ips.split('; ')[0]
 
 			# Now query it to see what it says about this domain.
 			ip = query_dns(domain, "A", at=ns_ip, nxdomain=None)

management/templates/custom-dns.html

@@ -90,7 +90,7 @@
     <div class="col-sm-offset-1 col-sm-11">
       <p class="small">
         Multiple secondary servers can be separated with commas or spaces (i.e., <code>ns2.hostingcompany.com ns3.hostingcompany.com</code>). 
-        To enable zone transfers to additional servers without listing them as secondary nameservers, add <code>xfr:IPADDRESS</code>.
+        To enable zone transfers to additional servers without listing them as secondary nameservers, add an IP address or subnet using <code>xfr:10.20.30.40</code> or <code>xfr:10.20.30.40/24</code>.
       </p>
       <p id="secondarydns-clear-instructions" style="display: none" class="small">
         Clear the input field above and click Update to use this machine itself as secondary DNS, which is the default/normal setup.
@@ -193,6 +193,22 @@ function show_current_custom_dns() {
       else
         $('#custom-dns-current').fadeOut();
         
+      var reverse_fqdn = function(el) {
+        el.qname = el.qname.split('.').reverse().join('.');
+        return el;
+      }
+      var sort = function(a, b) {
+        if(a.qname === b.qname) {
+          if(a.rtype === b.rtype) {
+            return a.value > b.value ? 1 : -1;
+          }
+          return a.rtype > b.rtype ? 1 : -1;
+        }
+        return a.qname > b.qname ? 1 : -1;
+      }
+
+      data = data.map(reverse_fqdn).sort(sort).map(reverse_fqdn);
+
       $('#custom-dns-current').find("tbody").text('');
       for (var i = 0; i < data.length; i++) {
         var tr = $("<tr/>");

management/templates/users.html

@@ -99,7 +99,7 @@
 <thead><th>Verb</th> <th>Action</th><th></th></thead>
 <tr><td>GET</td><td><i>(none)</i></td> <td>Returns a list of existing mail users. Adding <code>?format=json</code> to the URL will give JSON-encoded results.</td></tr>
 <tr><td>POST</td><td>/add</td> <td>Adds a new mail user. Required POST-body parameters are <code>email</code> and <code>password</code>.</td></tr>
-<tr><td>POST</td><td>/remove</td> <td>Removes a mail user. Required POST-by parameter is <code>email</code>.</td></tr>
+<tr><td>POST</td><td>/remove</td> <td>Removes a mail user. Required POST-body parameter is <code>email</code>.</td></tr>
 <tr><td>POST</td><td>/privileges/add</td> <td>Used to make a mail user an admin. Required POST-body parameters are <code>email</code> and <code>privilege=admin</code>.</td></tr>
 <tr><td>POST</td><td>/privileges/remove</td> <td>Used to remove the admin privilege from a mail user. Required POST-body parameter is <code>email</code>.</td></tr>
 </table>

management/web_update.py

@@ -24,17 +24,17 @@ def get_web_domains(env, include_www_redirects=True, exclude_dns_elsewhere=True)
 		# the topmost of each domain we serve.
 		domains |= set('www.' + zone for zone, zonefile in get_dns_zones(env))
 
-	if exclude_dns_elsewhere:
-		# ...Unless the domain has an A/AAAA record that maps it to a different
-		# IP address than this box. Remove those domains from our list.
-		domains -= get_domains_with_a_records(env)
-
 	# Add Autoconfiguration domains, allowing us to serve correct SSL certs.
 	# 'autoconfig.' for Mozilla Thunderbird auto setup.
 	# 'autodiscover.' for Activesync autodiscovery.
 	domains |= set('autoconfig.' + maildomain for maildomain in get_mail_domains(env))
 	domains |= set('autodiscover.' + maildomain for maildomain in get_mail_domains(env))
 
+	if exclude_dns_elsewhere:
+		# ...Unless the domain has an A/AAAA record that maps it to a different
+		# IP address than this box. Remove those domains from our list.
+		domains -= get_domains_with_a_records(env)
+
 	# Ensure the PRIMARY_HOSTNAME is in the list so we can serve webmail
 	# as well as Z-Push for Exchange ActiveSync. This can't be removed
 	# by a custom A/AAAA record and is never a 'www.' redirect.
@@ -159,6 +159,10 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 				nginx_conf_extra += "\n\t\tproxy_pass %s;" % url
 				nginx_conf_extra += "\n\t\tproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;"
 				nginx_conf_extra += "\n\t}\n"
+			for path, alias in yaml.get("aliases", {}).items():
+				nginx_conf_extra += "\tlocation %s {" % path
+				nginx_conf_extra += "\n\t\talias %s;" % alias
+				nginx_conf_extra += "\n\t}\n"
 			for path, url in yaml.get("redirects", {}).items():
 				nginx_conf_extra += "\trewrite %s %s permanent;\n" % (path, url)
 

security.md

@@ -39,9 +39,8 @@ These services are protected by [TLS](https://en.wikipedia.org/wiki/Transport_La
 The services all follow these rules:
 
 * TLS certificates are generated with 2048-bit RSA keys and SHA-256 fingerprints. The box provides a self-signed certificate by default. The [setup guide](https://mailinabox.email/guide.html) explains how to verify the certificate fingerprint on first login. Users are encouraged to replace the certificate with a proper CA-signed one. ([source](setup/ssl.sh))
-* Only TLSv1, TLSv1.1 and TLSv1.2 are offered (the older SSL protocols are not offered).
-* HTTPS, IMAP, and POP track the [Mozilla Intermediate Ciphers Recommendation](https://wiki.mozilla.org/Security/Server_Side_TLS), balancing security with supporting a wide range of mail clients. Diffie-Hellman ciphers use a 2048-bit key for forward secrecy. For more details, see the [output of SSLyze for these ports](tests/tls_results.txt).
-* SMTP (port 25) uses the Postfix medium grade ciphers and SMTP Submission (port 587) uses the Postfix high grade ciphers ([more info](http://www.postfix.org/postconf.5.html#smtpd_tls_mandatory_ciphers)).
+* Only TLSv1.2+ are offered (the older SSL protocols are not offered).
+* We track the [Mozilla Intermediate Ciphers Recommendation](https://wiki.mozilla.org/Security/Server_Side_TLS), balancing security with supporting a wide range of mail clients. Diffie-Hellman ciphers use a 2048-bit key for forward secrecy. For more details, see the [output of SSLyze for these ports](tests/tls_results.txt).
 
 Additionally:
 
@@ -95,7 +94,7 @@ Domain policy records allow recipient MTAs to detect when the _domain_ part of o
 
 ### User Policy
 
-While domain policy records prevent other servers from sending mail with a "From:" header that matches a domain hosted on the box (see above), those policy records do not guarnatee that the user portion of the sender email address matches the actual sender. In enterprise environments where the box may host the mail of untrusted users, it is important to guard against users impersonating other users.
+While domain policy records prevent other servers from sending mail with a "From:" header that matches a domain hosted on the box (see above), those policy records do not guarantee that the user portion of the sender email address matches the actual sender. In enterprise environments where the box may host the mail of untrusted users, it is important to guard against users impersonating other users.
 
 The box restricts the envelope sender address (also called the return path or MAIL FROM address --- this is different from the "From:" header) that users may put into outbound mail. The envelope sender address must be either their own email address (their SMTP login username) or any alias that they are listed as a permitted sender of. (There is currently no restriction on the contents of the "From:" header.)
 

setup/bootstrap.sh

@@ -20,7 +20,7 @@ if [ -z "$TAG" ]; then
 	# want to display in status checks.
 	if [ "`lsb_release -d | sed 's/.*:\s*//' | sed 's/18\.04\.[0-9]/18.04/' `" == "Ubuntu 18.04 LTS" ]; then
 		# This machine is running Ubuntu 18.04.
-		TAG=v0.42
+		TAG=v0.48
 
 	elif [ "`lsb_release -d | sed 's/.*:\s*//' | sed 's/14\.04\.[0-9]/14.04/' `" == "Ubuntu 14.04 LTS" ]; then
 		# This machine is running Ubuntu 14.04.
@@ -35,14 +35,14 @@ if [ -z "$TAG" ]; then
 
 	else
 		echo "This script must be run on a system running Ubuntu 18.04 or Ubuntu 14.04."
-		exit
+		exit 1
 	fi
 fi
 
 # Are we running as root?
 if [[ $EUID -ne 0 ]]; then
 	echo "This script must be run as root. Did you leave out sudo?"
-	exit
+	exit 1
 fi
 
 # Clone the Mail-in-a-Box repository if it doesn't exist.
@@ -73,7 +73,7 @@ if [ "$TAG" != `git describe` ]; then
 	git fetch --depth 1 --force --prune origin tag $TAG
 	if ! git checkout -q $TAG; then
 		echo "Update failed. Did you modify something in `pwd`?"
-		exit
+		exit 1
 	fi
 	echo
 fi

setup/dkim.sh

@@ -31,6 +31,7 @@ if grep -q "ExternalIgnoreList" /etc/opendkim.conf; then
 else
 	# Add various configuration options to the end of `opendkim.conf`.
 	cat >> /etc/opendkim.conf << EOF;
+Canonicalization		relaxed/simple
 MinimumKeyBits          1024
 ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
 InternalHosts           refile:/etc/opendkim/TrustedHosts

setup/functions.sh

@@ -57,15 +57,6 @@ function apt_install {
 	apt_get_quiet install $PACKAGES
 }
 
-function apt_add_repository_to_unattended_upgrades {
-	if [ -f /etc/apt/apt.conf.d/50unattended-upgrades ]; then
-		if ! grep -q "$1" /etc/apt/apt.conf.d/50unattended-upgrades; then
-			sed -i "/Allowed-Origins/a \
-	    \"$1\";" /etc/apt/apt.conf.d/50unattended-upgrades
-		fi
-	fi
-}
-
 function get_default_hostname {
 	# Guess the machine's hostname. It should be a fully qualified
 	# domain name suitable for DNS. None of these calls may provide
@@ -145,7 +136,14 @@ function get_default_privateip {
 function ufw_allow {
 	if [ -z "${DISABLE_FIREWALL:-}" ]; then
 		# ufw has completely unhelpful output
-		ufw allow $1 > /dev/null;
+		ufw allow "$1" > /dev/null;
+	fi
+}
+
+function ufw_limit {
+	if [ -z "${DISABLE_FIREWALL:-}" ]; then
+		# ufw has completely unhelpful output
+		ufw limit "$1" > /dev/null;
 	fi
 }
 

setup/mail-dovecot.sh

@@ -78,17 +78,16 @@ tools/editconf.py /etc/dovecot/conf.d/10-auth.conf \
 	"auth_mechanisms=plain login"
 
 # Enable SSL, specify the location of the SSL certificate and private key files.
-# Disable obsolete SSL protocols and allow only good ciphers per http://baldric.net/2013/12/07/tls-ciphers-in-postfix-and-dovecot/.
-# Enable strong ssl dh parameters
-
+# Use Mozilla's "Intermediate" recommendations at https://ssl-config.mozilla.org/#server=dovecot&server-version=2.2.33&config=intermediate&openssl-version=1.1.1,
+# except that the current version of Dovecot does not have a TLSv1.3 setting, so we only use TLSv1.2.
 tools/editconf.py /etc/dovecot/conf.d/10-ssl.conf \
 	ssl=required \
 	"ssl_cert=<$STORAGE_ROOT/ssl/ssl_certificate.pem" \
 	"ssl_key=<$STORAGE_ROOT/ssl/ssl_private_key.pem" \
-	"ssl_protocols=!SSLv3" \
-	"ssl_cipher_list=ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS" \
-	"ssl_prefer_server_ciphers = yes" \
-	"ssl_dh_parameters_length = 2048"
+	"ssl_protocols=TLSv1.2" \
+	"ssl_cipher_list=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" \
+	"ssl_prefer_server_ciphers=no" \
+	"ssl_dh_parameters_length=2048"
 
 # Disable in-the-clear IMAP/POP because there is no reason for a user to transmit
 # login credentials outside of an encrypted connection. Only the over-TLS versions

setup/mail-postfix.sh

@@ -80,8 +80,7 @@ tools/editconf.py /etc/postfix/main.cf \
 #   OpenDKIM milter only. See dkim.sh.
 # * Even though we dont allow auth over non-TLS connections (smtpd_tls_auth_only below, and without auth the client cant
 #   send outbound mail), don't allow non-TLS mail submission on this port anyway to prevent accidental misconfiguration.
-# * Require the best ciphers for incoming connections per http://baldric.net/2013/12/07/tls-ciphers-in-postfix-and-dovecot/.
-#   By putting this setting here we leave opportunistic TLS on incoming mail at default cipher settings (any cipher is better than none).
+#   Setting smtpd_tls_security_level=encrypt also triggers the use of the 'mandatory' settings below.
 # * Give it a different name in syslog to distinguish it from the port 25 smtpd server.
 # * Add a new cleanup service specific to the submission service ('authclean')
 #   that filters out privacy-sensitive headers on mail being sent out by
@@ -93,7 +92,6 @@ tools/editconf.py /etc/postfix/master.cf -s -w \
 	  -o syslog_name=postfix/submission
 	  -o smtpd_milters=inet:127.0.0.1:8891
 	  -o smtpd_tls_security_level=encrypt
-	  -o smtpd_tls_ciphers=high -o smtpd_tls_exclude_ciphers=aNULL,DES,3DES,MD5,DES+MD5,RC4 -o smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
 	  -o cleanup_service_name=authclean" \
 	"authclean=unix  n       -       -       -       0       cleanup
 	  -o header_checks=pcre:/etc/postfix/outgoing_mail_header_filters
@@ -108,20 +106,35 @@ cp conf/postfix_outgoing_mail_header_filters /etc/postfix/outgoing_mail_header_f
 sed -i "s/PRIMARY_HOSTNAME/$PRIMARY_HOSTNAME/" /etc/postfix/outgoing_mail_header_filters
 sed -i "s/PUBLIC_IP/$PUBLIC_IP/" /etc/postfix/outgoing_mail_header_filters
 
-# Enable TLS on these and all other connections (i.e. ports 25 *and* 587) and
-# require TLS before a user is allowed to authenticate. This also makes
-# opportunistic TLS available on *incoming* mail.
-# Set stronger DH parameters, which via openssl tend to default to 1024 bits
-# (see ssl.sh).
+# Enable TLS on incoming connections. It is not required on port 25, allowing for opportunistic
+# encryption. On port 587 it is mandatory (see above). Shared and non-shared settings are
+# given here. Shared settings include:
+# * Require TLS before a user is allowed to authenticate.
+# * Set the path to the server TLS certificate and 2048-bit DH parameters for old DH ciphers.
+# For port 25 only:
+# * Disable extremely old versions of TLS and extremely unsafe ciphers, but some mail servers out in
+#   the world are very far behind and if we disable too much, they may not be able to use TLS and
+#   won't fall back to cleartext. So we don't disable too much. smtpd_tls_exclude_ciphers applies to
+#   both port 25 and port 587, but because we override the cipher list for both, it probably isn't used.
+#   Use Mozilla's "Old" recommendations at https://ssl-config.mozilla.org/#server=postfix&server-version=3.3.0&config=old&openssl-version=1.1.1
+# For port 587 (via the 'mandatory' settings):
+# * Use Mozilla's "Intermediate" TLS recommendations from https://ssl-config.mozilla.org/#server=postfix&server-version=3.3.0&config=intermediate&openssl-version=1.1.1
+#   using and overriding the "high" cipher list so we don't conflict with the more permissive settings for port 25.
 tools/editconf.py /etc/postfix/main.cf \
 	smtpd_tls_security_level=may\
 	smtpd_tls_auth_only=yes \
 	smtpd_tls_cert_file=$STORAGE_ROOT/ssl/ssl_certificate.pem \
 	smtpd_tls_key_file=$STORAGE_ROOT/ssl/ssl_private_key.pem \
 	smtpd_tls_dh1024_param_file=$STORAGE_ROOT/ssl/dh2048.pem \
-	smtpd_tls_protocols=\!SSLv2,\!SSLv3 \
+	smtpd_tls_protocols="!SSLv2,!SSLv3" \
 	smtpd_tls_ciphers=medium \
+	tls_medium_cipherlist=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA \
 	smtpd_tls_exclude_ciphers=aNULL,RC4 \
+	smtpd_tls_mandatory_protocols="!SSLv2,!SSLv3,!TLSv1,!TLSv1.1" \
+	smtpd_tls_mandatory_ciphers=high \
+	tls_high_cipherlist=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 \
+	smtpd_tls_mandatory_exclude_ciphers=aNULL,DES,3DES,MD5,DES+MD5,RC4 \
+	tls_preempt_cipherlist=no \
 	smtpd_tls_received_header=yes
 
 # Prevent non-authenticated users from sending mail that requires being
@@ -143,8 +156,12 @@ tools/editconf.py /etc/postfix/main.cf \
 # offers it, otherwise it will transmit the message in the clear. Postfix will
 # accept whatever SSL certificate the remote end provides. Opportunistic TLS
 # protects against passive easvesdropping (but not man-in-the-middle attacks).
+# Since we'd rather have poor encryption than none at all, we use Mozilla's
+# "Old" recommendations at https://ssl-config.mozilla.org/#server=postfix&server-version=3.3.0&config=old&openssl-version=1.1.1
+# for opportunistic encryption but "Intermediate" recommendations when DANE
+# is used (see next and above). The cipher lists are set above.
+
 # DANE takes this a step further:
-#
 # Postfix queries DNS for the TLSA record on the destination MX host. If no TLSA records are found,
 # then opportunistic TLS is used. Otherwise the server certificate must match the TLSA records
 # or else the mail bounces. TLSA also requires DNSSEC on the MX host. Postfix doesn't do DNSSEC
@@ -157,11 +174,12 @@ tools/editconf.py /etc/postfix/main.cf \
 # now see notices about trusted certs. The CA file is provided by the package `ca-certificates`.
 tools/editconf.py /etc/postfix/main.cf \
 	smtp_tls_protocols=\!SSLv2,\!SSLv3 \
-	smtp_tls_mandatory_protocols=\!SSLv2,\!SSLv3 \
 	smtp_tls_ciphers=medium \
 	smtp_tls_exclude_ciphers=aNULL,RC4 \
 	smtp_tls_security_level=dane \
 	smtp_dns_support_level=dnssec \
+	smtp_tls_mandatory_protocols="!SSLv2,!SSLv3,!TLSv1,!TLSv1.1" \
+	smtp_tls_mandatory_ciphers=high \
 	smtp_tls_CAfile=/etc/ssl/certs/ca-certificates.crt \
 	smtp_tls_loglevel=2
 
@@ -210,6 +228,33 @@ tools/editconf.py /etc/postfix/main.cf \
 tools/editconf.py /etc/default/postgrey \
 	POSTGREY_OPTS=\"'--inet=127.0.0.1:10023 --delay=180'\"
 
+
+# We are going to setup a newer whitelist for postgrey, the version included in the distribution is old
+cat > /etc/cron.daily/mailinabox-postgrey-whitelist << EOF;
+#!/bin/bash
+
+# Mail-in-a-Box
+
+# check we have a postgrey_whitelist_clients file and that it is not older than 28 days
+if [ ! -f /etc/postgrey/whitelist_clients ] || find /etc/postgrey/whitelist_clients -mtime +28 | grep -q '.' ; then
+    # ok we need to update the file, so lets try to fetch it
+    if curl https://postgrey.schweikert.ch/pub/postgrey_whitelist_clients --output /tmp/postgrey_whitelist_clients -sS --fail > /dev/null 2>&1 ; then
+        # if fetching hasn't failed yet then check it is a plain text file
+        # curl manual states that --fail sometimes still produces output
+        # this final check will at least check the output is not html
+        # before moving it into place
+        if [ "\$(file -b --mime-type /tmp/postgrey_whitelist_clients)" == "text/plain" ]; then
+            mv /tmp/postgrey_whitelist_clients /etc/postgrey/whitelist_clients
+            service postgrey restart
+	else
+            rm /tmp/postgrey_whitelist_clients
+        fi
+    fi
+fi
+EOF
+chmod +x /etc/cron.daily/mailinabox-postgrey-whitelist
+/etc/cron.daily/mailinabox-postgrey-whitelist
+
 # Increase the message size limit from 10MB to 128MB.
 # The same limit is specified in nginx.conf for mail submitted via webmail and Z-Push.
 tools/editconf.py /etc/postfix/main.cf \

setup/management.sh

@@ -93,17 +93,19 @@ source $venv/bin/activate
 exec python `pwd`/management/daemon.py
 EOF
 chmod +x $inst_dir/start
-hide_output systemctl link -f conf/mailinabox.service
+cp --remove-destination conf/mailinabox.service /lib/systemd/system/mailinabox.service # target was previously a symlink so remove it first
+hide_output systemctl link -f /lib/systemd/system/mailinabox.service
 hide_output systemctl daemon-reload
 hide_output systemctl enable mailinabox.service
 
 # Perform nightly tasks at 3am in system time: take a backup, run
 # status checks and email the administrator any changes.
 
+minute=$((RANDOM % 60))  # avoid overloading mailinabox.email
 cat > /etc/cron.d/mailinabox-nightly << EOF;
 # Mail-in-a-Box --- Do not edit / will be overwritten on update.
 # Run nightly tasks: backup, status checks.
-0 3 * * *	root	(cd `pwd` && management/daily_tasks.sh)
+$minute 3 * * *	root	(cd `pwd` && management/daily_tasks.sh)
 EOF
 
 # Start the management server.

setup/munin.sh

@@ -53,7 +53,7 @@ find /etc/munin/plugins/ -lname /usr/share/munin/plugins/ntp_ -print0 | xargs -0
 # Deactivate monitoring of network interfaces that are not up. Otherwise we can get a lot of empty charts.
 for f in $(find /etc/munin/plugins/ \( -lname /usr/share/munin/plugins/if_ -o -lname /usr/share/munin/plugins/if_err_ -o -lname /usr/share/munin/plugins/bonding_err_ \)); do
 	IF=$(echo $f | sed s/.*_//);
-	if ! ifquery $IF >/dev/null 2>/dev/null; then
+	if ! grep -qFx up /sys/class/net/$IF/operstate 2>/dev/null; then
 		rm $f;
 	fi;
 done
@@ -64,7 +64,8 @@ mkdir -p /var/lib/munin-node/plugin-state/
 # Create a systemd service for munin.
 ln -sf $(pwd)/management/munin_start.sh /usr/local/lib/mailinabox/munin_start.sh
 chmod 0744 /usr/local/lib/mailinabox/munin_start.sh
-hide_output systemctl link -f conf/munin.service
+cp --remove-destination conf/munin.service /lib/systemd/system/munin.service # target was previously a symlink so remove first
+hide_output systemctl link -f /lib/systemd/system/munin.service
 hide_output systemctl daemon-reload
 hide_output systemctl unmask munin.service
 hide_output systemctl enable munin.service

setup/nextcloud.sh

@@ -25,12 +25,12 @@ InstallNextcloud() {
 	echo "Upgrading to Nextcloud version $version"
 	echo
 
-	# Remove the current owncloud/Nextcloud
-	rm -rf /usr/local/lib/owncloud
-
         # Download and verify
         wget_verify https://download.nextcloud.com/server/releases/nextcloud-$version.zip $hash /tmp/nextcloud.zip
 
+	# Remove the current owncloud/Nextcloud
+	rm -rf /usr/local/lib/owncloud
+
 	# Extract ownCloud/Nextcloud
 	unzip -q /tmp/nextcloud.zip -d /usr/local/lib
 	mv /usr/local/lib/nextcloud /usr/local/lib/owncloud
@@ -40,19 +40,21 @@ InstallNextcloud() {
 	# their github repositories.
 	mkdir -p /usr/local/lib/owncloud/apps
 
-	wget_verify https://github.com/nextcloud/contacts/releases/download/v3.1.1/contacts.tar.gz a06bd967197dcb03c94ec1dbd698c037018669e5 /tmp/contacts.tgz
+	wget_verify https://github.com/nextcloud/contacts/releases/download/v3.3.0/contacts.tar.gz e55d0357c6785d3b1f3b5f21780cb6d41d32443a /tmp/contacts.tgz
 	tar xf /tmp/contacts.tgz -C /usr/local/lib/owncloud/apps/
 	rm /tmp/contacts.tgz
 
-	wget_verify https://github.com/nextcloud/calendar/releases/download/v1.6.5/calendar.tar.gz 79941255521a5172f7e4ce42dc7773838b5ede2f /tmp/calendar.tgz
+	wget_verify https://github.com/nextcloud/calendar/releases/download/v2.0.3/calendar.tar.gz 9d9717b29337613b72c74e9914c69b74b346c466 /tmp/calendar.tgz
 	tar xf /tmp/calendar.tgz -C /usr/local/lib/owncloud/apps/
 	rm /tmp/calendar.tgz
 
 	# Starting with Nextcloud 15, the app user_external is no longer included in Nextcloud core,
 	# we will install from their github repository.
-	wget_verify https://github.com/nextcloud/user_external/releases/download/v0.6.3/user_external-0.6.3.tar.gz 0f756d35fef6b64a177d6a16020486b76ea5799c /tmp/user_external.tgz
+	if [[ $version =~ ^1[567] ]]; then
+		wget_verify https://github.com/nextcloud/user_external/releases/download/v0.7.0/user_external-0.7.0.tar.gz 555a94811daaf5bdd336c5e48a78aa8567b86437 /tmp/user_external.tgz
 		tar -xf /tmp/user_external.tgz -C /usr/local/lib/owncloud/apps/
 		rm /tmp/user_external.tgz
+	fi
 
 	# Fix weird permissions.
 	chmod 750 /usr/local/lib/owncloud/{apps,config}
@@ -88,11 +90,26 @@ InstallNextcloud() {
 	fi
 }
 
-nextcloud_ver=15.0.8
-nextcloud_hash=4129d8d4021c435f2e86876225fb7f15adf764a3
-# Check if Nextcloud dir exist, and check if version matches nextcloud_ver (if either doesn't - install/upgrade)
-if [ ! -d /usr/local/lib/owncloud/ ] \
-		|| ! grep -q $nextcloud_ver /usr/local/lib/owncloud/version.php; then
+# Nextcloud Version to install. Checks are done down below to step through intermediate versions.
+nextcloud_ver=17.0.6
+nextcloud_hash=50b98d2c2f18510b9530e558ced9ab51eb4f11b0
+
+# Current Nextcloud Version, #1623
+# Checking /usr/local/lib/owncloud/version.php shows version of the Nextcloud application, not the DB
+# $STORAGE_ROOT/owncloud is kept together even during a backup.  It is better to rely on config.php than
+# version.php since the restore procedure can leave the system in a state where you have a newer Nextcloud
+# application version than the database.
+
+# If config.php exists, get version number, otherwise CURRENT_NEXTCLOUD_VER is empty.
+if [ -f "$STORAGE_ROOT/owncloud/config.php" ]; then
+	CURRENT_NEXTCLOUD_VER=$(php -r "include(\"$STORAGE_ROOT/owncloud/config.php\"); echo(\$CONFIG['version']);")
+else
+	CURRENT_NEXTCLOUD_VER=""
+fi
+
+# If the Nextcloud directory is missing (never been installed before, or the nextcloud version to be installed is different
+# from the version currently installed, do the install/upgrade
+if [ ! -d /usr/local/lib/owncloud/ ] || [[ ! ${CURRENT_NEXTCLOUD_VER} =~ ^$nextcloud_ver ]]; then
 
 	# Stop php-fpm if running. If theyre not running (which happens on a previously failed install), dont bail.
 	service php7.2-fpm stop &> /dev/null || /bin/true
@@ -113,21 +130,30 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 	fi
 
 	# If ownCloud or Nextcloud was previously installed....
-	if [ -e /usr/local/lib/owncloud/version.php ]; then
+	if [ ! -z ${CURRENT_NEXTCLOUD_VER} ]; then
 		# Database migrations from ownCloud are no longer possible because ownCloud cannot be run under
 		# PHP 7.
-		if grep -q "OC_VersionString = '[89]\." /usr/local/lib/owncloud/version.php; then
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^[89] ]]; then
 			echo "Upgrades from Mail-in-a-Box prior to v0.28 (dated July 30, 2018) with Nextcloud < 13.0.6 (you have ownCloud 8 or 9) are not supported. Upgrade to Mail-in-a-Box version v0.30 first. Setup aborting."
 			exit 1
-		fi
-		if grep -q "OC_VersionString = '1[012]\." /usr/local/lib/owncloud/version.php; then
+		elif [[ ${CURRENT_NEXTCLOUD_VER} =~ ^1[012] ]]; then
 			echo "Upgrades from Mail-in-a-Box prior to v0.28 (dated July 30, 2018) with Nextcloud < 13.0.6 (you have ownCloud 10, 11 or 12) are not supported. Upgrade to Mail-in-a-Box version v0.30 first. Setup aborting."
 			exit 1
+		elif [[ ${CURRENT_NEXTCLOUD_VER} =~ ^13 ]]; then
+			# If we are running Nextcloud 13, upgrade to Nextcloud 14
+			InstallNextcloud 14.0.6 4e43a57340f04c2da306c8eea98e30040399ae5a
+			CURRENT_NEXTCLOUD_VER="14.0.6"
 		fi
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^14 ]]; then
 			# During the upgrade from Nextcloud 14 to 15, user_external may cause the upgrade to fail.
 			# We will disable it here before the upgrade and install it again after the upgrade.
-		if grep -q "OC_VersionString = '14\." /usr/local/lib/owncloud/version.php; then
 			hide_output sudo -u www-data php /usr/local/lib/owncloud/console.php app:disable user_external
+			InstallNextcloud 15.0.8 4129d8d4021c435f2e86876225fb7f15adf764a3
+			CURRENT_NEXTCLOUD_VER="15.0.8"
+		fi
+		if [[ ${CURRENT_NEXTCLOUD_VER} =~ ^15 ]]; then
+                        InstallNextcloud 16.0.6 0bb3098455ec89f5af77a652aad553ad40a88819
+                        CURRENT_NEXTCLOUD_VER="16.0.6"
                 fi
 	fi
 
@@ -277,10 +303,6 @@ tools/editconf.py /etc/php/7.2/cli/conf.d/10-opcache.ini -c ';' \
 	opcache.save_comments=1 \
 	opcache.revalidate_freq=1
 
-# Configure the path environment for php-fpm
-tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
-		env[PATH]=/usr/local/bin:/usr/bin:/bin
-
 # If apc is explicitly disabled we need to enable it
 if grep -q apc.enabled=0 /etc/php/7.2/mods-available/apcu.ini; then
 	tools/editconf.py /etc/php/7.2/mods-available/apcu.ini -c ';' \
@@ -288,12 +310,15 @@ if grep -q apc.enabled=0 /etc/php/7.2/mods-available/apcu.ini; then
 fi
 
 # Set up a cron job for Nextcloud.
-cat > /etc/cron.hourly/mailinabox-owncloud << EOF;
+cat > /etc/cron.d/mailinabox-nextcloud << EOF;
 #!/bin/bash
 # Mail-in-a-Box
-sudo -u www-data php -f /usr/local/lib/owncloud/cron.php
+*/5 * * * *	root	sudo -u www-data php -f /usr/local/lib/owncloud/cron.php
 EOF
-chmod +x /etc/cron.hourly/mailinabox-owncloud
+chmod +x /etc/cron.d/mailinabox-nextcloud
+
+# Remove previous hourly cronjob
+rm -f /etc/cron.hourly/mailinabox-owncloud
 
 # There's nothing much of interest that a user could do as an admin for Nextcloud,
 # and there's a lot they could mess up, so we don't make any users admins of Nextcloud.

setup/preflight.sh

@@ -4,7 +4,7 @@ if [[ $EUID -ne 0 ]]; then
 	echo
 	echo "sudo $0"
 	echo
-	exit
+	exit 1
 fi
 
 # Check that we are running on Ubuntu 18.04 LTS (or 18.04.xx).
@@ -14,7 +14,7 @@ if [ "`lsb_release -d | sed 's/.*:\s*//' | sed 's/18\.04\.[0-9]/18.04/' `" != "U
 	lsb_release -d | sed 's/.*:\s*//'
 	echo
 	echo "We can't write scripts that run on every possible setup, sorry."
-	exit
+	exit 1
 fi
 
 # Check that we have enough memory.

setup/spamassassin.sh

@@ -64,8 +64,8 @@ tools/editconf.py /etc/default/spampd \
 # the X-Spam-Status & X-Spam-Score mail headers and related headers.
 tools/editconf.py /etc/spamassassin/local.cf -s \
 	report_safe=0 \
-	add_header="all Report _REPORT_" \
-    add_header="all Score _SCORE_"
+	"add_header all Report"=_REPORT_ \
+	"add_header all Score"=_SCORE_
 
 # Bayesean learning
 # -----------------

setup/system.sh

@@ -86,6 +86,10 @@ if [ ! -f /usr/bin/add-apt-repository ]; then
 	apt_install software-properties-common
 fi
 
+# Ensure the universe repository is enabled since some of our packages
+# come from there and minimal Ubuntu installs may have it turned off.
+hide_output add-apt-repository -y universe
+
 # Install the certbot PPA.
 hide_output add-apt-repository -y ppa:certbot/certbot
 
@@ -121,11 +125,12 @@ apt_get_quiet autoremove
 # * sudo: allows privileged users to execute commands as root without being root
 # * coreutils: includes `nproc` tool to report number of processors, mktemp
 # * bc: allows us to do math to compute sane defaults
+# * openssh-client: provides ssh-keygen
 
 echo Installing system packages...
 apt_install python3 python3-dev python3-pip \
 	netcat-openbsd wget curl git sudo coreutils bc \
-	haveged pollinate unzip \
+	haveged pollinate openssh-client unzip \
 	unattended-upgrades cron ntp fail2ban rsyslog
 
 # ### Suppress Upgrade Prompts
@@ -251,7 +256,7 @@ if [ -z "${DISABLE_FIREWALL:-}" ]; then
 	apt_install ufw
 
 	# Allow incoming connections to SSH.
-	ufw_allow ssh;
+	ufw_limit ssh;
 
 	# ssh might be running on an alternate port. Use sshd -T to dump sshd's #NODOC
 	# settings, find the port it is supposedly running on, and open that port #NODOC
@@ -261,7 +266,7 @@ if [ -z "${DISABLE_FIREWALL:-}" ]; then
 	if [ "$SSH_PORT" != "22" ]; then
 
 	echo Opening alternate SSH port $SSH_PORT. #NODOC
-	ufw_allow $SSH_PORT #NODOC
+	ufw_limit $SSH_PORT #NODOC
 
 	fi
 	fi

setup/web.sh

@@ -31,14 +31,19 @@ sed "s#STORAGE_ROOT#$STORAGE_ROOT#" \
 	conf/nginx-ssl.conf > /etc/nginx/conf.d/ssl.conf
 
 # Fix some nginx defaults.
+#
 # The server_names_hash_bucket_size seems to prevent long domain names!
 # The default, according to nginx's docs, depends on "the size of the
 # processor’s cache line." It could be as low as 32. We fixed it at
 # 64 in 2014 to accommodate a long domain name (20 characters?). But
 # even at 64, a 58-character domain name won't work (#93), so now
 # we're going up to 128.
+#
+# Drop TLSv1.0, TLSv1.1, following the Mozilla "Intermediate" recommendations
+# at https://ssl-config.mozilla.org/#server=nginx&server-version=1.17.0&config=intermediate&openssl-version=1.1.1.
 tools/editconf.py /etc/nginx/nginx.conf -s \
-	server_names_hash_bucket_size="128;"
+	server_names_hash_bucket_size="128;" \
+	ssl_protocols="TLSv1.2 TLSv1.3;"
 
 # Tell PHP not to expose its version number in the X-Powered-By header.
 tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
@@ -48,13 +53,47 @@ tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
 tools/editconf.py /etc/php/7.2/fpm/php.ini -c ';' \
         default_charset="UTF-8"
 
-# Switch from the dynamic process manager to the ondemand manager see #1216
+# Configure the path environment for php-fpm
 tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
-	pm=ondemand
+	env[PATH]=/usr/local/bin:/usr/bin:/bin \
 
-# Bump up PHP's max_children to support more concurrent connections
-tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
-	pm.max_children=8
+# Configure php-fpm based on the amount of memory the machine has
+# This is based on the nextcloud manual for performance tuning: https://docs.nextcloud.com/server/17/admin_manual/installation/server_tuning.html
+# Some synchronisation issues can occur when many people access the site at once.
+# The pm=ondemand setting is used for memory constrained machines < 2GB, this is copied over from PR: 1216
+TOTAL_PHYSICAL_MEM=$(head -n 1 /proc/meminfo | awk '{print $2}' || /bin/true)
+if [ $TOTAL_PHYSICAL_MEM -lt 1000000 ]
+then
+        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+                pm=ondemand \
+                pm.max_children=8 \
+                pm.start_servers=2 \
+                pm.min_spare_servers=1 \
+                pm.max_spare_servers=3
+elif [ $TOTAL_PHYSICAL_MEM -lt 2000000 ]
+then
+        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+                pm=ondemand \
+                pm.max_children=16 \
+                pm.start_servers=4 \
+                pm.min_spare_servers=1 \
+                pm.max_spare_servers=6
+elif [ $TOTAL_PHYSICAL_MEM -lt 3000000 ]
+then
+        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+                pm=dynamic \
+                pm.max_children=60 \
+                pm.start_servers=6 \
+                pm.min_spare_servers=3 \
+                pm.max_spare_servers=9
+else
+        tools/editconf.py /etc/php/7.2/fpm/pool.d/www.conf -c ';' \
+                pm=dynamic \
+                pm.max_children=120 \
+                pm.start_servers=12 \
+                pm.min_spare_servers=6 \
+                pm.max_spare_servers=18
+fi
 
 # Other nginx settings will be configured by the management service
 # since it depends on what domains we're serving, which we don't know

setup/webmail.sh

@@ -22,15 +22,15 @@ source /etc/mailinabox.conf # load global vars
 echo "Installing Roundcube (webmail)..."
 apt_install \
 	dbconfig-common \
-	php-cli php-sqlite3 php-intl php-json php-common php-curl \
+	php-cli php-sqlite3 php-intl php-json php-common php-curl php-ldap \
 	php-gd php-pspell tinymce libjs-jquery libjs-jquery-mousewheel libmagic1 php-mbstring
 
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of plugins to track
 # whether we have the latest version of everything.
-VERSION=1.3.9
-HASH=02850972b416bbfa1c13580f16d06fd7ae2774aa
-PERSISTENT_LOGIN_VERSION=dc5ca3d3f4415cc41edb2fde533c8a8628a94c76
+VERSION=1.4.8
+HASH=3a6824fd68fef2e0d24f186cfbee5c6f9d6edbe9
+PERSISTENT_LOGIN_VERSION=6b3fc450cae23ccb2f393d0ef67aa319e877e435
 HTML5_NOTIFIER_VERSION=4b370e3cd60dabd2f428a26f45b677ad1b7118d5
 CARDDAV_VERSION=3.0.3
 CARDDAV_HASH=d1e3b0d851ffa2c6bd42bf0c04f70d0e1d0d78f8
@@ -51,6 +51,13 @@ elif [[ "$UPDATE_KEY" != `cat /usr/local/lib/roundcubemail/version` ]]; then
 	needs_update=1 #NODOC
 fi
 if [ $needs_update == 1 ]; then
+  # if upgrading from 1.3.x, clear the temp_dir
+  if [ -f /usr/local/lib/roundcubemail/version ]; then
+    if [ "$(cat /usr/local/lib/roundcubemail/version | cut -c1-3)" == '1.3' ]; then
+      find /var/tmp/roundcubemail/ -type f ! -name 'RCMTEMP*' -delete
+    fi
+  fi
+
 	# install roundcube
 	wget_verify \
 		https://github.com/roundcube/roundcubemail/releases/download/$VERSION/roundcubemail-$VERSION-complete.tar.gz \
@@ -110,9 +117,6 @@ cat > $RCM_CONFIG <<EOF;
  );
 \$config['imap_timeout'] = 15;
 \$config['smtp_server'] = 'tls://127.0.0.1';
-\$config['smtp_port'] = 587;
-\$config['smtp_user'] = '%u';
-\$config['smtp_pass'] = '%p';
 \$config['smtp_conn_options'] = array(
   'ssl'         => array(
      'verify_peer'  => false,
@@ -123,7 +127,7 @@ cat > $RCM_CONFIG <<EOF;
 \$config['product_name'] = '$PRIMARY_HOSTNAME Webmail';
 \$config['des_key'] = '$SECRET_KEY';
 \$config['plugins'] = array('html5_notifier', 'archive', 'zipdownload', 'password', 'managesieve', 'jqueryui', 'persistent_login', 'carddav');
-\$config['skin'] = 'larry';
+\$config['skin'] = 'elastic';
 \$config['login_autocomplete'] = 2;
 \$config['password_charset'] = 'UTF-8';
 \$config['junk_mbox'] = 'Spam';
@@ -156,7 +160,7 @@ mkdir -p /var/log/roundcubemail /var/tmp/roundcubemail $STORAGE_ROOT/mail/roundc
 chown -R www-data.www-data /var/log/roundcubemail /var/tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
 
 # Ensure the log file monitored by fail2ban exists, or else fail2ban can't start.
-sudo -u www-data touch /var/log/roundcubemail/errors
+sudo -u www-data touch /var/log/roundcubemail/errors.log
 
 # Password changing plugin settings
 # The config comes empty by default, so we need the settings

setup/zpush.sh

@@ -22,8 +22,8 @@ apt_install \
 phpenmod -v php imap
 
 # Copy Z-Push into place.
-VERSION=2.5.0
-TARGETHASH=30ce5c1af3f10939036361b6032d1187651b621e
+VERSION=2.5.2
+TARGETHASH=2dc3dbd791b96b0ba2638df0d3d1e03c7e1cbab2
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/z-push/version ]; then
 	needs_update=1 #NODOC

tests/fail2ban.py

@@ -90,6 +90,26 @@ def pop_test():
 		if M:
 			M.quit()
 
+def managesieve_test():
+	# We don't have a Python sieve client, so we'll
+	# just run the IMAP client and see what happens.
+	import imaplib
+
+	try:
+		M = imaplib.IMAP4(hostname, 4190)
+	except ConnectionRefusedError:
+		# looks like fail2ban worked
+		raise IsBlocked()
+
+	try:
+		M.login("fakeuser", "fakepassword")
+		raise Exception("authentication didn't fail")
+	except imaplib.IMAP4.error:
+		# authentication should fail
+		pass
+	finally:
+		M.logout() # shuts down connection, has nothing to do with login()
+
 def http_test(url, expected_status, postdata=None, qsargs=None, auth=None):
 	import urllib.parse
 	import requests
@@ -208,6 +228,9 @@ if __name__ == "__main__":
 	# POP
 	run_test(pop_test, [], 20, 30, 4)
 
+	# Managesieve
+	run_test(managesieve_test, [], 20, 30, 4)
+
 	# Mail-in-a-Box control panel
 	run_test(http_test, ["/admin/me", 200], 20, 30, 1)
 

tools/owncloud-restore.sh

@@ -22,7 +22,7 @@ fi
 
 if [ ! -f $1/config.php ]; then
 	echo "This isn't a valid backup location"
-	exit
+	exit 1
 fi
 
 echo "Restoring backup from $1"

tools/readable_bash.py

@@ -261,6 +261,10 @@ class UfwAllow(Grammar):
 	grammar = (ZERO_OR_MORE(SPACE), L("ufw_allow "), REST_OF_LINE, EOL)
 	def value(self):
 		return shell_line("ufw allow " + self[2].string)
+class UfwLimit(Grammar):
+	grammar = (ZERO_OR_MORE(SPACE), L("ufw_limit "), REST_OF_LINE, EOL)
+	def value(self):
+		return shell_line("ufw limit " + self[2].string)
 class RestartService(Grammar):
 	grammar = (ZERO_OR_MORE(SPACE), L("restart_service "), REST_OF_LINE, EOL)
 	def value(self):
@@ -275,7 +279,7 @@ class OtherLine(Grammar):
 		return "<pre class='shell'><div>" + recode_bash(self.string.strip()) + "</div></pre>\n"
 
 class BashElement(Grammar):
-	grammar = Comment | CatEOF | EchoPipe | EchoLine | HideOutput | EditConf | SedReplace | AptGet | UfwAllow | RestartService | OtherLine
+	grammar = Comment | CatEOF | EchoPipe | EchoLine | HideOutput | EditConf | SedReplace | AptGet | UfwAllow | UfwLimit | RestartService | OtherLine
 	def value(self):
 		return self[0].value()