v0.30 - the last Ubuntu 14.04 release

changelog
roundcube: upgrade carddav plugin to 3.0.3 & updated migrate.py (#1479 )
2026-03-12 17:07:23 +01:00 · 2019-01-09 06:31:56 -05:00 · 2019-01-09 06:17:27 -05:00 · 2018-12-03 15:33:36 -05:00 · 2018-12-02 18:03:17 -05:00 · 2018-12-02 18:02:00 -05:00
33 changed files with 700 additions and 667 deletions
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,84 @@
 CHANGELOG
 =========

+v0.30 (January 9, 2019)
+-----------------------
+
+Setup:
+
+* Update to Roundcube 1.3.8 and the CardDAV plugin to 3.0.3.
+* Add missing rsyslog package to install line since some OS images don't have it installed by default.
+* A log file for nsd was added.
+
+Control Panel:
+
+* The users page now documents that passwords should only have ASCII characters to prevent character encoding mismaches between clients and the server.
+* The users page no longer shows user mailbox sizes because this was extremely slow for very large mailboxes.
+* The Mail-in-a-Box version is now shown in the system status checks even when the new-version check is disabled.
+* The alises page now warns that alises should not be used to forward mail off of the box. Mail filters within Roundcube are better for that.
+* The explanation of greylisting has been improved.
+
+v0.29 (October 25, 2018)
+------------------------
+
+* Starting with v0.28, TLS certificate provisioning wouldn't work on new boxes until the mailinabox setup command was run a second time because of a problem with the non-interactive setup.
+* Update to Nextcloud 13.0.6.
+* Update to Roundcube 1.3.7.
+* Update to Z-Push 2.4.4.
+* Backup dates listed in the control panel now use an internationalized format.
+
+v0.28 (July 30, 2018)
+---------------------
+
+System:
+
+* We now use EFF's `certbot` to provision TLS certificates (from Let's Encrypt) instead of our home-grown ACME library.
+
+Contacts/Calendar:
+
+* Fix for Mac OS X autoconfig of the calendar.
+
+Setup:
+
+* Installing Z-Push broke because of what looks like a change or problem in their git server HTTPS certificate. That's fixed.
+
+v0.27 (June 14, 2018)
+---------------------
+
+Mail:
+
+* A report of box activity, including sent/received mail totals and logins by user, is now emailed to the box's administrator user each week.
+* Update Roundcube to version 1.3.6 and Z-Push to version 2.3.9.
+
+Control Panel:
+
+* The undocumented feature for proxying web requests to another server now sets X-Forwarded-For.
+
+v0.26c (February 13, 2018)
+--------------------------
+
+Setup:
+
+* Upgrades from v0.21c (February 1, 2017) or earlier were broken because the intermediate versions of ownCloud used in setup were no longer available from ownCloud.
+* Some download errors had no output --- there is more output on error now.
+
+Control Panel:
+
+* The background service for the control panel was not restarting on updates, leaving the old version running. This was broken in v0.26 and is now fixed.
+* Installing your own TLS/SSL certificate had been broken since v0.24 because the new version of openssl became stricter about CSR generation parameters.
+* Fixed password length help text.
+
+Contacts/Calendar:
+
+* Upgraded Nextcloud from 12.0.3 to 12.0.5.
+
+v0.26b (January 25, 2018)
+-------------------------
+
+* Fix new installations which broke at the step of asking for the user's desired email address, which was broken by v0.26's changes related to the control panel.
+* Fix the provisioning of TLS certificates by pinning a Python package we rely on (acme) to an earlier version because our code isn't yet compatible with its current version.
+* Reduce munin's log_level from debug to warning to prevent massive log files.
+
 v0.26 (January 18, 2018)
 ------------------------

@@ -24,7 +102,7 @@ Installer:
 * We now run `apt-get autoremove` at the start of setup to clear out old packages, especially old kernels that take up a lot of space. On the first run, this step may take a long time.
 * We now fetch Z-Push from its tagged git repository, fixing an installation problem.
 * Some old PHP5 packages are removed from setup, fixing an installation bug where Apache would get installed.
-* Python 3 packages for the control panel are now installed using a virtualenv to prevent installation errors.
+* Python 3 packages for the control panel are now installed using a virtualenv to prevent installation errors due to conflicts in the cryptography/openssl packages between OS-installed packages and pip-installed packages.

 v0.25 (November 15, 2017)
 -------------------------
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,3 +1,50 @@
+# Contributing
+
+Mail-in-a-Box is an open source project. Your contributions and pull requests are welcome.
+
+## Development
+
+To start developing Mail-in-a-Box, [clone the repository](https://github.com/mail-in-a-box/mailinabox) and familiarize yourself with the code.
+
+    $ git clone https://github.com/mail-in-a-box/mailinabox
+
+### Vagrant and VirtualBox
+
+We recommend you use [Vagrant](https://www.vagrantup.com/intro/getting-started/install.html) and [VirtualBox](https://www.virtualbox.org/wiki/Downloads) for development. Please install them first.
+
+With Vagrant set up, the following should boot up Mail-in-a-Box inside a virtual machine:
+
+    $ vagrant up --provision
+
+_If you're seeing an error message about your *IP address being listed in the Spamhaus Block List*, simply uncomment the `export SKIP_NETWORK_CHECKS=1` line in `Vagrantfile`. It's normal, you're probably using a dynamic IP address assigned by your Internet provider–they're almost all listed._
+
+### Modifying your `hosts` file
+
+After a while, Mail-in-a-Box will be available at `192.168.50.4` (unless you changed that in your `Vagrantfile`). To be able to use the web-based bits, we recommend to add a hostname to your `hosts` file:
+
+    $ echo "192.168.50.4 mailinabox.lan" | sudo tee -a /etc/hosts
+
+You should now be able to navigate to https://mailinabox.lan/admin using your browser. There should be an initial admin user with the name `me@mailinabox.lan` and the password `12345678`.
+
+### Making changes
+
+Your working copy of Mail-in-a-Box will be mounted inside your VM at `/vagrant`. Any change you make locally will appear inside your VM automatically.
+
+Running `vagrant up --provision` again will repeat the installation with your modifications.
+
+Alternatively, you can also ssh into the VM using:
+
+    $ vagrant ssh
+
+Once inside the VM, you can re-run individual parts of the setup like in this example:
+
+    vm$ cd /vagrant
+    vm$ sudo setup/owncloud.sh # replace with script you'd like to re-run
+
+### Tests
+
+Mail-in-a-Box needs more tests. If you're still looking for a way to help out, writing and contributing tests would be a great start!
+
 ## Public domain

 This project is in the public domain. Copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication][CC0]. See the LICENSE file in this directory.
--- a/README.md
+++ b/README.md
@@ -59,7 +59,7 @@ by me:
 	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
 	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported

-	$ git verify-tag v0.26
+	$ git verify-tag v0.30
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
@@ -72,7 +72,7 @@ and on my [personal homepage](https://razor.occams.info/). (Of course, if this r

 Checkout the tag corresponding to the most recent release:

-	$ git checkout v0.26
+	$ git checkout v0.30

 Begin the installation.

@@ -82,6 +82,12 @@ For help, DO NOT contact me directly --- I don't do tech support by email or twe

 Post your question on the [discussion forum](https://discourse.mailinabox.email/) instead, where me and other Mail-in-a-Box users may be able to help you.

+Contributing and Development
+----------------------------
+
+Mail-in-a-Box is an open source project. Your contributions and pull requests are welcome. See [CONTRIBUTING](CONTRIBUTING.md) to get started. 
+
+
 The Acknowledgements
 --------------------

--- a/6
+++ b/6
@@ -19,9 +19,9 @@ Vagrant.configure("2") do |config|
  config.vm.network "private_network", ip: "192.168.50.4"

  config.vm.provision :shell, :inline => <<-SH
-	# Set environment variables so that the setup script does
-	# not ask any questions during provisioning. We'll let the
-	# machine figure out its own public IP.
+    # Set environment variables so that the setup script does
+    # not ask any questions during provisioning. We'll let the
+    # machine figure out its own public IP.
    export NONINTERACTIVE=1
    export PUBLIC_IP=auto
    export PUBLIC_IPV6=auto
--- a/conf/ios-profile.xml
+++ b/conf/ios-profile.xml
@@ -18,8 +18,6 @@
      <string>PRIMARY_HOSTNAME</string>
      <key>CalDAVPort</key>
      <real>443</real>
-      <key>CalDAVPrincipalURL</key>
-      <string>/cloud/remote.php/caldav/calendars/</string>
      <key>CalDAVUseSSL</key>
      <true/>
      <key>PayloadDescription</key>
--- a/conf/nginx.conf
+++ b/conf/nginx.conf
@@ -25,7 +25,7 @@ server {
 		# This path must be served over HTTP for ACME domain validation.
 		# We map this to a special path where our TLS cert provisioning
 		# tool knows to store challenge response files.
-		alias $STORAGE_ROOT/ssl/lets_encrypt/acme_challenges/;
+		alias $STORAGE_ROOT/ssl/lets_encrypt/webroot/.well-known/acme-challenge/;
 	}
 }

--- a/management/backup.py
+++ b/management/backup.py
@@ -54,7 +54,7 @@ def backup_status(env):
 		date = dateutil.parser.parse(keys[1]).astimezone(dateutil.tz.tzlocal())
 		return {
 			"date": keys[1],
-			"date_str": date.strftime("%x %X") + " " + now.tzname(),
+			"date_str": date.strftime("%Y-%m-%d %X") + " " + now.tzname(),
 			"date_delta": reldate(date, now, "the future?"),
 			"full": keys[0] == "full",
 			"size": 0, # collection-status doesn't give us the size
--- a/management/daemon.py
+++ b/management/daemon.py
@@ -146,7 +146,7 @@ def me():
@authorized_personnel_only
 def mail_users():
 	if request.args.get("format", "") == "json":
-		return json_response(get_mail_users_ex(env, with_archived=True, with_slow_info=True))
+		return json_response(get_mail_users_ex(env, with_archived=True))
 	else:
 		return "".join(x+"\n" for x in get_mail_users(env))

@@ -333,11 +333,16 @@ def ssl_get_status():
 	from web_update import get_web_domains_info, get_web_domains

 	# What domains can we provision certificates for? What unexpected problems do we have?
-	provision, cant_provision = get_certificates_to_provision(env, show_extended_problems=False)
+	provision, cant_provision = get_certificates_to_provision(env, show_valid_certs=False)
 	
 	# What's the current status of TLS certificates on all of the domain?
 	domains_status = get_web_domains_info(env)
-	domains_status = [{ "domain": d["domain"], "status": d["ssl_certificate"][0], "text": d["ssl_certificate"][1] } for d in domains_status ]
+	domains_status = [
+		{
+			"domain": d["domain"],
+			"status": d["ssl_certificate"][0],
+			"text": d["ssl_certificate"][1] + ((" " + cant_provision[d["domain"]] if d["domain"] in cant_provision else ""))
+		} for d in domains_status ]

 	# Warn the user about domain names not hosted here because of other settings.
 	for domain in set(get_web_domains(env, exclude_dns_elsewhere=False)) - set(get_web_domains(env)):
@@ -349,7 +354,6 @@ def ssl_get_status():

 	return json_response({
 		"can_provision": utils.sort_domains(provision, env),
-		"cant_provision": [{ "domain": domain, "problem": cant_provision[domain] } for domain in utils.sort_domains(cant_provision, env) ],
 		"status": domains_status,
 	})

@@ -376,11 +380,8 @@ def ssl_install_cert():
@authorized_personnel_only
 def ssl_provision_certs():
 	from ssl_certificates import provision_certificates
-	agree_to_tos_url = request.form.get('agree_to_tos_url')
-	status = provision_certificates(env,
-		agree_to_tos_url=agree_to_tos_url,
-		jsonable=True)
-	return json_response(status)
+	requests = provision_certificates(env, limit_domains=None)
+	return json_response({ "requests": requests })


 # WEB
--- a/management/daily_tasks.sh
+++ b/management/daily_tasks.sh
@@ -9,11 +9,17 @@ export LC_ALL=en_US.UTF-8
 export LANG=en_US.UTF-8
 export LC_TYPE=en_US.UTF-8

+# On Mondays, i.e. once a week, send the administrator a report of total emails
+# sent and received so the admin might notice server abuse.
+if [ `date "+%u"` -eq 1 ]; then
+    management/mail_log.py -t week | management/email_administrator.py "Mail-in-a-Box Usage Report"
+fi
+
 # Take a backup.
 management/backup.py | management/email_administrator.py "Backup Status"

 # Provision any new certificates for new domains or domains with expiring certificates.
-management/ssl_certificates.py -q --headless | management/email_administrator.py "Error Provisioning TLS Certificate"
+management/ssl_certificates.py -q | management/email_administrator.py "Error Provisioning TLS Certificate"

 # Run status checks and email the administrator if anything changed.
 management/status_checks.py --show-changes | management/email_administrator.py "Status Checks Change Notice"
--- a/management/email_administrator.py
+++ b/management/email_administrator.py
@@ -4,8 +4,14 @@

 import sys

+import html
 import smtplib
-from email.message import Message
+
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+
+# In Python 3.6:
+#from email.message import Message

 from utils import load_environment

@@ -26,11 +32,23 @@ if content == "":
 	sys.exit(0)

 # create MIME message
-msg = Message()
+msg = MIMEMultipart('alternative')
+
+# In Python 3.6:
+#msg = Message()
+
 msg['From'] = "\"%s\" <%s>" % (env['PRIMARY_HOSTNAME'], admin_addr)
 msg['To'] = admin_addr
 msg['Subject'] = "[%s] %s" % (env['PRIMARY_HOSTNAME'], subject)
-msg.set_payload(content, "UTF-8")
+
+content_html = "<html><body><pre>{}</pre></body></html>".format(html.escape(content))
+
+msg.attach(MIMEText(content, 'plain'))
+msg.attach(MIMEText(content_html, 'html'))
+
+# In Python 3.6:
+#msg.set_content(content)
+#msg.add_alternative(content_html, "html")

 # send
 smtpclient = smtplib.SMTP('127.0.0.1', 25)
--- a/management/mail_log.py
+++ b/management/mail_log.py
@@ -53,10 +53,10 @@ VERBOSE = False
 # List of strings to filter users with
 FILTERS = None

-# What to show by default
+# What to show (with defaults)
 SCAN_OUT = True  # Outgoing email
 SCAN_IN = True  # Incoming email
-SCAN_CONN = False  # IMAP and POP3 logins
+SCAN_DOVECOT_LOGIN = True  # Dovecot Logins
 SCAN_GREY = False  # Greylisted email
 SCAN_BLOCKED = False  # Rejected email

@@ -76,7 +76,8 @@ def scan_files(collector):
            tmp_file = tempfile.NamedTemporaryFile()
            shutil.copyfileobj(gzip.open(fn), tmp_file)

-        print("Processing file", fn, "...")
+        if VERBOSE:
+            print("Processing file", fn, "...")
        fn = tmp_file.name if tmp_file else fn

        for line in reverse_readline(fn):
@@ -105,7 +106,7 @@ def scan_mail_log(env):
        "scan_time": time.time(),  # The time in seconds the scan took
        "sent_mail": OrderedDict(),  # Data about email sent by users
        "received_mail": OrderedDict(),  # Data about email received by users
-        "dovecot": OrderedDict(),  # Data about Dovecot activity
+        "logins": OrderedDict(),  # Data about login activity
        "postgrey": {},  # Data about greylisting of email addresses
        "rejected": OrderedDict(),  # Emails that were blocked
        "known_addresses": None,  # Addresses handled by the Miab installation
@@ -119,8 +120,8 @@ def scan_mail_log(env):
    except ImportError:
        pass

-    print("Scanning from {:%Y-%m-%d %H:%M:%S} back to {:%Y-%m-%d %H:%M:%S}".format(
-        START_DATE, END_DATE)
+    print("Scanning logs from {:%Y-%m-%d %H:%M:%S} to {:%Y-%m-%d %H:%M:%S}".format(
+        END_DATE, START_DATE)
    )

    # Scan the lines in the log files until the date goes out of range
@@ -138,8 +139,8 @@ def scan_mail_log(env):
    # Print Sent Mail report

    if collector["sent_mail"]:
-        msg = "Sent email between {:%Y-%m-%d %H:%M:%S} and {:%Y-%m-%d %H:%M:%S}"
-        print_header(msg.format(END_DATE, START_DATE))
+        msg = "Sent email"
+        print_header(msg)

        data = OrderedDict(sorted(collector["sent_mail"].items(), key=email_sort))

@@ -173,8 +174,8 @@ def scan_mail_log(env):
    # Print Received Mail report

    if collector["received_mail"]:
-        msg = "Received email between {:%Y-%m-%d %H:%M:%S} and {:%Y-%m-%d %H:%M:%S}"
-        print_header(msg.format(END_DATE, START_DATE))
+        msg = "Received email"
+        print_header(msg)

        data = OrderedDict(sorted(collector["received_mail"].items(), key=email_sort))

@@ -199,43 +200,55 @@ def scan_mail_log(env):
            [accum]
        )

-    # Print Dovecot report
+    # Print login report

-    if collector["dovecot"]:
-        msg = "Email client logins between {:%Y-%m-%d %H:%M:%S} and {:%Y-%m-%d %H:%M:%S}"
-        print_header(msg.format(END_DATE, START_DATE))
+    if collector["logins"]:
+        msg = "User logins per hour"
+        print_header(msg)

-        data = OrderedDict(sorted(collector["dovecot"].items(), key=email_sort))
+        data = OrderedDict(sorted(collector["logins"].items(), key=email_sort))
+
+        # Get a list of all of the protocols seen in the logs in reverse count order.
+        all_protocols = defaultdict(int)
+        for u in data.values():
+            for protocol_name, count in u["totals_by_protocol"].items():
+                all_protocols[protocol_name] += count
+        all_protocols = [k for k, v in sorted(all_protocols.items(), key=lambda kv : -kv[1])]

        print_user_table(
            data.keys(),
            data=[
-                ("imap", [u["imap"] for u in data.values()]),
-                ("pop3", [u["pop3"] for u in data.values()]),
+                (protocol_name, [
+                    round(u["totals_by_protocol"][protocol_name] / (u["latest"]-u["earliest"]).total_seconds() * 60*60, 1)
+                    if (u["latest"]-u["earliest"]).total_seconds() > 0
+                    else 0 # prevent division by zero
+                  for u in data.values()])
+                for protocol_name in all_protocols
            ],
            sub_data=[
-                ("IMAP IP addresses", [[k + " (%d)" % v for k, v in u["imap-logins"].items()]
-                                       for u in data.values()]),
-                ("POP3 IP addresses", [[k + " (%d)" % v for k, v in u["pop3-logins"].items()]
-                                       for u in data.values()]),
+                ("Protocol and Source", [[
+                    "{} {}: {} times".format(protocol_name, host, count)
+                    for (protocol_name, host), count
+                    in sorted(u["totals_by_protocol_and_host"].items(), key=lambda kv:-kv[1])
+                  ] for u in data.values()])
            ],
            activity=[
-                ("imap", [u["activity-by-hour"]["imap"] for u in data.values()]),
-                ("pop3", [u["activity-by-hour"]["pop3"] for u in data.values()]),
+                (protocol_name, [u["activity-by-hour"][protocol_name] for u in data.values()])
+                for protocol_name in all_protocols
            ],
            earliest=[u["earliest"] for u in data.values()],
            latest=[u["latest"] for u in data.values()],
+            numstr=lambda n : str(round(n, 1)),
        )

-        accum = {"imap": defaultdict(int), "pop3": defaultdict(int), "both": defaultdict(int)}
+        accum = { protocol_name: defaultdict(int) for protocol_name in all_protocols }
        for h in range(24):
-            accum["imap"][h] = sum(d["activity-by-hour"]["imap"][h] for d in data.values())
-            accum["pop3"][h] = sum(d["activity-by-hour"]["pop3"][h] for d in data.values())
-            accum["both"][h] = accum["imap"][h] + accum["pop3"][h]
+            for protocol_name in all_protocols:
+              accum[protocol_name][h] = sum(d["activity-by-hour"][protocol_name][h] for d in data.values())

        print_time_table(
-            ["imap", "pop3", "   +"],
-            [accum["imap"], accum["pop3"], accum["both"]]
+            all_protocols,
+            [accum[protocol_name] for protocol_name in all_protocols]
        )

    if collector["postgrey"]:
@@ -244,7 +257,7 @@ def scan_mail_log(env):

        print(textwrap.fill(
            "The following mail was greylisted, meaning the emails were temporarily rejected. "
-            "Legitimate senders will try again within ten minutes.",
+            "Legitimate senders must try again after three minutes.",
            width=80, initial_indent=" ", subsequent_indent=" "
        ), end='\n\n')

@@ -348,9 +361,9 @@ def scan_mail_log_line(line, collector):
    elif service == "postfix/lmtp":
        if SCAN_IN:
            scan_postfix_lmtp_line(date, log, collector)
-    elif service in ("imap-login", "pop3-login"):
-        if SCAN_CONN:
-            scan_dovecot_line(date, log, collector, service[:4])
+    elif service.endswith("-login"):
+        if SCAN_DOVECOT_LOGIN:
+            scan_dovecot_login_line(date, log, collector, service[:4])
    elif service == "postgrey":
        if SCAN_GREY:
            scan_postgrey_line(date, log, collector)
@@ -448,44 +461,43 @@ def scan_postfix_smtpd_line(date, log, collector):
                collector["rejected"][user] = data


-def scan_dovecot_line(date, log, collector, prot):
-    """ Scan a dovecot log line and extract interesting data """
+def scan_dovecot_login_line(date, log, collector, protocol_name):
+    """ Scan a dovecot login log line and extract interesting data """

    m = re.match("Info: Login: user=<(.*?)>, method=PLAIN, rip=(.*?),", log)

    if m:
        # TODO: CHECK DIT
-        user, rip = m.groups()
+        user, host = m.groups()

        if user_match(user):
+            add_login(user, date, protocol_name, host, collector)
+
+
+def add_login(user, date, protocol_name, host, collector):
            # Get the user data, or create it if the user is new
-            data = collector["dovecot"].get(
+            data = collector["logins"].get(
                user,
                {
-                    "imap": 0,
-                    "pop3": 0,
                    "earliest": None,
                    "latest": None,
-                    "imap-logins": defaultdict(int),
-                    "pop3-logins": defaultdict(int),
-                    "activity-by-hour": {
-                        "imap": defaultdict(int),
-                        "pop3": defaultdict(int),
-                    },
+                    "totals_by_protocol": defaultdict(int),
+                    "totals_by_protocol_and_host": defaultdict(int),
+                    "activity-by-hour": defaultdict(lambda : defaultdict(int)),
                }
            )

-            data[prot] += 1
-            data["activity-by-hour"][prot][date.hour] += 1
-
            if data["latest"] is None:
                data["latest"] = date
            data["earliest"] = date

-            if rip not in ("127.0.0.1", "::1") or True:
-                data["%s-logins" % prot][rip] += 1
+            data["totals_by_protocol"][protocol_name] += 1
+            data["totals_by_protocol_and_host"][(protocol_name, host)] += 1

-            collector["dovecot"][user] = data
+            if host not in ("127.0.0.1", "::1") or True:
+                data["activity-by-hour"][protocol_name][date.hour] += 1
+
+            collector["logins"][user] = data


 def scan_postfix_lmtp_line(date, log, collector):
@@ -561,6 +573,8 @@ def scan_postfix_submission_line(date, log, collector):

            collector["sent_mail"][user] = data

+            # Also log this as a login.
+            add_login(user, date, "smtp", client, collector)

 # Utility functions

@@ -640,7 +654,7 @@ def print_time_table(labels, data, do_print=True):
        for i, d in enumerate(data):
            lines[i] += base.format(d[h])

-    lines.insert(0, "┬")
+    lines.insert(0, "┬ totals by time of day:")
    lines.append("└" + (len(lines[-1]) - 2) * "─")

    if do_print:
@@ -650,7 +664,7 @@ def print_time_table(labels, data, do_print=True):


 def print_user_table(users, data=None, sub_data=None, activity=None, latest=None, earliest=None,
-                     delimit=False):
+                     delimit=False, numstr=str):
    str_temp = "{:<32} "
    lines = []
    data = data or []
@@ -764,7 +778,7 @@ def print_user_table(users, data=None, sub_data=None, activity=None, latest=None

    # Print totals

-    data_accum = [str(a) for a in data_accum]
+    data_accum = [numstr(a) for a in data_accum]
    footer = str_temp.format("Totals:" if do_accum else " ")
    for row, (l, _) in enumerate(data):
        temp = "{:>%d}" % max(5, len(l) + 1)
@@ -818,7 +832,7 @@ if __name__ == "__main__":
                        action="store_true")
    parser.add_argument("-s", "--sent", help="Scan for sent emails.",
                        action="store_true")
-    parser.add_argument("-l", "--logins", help="Scan for IMAP/POP logins.",
+    parser.add_argument("-l", "--logins", help="Scan for user logins to IMAP/POP3.",
                        action="store_true")
    parser.add_argument("-g", "--grey", help="Scan for greylisted emails.",
                        action="store_true")
@@ -863,8 +877,8 @@ if __name__ == "__main__":
        if not SCAN_OUT:
            print("Ignoring sent emails")

-        SCAN_CONN = args.logins
-        if not SCAN_CONN:
+        SCAN_DOVECOT_LOGIN = args.logins
+        if not SCAN_DOVECOT_LOGIN:
            print("Ignoring logins")

        SCAN_GREY = args.grey
--- a/management/mailconfig.py
+++ b/management/mailconfig.py
@@ -1,5 +1,14 @@
 #!/usr/local/lib/mailinabox/env/bin/python

+# NOTE:
+# This script is run both using the system-wide Python 3
+# interpreter (/usr/bin/python3) as well as through the
+# virtualenv (/usr/local/lib/mailinabox/env). So only
+# import packages at the top level of this script that
+# are installed in *both* contexts. We use the system-wide
+# Python 3 in setup/questions.sh to validate the email
+# address entered by the user.
+
 import subprocess, shutil, os, sqlite3, re
 import utils
 from email_validator import validate_email as validate_email_, EmailNotValidError
@@ -96,7 +105,7 @@ def get_mail_users(env):
 	users = [ row[0] for row in c.fetchall() ]
 	return utils.sort_email_addresses(users, env)

-def get_mail_users_ex(env, with_archived=False, with_slow_info=False):
+def get_mail_users_ex(env, with_archived=False):
 	# Returns a complex data structure of all user accounts, optionally
 	# including archived (status="inactive") accounts.
 	#
@@ -130,9 +139,6 @@ def get_mail_users_ex(env, with_archived=False, with_slow_info=False):
 		}
 		users.append(user)

-		if with_slow_info:
-			user["mailbox_size"] = utils.du(os.path.join(env['STORAGE_ROOT'], 'mail/mailboxes', *reversed(email.split("@"))))
-
 	# Add in archived accounts.
 	if with_archived:
 		root = os.path.join(env['STORAGE_ROOT'], 'mail/mailboxes')
@@ -149,8 +155,6 @@ def get_mail_users_ex(env, with_archived=False, with_slow_info=False):
 						"mailbox": mbox,
 					}
 					users.append(user)
-					if with_slow_info:
-						user["mailbox_size"] = utils.du(mbox)

 	# Group by domain.
 	domains = { }
--- a/management/ssl_certificates.py
+++ b/management/ssl_certificates.py
@@ -1,7 +1,7 @@
 #!/usr/local/lib/mailinabox/env/bin/python
 # Utilities for installing and selecting SSL certificates.

-import os, os.path, re, shutil
+import os, os.path, re, shutil, subprocess, tempfile

 from utils import shell, safe_domain_name, sort_domains
 import idna
@@ -24,6 +24,16 @@ def get_ssl_certificates(env):
 		if not os.path.exists(ssl_root):
 			return
 		for fn in os.listdir(ssl_root):
+			if fn == 'ssl_certificate.pem':
+				# This is always a symbolic link
+				# to the certificate to use for
+				# PRIMARY_HOSTNAME. Don't let it
+				# be eligible for use because we
+				# could end up creating a symlink
+				# to itself --- we want to find
+				# the cert that it should be a
+				# symlink to.
+				continue
 			fn = os.path.join(ssl_root, fn)
 			if os.path.isfile(fn):
 				yield fn
@@ -74,6 +84,12 @@ def get_ssl_certificates(env):

 		# Add this cert to the list of certs usable for the domains.
 		for domain in cert_domains:
+			# The primary hostname can only use a certificate mapped
+			# to the system private key.
+			if domain == env['PRIMARY_HOSTNAME']:
+				if cert._private_key._filename != os.path.join(env['STORAGE_ROOT'], 'ssl', 'ssl_private_key.pem'):
+					continue
+
 			domains.setdefault(domain, []).append(cert)

 	# Sort the certificates to prefer good ones.
@@ -81,6 +97,7 @@ def get_ssl_certificates(env):
 	now = datetime.datetime.utcnow()
 	ret = { }
 	for domain, cert_list in domains.items():
+		#for c in cert_list: print(domain, c.not_valid_before, c.not_valid_after, "("+str(now)+")", c.issuer, c.subject, c._filename)
 		cert_list.sort(key = lambda cert : (
 			# must be valid NOW
 			cert.not_valid_before <= now <= cert.not_valid_after,
@@ -124,21 +141,23 @@ def get_ssl_certificates(env):

 	return ret

-def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False, raw=False):
-	# Get the system certificate info.
-	ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
-	ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
-	system_certificate = {
-		"private-key": ssl_private_key,
-		"certificate": ssl_certificate,
-		"primary-domain": env['PRIMARY_HOSTNAME'],
-		"certificate_object": load_pem(load_cert_chain(ssl_certificate)[0]),
-	}
+def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False, use_main_cert=True):
+	if use_main_cert or not allow_missing_cert:
+		# Get the system certificate info.
+		ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
+		ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
+		system_certificate = {
+			"private-key": ssl_private_key,
+			"certificate": ssl_certificate,
+			"primary-domain": env['PRIMARY_HOSTNAME'],
+			"certificate_object": load_pem(load_cert_chain(ssl_certificate)[0]),
+		}

-	if domain == env['PRIMARY_HOSTNAME']:
-		# The primary domain must use the server certificate because
-		# it is hard-coded in some service configuration files.
-		return system_certificate
+	if use_main_cert:
+		if domain == env['PRIMARY_HOSTNAME']:
+			# The primary domain must use the server certificate because
+			# it is hard-coded in some service configuration files.
+			return system_certificate

 	wildcard_domain = re.sub("^[^\.]+", "*", domain)
 	if domain in ssl_certificates:
@@ -155,136 +174,97 @@ def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False

 # PROVISIONING CERTIFICATES FROM LETSENCRYPT

-def get_certificates_to_provision(env, show_extended_problems=True, force_domains=None):
-	# Get a set of domain names that we should now provision certificates
-	# for. Provision if a domain name has no valid certificate or if any
-	# certificate is expiring in 14 days. If provisioning anything, also
-	# provision certificates expiring within 30 days. The period between
-	# 14 and 30 days allows us to consolidate domains into multi-domain
-	# certificates for domains expiring around the same time.
+def get_certificates_to_provision(env, limit_domains=None, show_valid_certs=True):
+	# Get a set of domain names that we can provision certificates for
+	# using certbot. We start with domains that the box is serving web
+	# for and subtract:
+	# * domains not in limit_domains if limit_domains is not empty
+	# * domains with custom "A" records, i.e. they are hosted elsewhere
+	# * domains with actual "A" records that point elsewhere
+	# * domains that already have certificates that will be valid for a while

 	from web_update import get_web_domains
+	from status_checks import query_dns, normalize_ip

-	import datetime
-	now = datetime.datetime.utcnow()
+	existing_certs = get_ssl_certificates(env)

-	# Get domains with missing & expiring certificates.
-	certs = get_ssl_certificates(env)
-	domains = set()
-	domains_if_any = set()
-	problems = { }
-	for domain in get_web_domains(env):
-		# If the user really wants a cert for certain domains, include it.
-		if force_domains:
-			if force_domains == "ALL" or (isinstance(force_domains, list) and domain in force_domains):
-				domains.add(domain)
+	plausible_web_domains = get_web_domains(env, exclude_dns_elsewhere=False)
+	actual_web_domains = get_web_domains(env)
+
+	domains_to_provision = set()
+	domains_cant_provision = { }
+
+	for domain in plausible_web_domains:
+		# Skip domains that the user doesn't want to provision now.
+		if limit_domains and domain not in limit_domains:
 			continue

-		# Include this domain if its certificate is missing, self-signed, or expiring soon.
-		try:
-			cert = get_domain_ssl_files(domain, certs, env, allow_missing_cert=True)
-		except FileNotFoundError as e:
-			# system certificate is not present
-			problems[domain] = "Error: " + str(e)
-			continue
-		if cert is None:
-			# No valid certificate available.
-			domains.add(domain)
+		# Check that there isn't an explicit A/AAAA record.
+		if domain not in actual_web_domains:
+			domains_cant_provision[domain] = "The domain has a custom DNS A/AAAA record that points the domain elsewhere, so there is no point to installing a TLS certificate here and we could not automatically provision one anyway because provisioning requires access to the website (which isn't here)."
+
+		# Check that the DNS resolves to here.
 		else:
-			cert = cert["certificate_object"]
-			if cert.issuer == cert.subject:
-				# This is self-signed. Get a real one.
-				domains.add(domain)
+
+			# Does the domain resolve to this machine in public DNS? If not,
+			# we can't do domain control validation. For IPv6 is configured,
+			# make sure both IPv4 and IPv6 are correct because we don't know
+			# how Let's Encrypt will connect.
+			bad_dns = []
+			for rtype, value in [("A", env["PUBLIC_IP"]), ("AAAA", env.get("PUBLIC_IPV6"))]:
+				if not value: continue # IPv6 is not configured
+				response = query_dns(domain, rtype)
+				if response != normalize_ip(value):
+					bad_dns.append("%s (%s)" % (response, rtype))
+	
+			if bad_dns:
+				domains_cant_provision[domain] = "The domain name does not resolve to this machine: " \
+					+ (", ".join(bad_dns)) \
+					+ "."
 			
-			# Valid certificate today, but is it expiring soon?
-			elif cert.not_valid_after-now < datetime.timedelta(days=14):
-				domains.add(domain)
-			elif cert.not_valid_after-now < datetime.timedelta(days=30):
-				domains_if_any.add(domain)
+			else:
+				# DNS is all good.

-			# It's valid. Should we report its validness?
-			elif show_extended_problems:
-				problems[domain] = "The certificate is valid for at least another 30 days --- no need to replace."
+				# Check for a good existing cert.
+				existing_cert = get_domain_ssl_files(domain, existing_certs, env, use_main_cert=False, allow_missing_cert=True)
+				if existing_cert:
+					existing_cert_check = check_certificate(domain, existing_cert['certificate'], existing_cert['private-key'],
+						warn_if_expiring_soon=14)
+					if existing_cert_check[0] == "OK":
+						if show_valid_certs:
+							domains_cant_provision[domain] = "The domain has a valid certificate already. ({} Certificate: {}, private key {})".format(
+								existing_cert_check[1],
+								existing_cert['certificate'],
+								existing_cert['private-key'])
+						continue

-	# Warn the user about domains hosted elsewhere.
-	if not force_domains and show_extended_problems:
-		for domain in set(get_web_domains(env, exclude_dns_elsewhere=False)) - set(get_web_domains(env)):
-			problems[domain] = "The domain's DNS is pointed elsewhere, so there is no point to installing a TLS certificate here and we could not automatically provision one anyway because provisioning requires access to the website (which isn't here)."
+				domains_to_provision.add(domain)

-	# Filter out domains that we can't provision a certificate for.
-	def can_provision_for_domain(domain):
-		from status_checks import normalize_ip
-
-		# Does the domain resolve to this machine in public DNS? If not,
-		# we can't do domain control validation. For IPv6 is configured,
-		# make sure both IPv4 and IPv6 are correct because we don't know
-		# how Let's Encrypt will connect.
-		import dns.resolver
-		for rtype, value in [("A", env["PUBLIC_IP"]), ("AAAA", env.get("PUBLIC_IPV6"))]:
-			if not value: continue # IPv6 is not configured
-			try:
-				# Must make the qname absolute to prevent a fall-back lookup with a
-				# search domain appended, by adding a period to the end.
-				response = dns.resolver.query(domain + ".", rtype)
-			except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as e:
-				problems[domain] = "DNS isn't configured properly for this domain: DNS resolution failed (%s: %s)." % (rtype, str(e) or repr(e)) # NoAnswer's str is empty
-				return False
-			except Exception as e:
-				problems[domain] = "DNS isn't configured properly for this domain: DNS lookup had an error: %s." % str(e)
-				return False
-
-			# Unfortunately, the response.__str__ returns bytes
-			# instead of string, if it resulted from an AAAA-query.
-			# We need to convert manually, until this is fixed:
-			# https://github.com/rthalley/dnspython/issues/204
-			#
-			# BEGIN HOTFIX
-			def rdata__str__(r):
-				s = r.to_text()
-				if isinstance(s, bytes):
-					 s = s.decode('utf-8')
-				return s
-			# END HOTFIX
-
-			if len(response) != 1 or normalize_ip(rdata__str__(response[0])) != normalize_ip(value):
-				problems[domain] = "Domain control validation cannot be performed for this domain because DNS points the domain to another machine (%s %s)." % (rtype, ", ".join(rdata__str__(r) for r in response))
-				return False
-
-		return True
-
-	domains = set(filter(can_provision_for_domain, domains))
-
-	# If there are any domains we definitely will provision for, add in
-	# additional domains to do at this time.
-	if len(domains) > 0:
-		domains |= set(filter(can_provision_for_domain, domains_if_any))
-
-	return (domains, problems)
-
-def provision_certificates(env, agree_to_tos_url=None, logger=None, show_extended_problems=True, force_domains=None, jsonable=False):
-	import requests.exceptions
-	import acme.messages
-
-	from free_tls_certificates import client
+	return (domains_to_provision, domains_cant_provision)

+def provision_certificates(env, limit_domains):
 	# What domains should we provision certificates for? And what
 	# errors prevent provisioning for other domains.
-	domains, problems = get_certificates_to_provision(env, force_domains=force_domains, show_extended_problems=show_extended_problems)
+	domains, domains_cant_provision = get_certificates_to_provision(env, limit_domains=limit_domains)
+
+	# Build a list of what happened on each domain or domain-set.
+	ret = []
+	for domain, error in domains_cant_provision.items():
+		ret.append({
+			"domains": [domain],
+			"log": [error],
+			"result": "skipped",
+		})

-	# Exit fast if there is nothing to do.
-	if len(domains) == 0:
-		return {
-			"requests": [],
-			"problems": problems,
-		}

 	# Break into groups of up to 100 certificates at a time, which is Let's Encrypt's
 	# limit for a single certificate. We'll sort to put related domains together.
+	max_domains_per_group = 100
 	domains = sort_domains(domains, env)
 	certs = []
 	while len(domains) > 0:
-		certs.append( domains[0:100] )
-		domains = domains[100:]
+		certs.append( domains[:max_domains_per_group] )
+		domains = domains[max_domains_per_group:]

 	# Prepare to provision.

@@ -293,115 +273,82 @@ def provision_certificates(env, agree_to_tos_url=None, logger=None, show_extende
 	if not os.path.exists(account_path):
 		os.mkdir(account_path)

-	# Where should we put ACME challenge files. This is mapped to /.well-known/acme_challenge
-	# by the nginx configuration.
-	challenges_path = os.path.join(account_path, 'acme_challenges')
-	if not os.path.exists(challenges_path):
-		os.mkdir(challenges_path)
-
-	# Read in the private key that we use for all TLS certificates. We'll need that
-	# to generate a CSR (done by free_tls_certificates).
-	with open(os.path.join(env['STORAGE_ROOT'], 'ssl/ssl_private_key.pem'), 'rb') as f:
-		private_key = f.read()
-
 	# Provision certificates.
-
-	ret = []
 	for domain_list in certs:
-		# For return.
-		ret_item = {
+		ret.append({
 			"domains": domain_list,
 			"log": [],
-		}
-		ret.append(ret_item)
-
-		# Logging for free_tls_certificates.
-		def my_logger(message):
-			if logger: logger(message)
-			ret_item["log"].append(message)
-
-		# Attempt to provision a certificate.
+		})
 		try:
-			try:
-				cert = client.issue_certificate(
-					domain_list,
-					account_path,
-					agree_to_tos_url=agree_to_tos_url,
-					private_key=private_key,
-					logger=my_logger)
+			# Create a CSR file for our master private key so that certbot
+			# uses our private key.
+			key_file = os.path.join(env['STORAGE_ROOT'], 'ssl', 'ssl_private_key.pem')
+			with tempfile.NamedTemporaryFile() as csr_file:
+				# We could use openssl, but certbot requires
+				# that the CN domain and SAN domains match
+				# the domain list passed to certbot, and adding
+				# SAN domains openssl req is ridiculously complicated.
+				# subprocess.check_output([
+				# 	"openssl", "req", "-new",
+				# 	"-key", key_file,
+				# 	"-out", csr_file.name,
+				# 	"-subj", "/CN=" + domain_list[0],
+				# 	"-sha256" ])
+				from cryptography import x509
+				from cryptography.hazmat.backends import default_backend
+				from cryptography.hazmat.primitives.serialization import Encoding
+				from cryptography.hazmat.primitives import hashes
+				from cryptography.x509.oid import NameOID
+				builder = x509.CertificateSigningRequestBuilder()
+				builder = builder.subject_name(x509.Name([ x509.NameAttribute(NameOID.COMMON_NAME, domain_list[0]) ]))
+				builder = builder.add_extension(x509.BasicConstraints(ca=False, path_length=None), critical=True)
+				builder = builder.add_extension(x509.SubjectAlternativeName(
+					[x509.DNSName(d) for d in domain_list]
+				), critical=False)
+				request = builder.sign(load_pem(load_cert_chain(key_file)[0]), hashes.SHA256(), default_backend())
+				with open(csr_file.name, "wb") as f:
+					f.write(request.public_bytes(Encoding.PEM))

-			except client.NeedToTakeAction as e:
-				# Write out the ACME challenge files.
-				for action in e.actions:
-					if isinstance(action, client.NeedToInstallFile):
-						fn = os.path.join(challenges_path, action.file_name)
-						with open(fn, 'w') as f:
-							f.write(action.contents)
-					else:
-						raise ValueError(str(action))
+				# Provision, writing to a temporary file.
+				webroot = os.path.join(account_path, 'webroot')
+				os.makedirs(webroot, exist_ok=True)
+				with tempfile.TemporaryDirectory() as d:
+					cert_file = os.path.join(d, 'cert_and_chain.pem')
+					print("Provisioning TLS certificates for " + ", ".join(domain_list) + ".")
+					certbotret = subprocess.check_output([
+						"certbot",
+						"certonly",
+						#"-v", # just enough to see ACME errors
+						"--non-interactive", # will fail if user hasn't registered during Mail-in-a-Box setup

-				# Try to provision now that the challenge files are installed.
+						"-d", ",".join(domain_list), # first will be main domain

-				cert = client.issue_certificate(
-					domain_list,
-					account_path,
-					private_key=private_key,
-					logger=my_logger)
+						"--csr", csr_file.name, # use our private key; unfortunately this doesn't work with auto-renew so we need to save cert manually
+						"--cert-path", os.path.join(d, 'cert'), # we only use the full chain
+						"--chain-path", os.path.join(d, 'chain'), # we only use the full chain
+						"--fullchain-path", cert_file,

-		except client.NeedToAgreeToTOS as e:
-			# The user must agree to the Let's Encrypt terms of service agreement
-			# before any further action can be taken.
-			ret_item.update({
-				"result": "agree-to-tos",
-				"url": e.url,
-			})
+						"--webroot", "--webroot-path", webroot,

-		except client.WaitABit as e:
-			# We need to hold on for a bit before querying again to see if we can
-			# acquire a provisioned certificate.
-			import time, datetime
-			ret_item.update({
-				"result": "wait",
-				"until": e.until_when if not jsonable else e.until_when.isoformat(),
-				"seconds": (e.until_when - datetime.datetime.now()).total_seconds()
-			})
+						"--config-dir", account_path,
+						#"--staging",
+					], stderr=subprocess.STDOUT).decode("utf8")
+					install_cert_copy_file(cert_file, env)

-		except client.AccountDataIsCorrupt as e:
-			# This is an extremely rare condition.
-			ret_item.update({
-				"result": "error",
-				"message": "Something unexpected went wrong. It looks like your local Let's Encrypt account data is corrupted. There was a problem with the file " + e.account_file_path + ".",
-			})
+			ret[-1]["log"].append(certbotret)
+			ret[-1]["result"] = "installed"
+		except subprocess.CalledProcessError as e:
+			ret[-1]["log"].append(e.output.decode("utf8"))
+			ret[-1]["result"] = "error"
+		except Exception as e:
+			ret[-1]["log"].append(str(e))
+			ret[-1]["result"] = "error"

-		except (client.InvalidDomainName, client.NeedToTakeAction, client.ChallengeFailed, client.RateLimited, acme.messages.Error, requests.exceptions.RequestException) as e:
-			ret_item.update({
-				"result": "error",
-				"message": "Something unexpected went wrong: " + str(e),
-			})
-
-		else:
-			# A certificate was issued.
-
-			install_status = install_cert(domain_list[0], cert['cert'].decode("ascii"), b"\n".join(cert['chain']).decode("ascii"), env, raw=True)
-
-			# str indicates the certificate was not installed.
-			if isinstance(install_status, str):
-				ret_item.update({
-					"result": "error",
-					"message": "Something unexpected was wrong with the provisioned certificate: " + install_status,
-				})
-			else:
-				# A list indicates success and what happened next.
-				ret_item["log"].extend(install_status)
-				ret_item.update({
-					"result": "installed",
-				})
+	# Run post-install steps.
+	ret.extend(post_install_func(env))

 	# Return what happened with each certificate request.
-	return {
-		"requests": ret,
-		"problems": problems,
-	}
+	return ret

 def provision_certificates_cmdline():
 	import sys
@@ -412,151 +359,39 @@ def provision_certificates_cmdline():
 	Lock(die=True).forever()
 	env = load_environment()

-	verbose = False
-	headless = False
-	force_domains = None
-	show_extended_problems = True
-	
-	args = list(sys.argv)
-	args.pop(0) # program name
-	if args and args[0] == "-v":
-		verbose = True
-		args.pop(0)
-	if args and args[0] == "-q":
-		show_extended_problems = False
-		args.pop(0)
-	if args and args[0] == "--headless":
-		headless = True
-		args.pop(0)
-	if args and args[0] == "--force":
-		force_domains = "ALL"
-		args.pop(0)
-	else:
-		force_domains = args
+	quiet = False
+	domains = []

-	agree_to_tos_url = None
-	while True:
-		# Run the provisioning script. This installs certificates. If there are
-		# a very large number of domains on this box, it issues separate
-		# certificates for groups of domains. We have to check the result for
-		# each group.
-		def my_logger(message):
-			if verbose:
-				print(">", message)
-		status = provision_certificates(env, agree_to_tos_url=agree_to_tos_url, logger=my_logger, force_domains=force_domains, show_extended_problems=show_extended_problems)
-		agree_to_tos_url = None # reset to prevent infinite looping
+	for arg in sys.argv[1:]:
+		if arg == "-q":
+			quiet = True
+		else:
+			domains.append(arg)

-		if not status["requests"]:
-			# No domains need certificates.
-			if not headless or verbose:
-				if len(status["problems"]) == 0:
-					print("No domains hosted on this box need a new TLS certificate at this time.")
-				elif len(status["problems"]) > 0:
-					print("No TLS certificates could be provisoned at this time:")
-					print()
-					for domain in sort_domains(status["problems"], env):
-						print("%s: %s" % (domain, status["problems"][domain]))
+	# Go.
+	status = provision_certificates(env, limit_domains=domains)

-			sys.exit(0)
-
-		# What happened?
-		wait_until = None
-		wait_domains = []
-		for request in status["requests"]:
-			if request["result"] == "agree-to-tos":
-				# We may have asked already in a previous iteration.
-				if agree_to_tos_url is not None:
-					continue
-
-				# Can't ask the user a question in this mode. Warn the user that something
-				# needs to be done.
-				if headless:
-					print(", ".join(request["domains"]) + " need a new or renewed TLS certificate.")
-					print()
-					print("This box can't do that automatically for you until you agree to Let's Encrypt's")
-					print("Terms of Service agreement. Use the Mail-in-a-Box control panel to provision")
-					print("certificates for these domains.")
-					sys.exit(1)
-
-				print("""
-I'm going to provision a TLS certificate (formerly called a SSL certificate)
-for you from Let's Encrypt (letsencrypt.org).
-
-TLS certificates are cryptographic keys that ensure communication between
-you and this box are secure when getting and sending mail and visiting
-websites hosted on this box. Let's Encrypt is a free provider of TLS
-certificates.
-
-Please open this document in your web browser:
-
-%s
-
-It is Let's Encrypt's terms of service agreement. If you agree, I can
-provision that TLS certificate. If you don't agree, you will have an
-opportunity to install your own TLS certificate from the Mail-in-a-Box
-control panel.
-
-Do you agree to the agreement? Type Y or N and press <ENTER>: """
-				 % request["url"], end='', flush=True)
-			
-				if sys.stdin.readline().strip().upper() != "Y":
-					print("\nYou didn't agree. Quitting.")
-					sys.exit(1)
-
-				# Okay, indicate agreement on next iteration.
-				agree_to_tos_url = request["url"]
-
-			if request["result"] == "wait":
-				# Must wait. We'll record until when. The wait occurs below.
-				if wait_until is None:
-					wait_until = request["until"]
-				else:
-					wait_until = max(wait_until, request["until"])
-				wait_domains += request["domains"]
-
-			if request["result"] == "error":
-				print(", ".join(request["domains"]) + ":")
-				print(request["message"])
-
-			if request["result"] == "installed":
-				print("A TLS certificate was successfully installed for " + ", ".join(request["domains"]) + ".")
-
-		if wait_until:
-			# Wait, then loop.
-			import time, datetime
+	# Show what happened.
+	for request in status:
+		if isinstance(request, str):
+			print(request)
+		else:
+			if quiet and request['result'] == 'skipped':
+				continue
+			print(request['result'] + ":", ", ".join(request['domains']) + ":")
+			for line in request["log"]:
+				print(line)
 			print()
-			print("A TLS certificate was requested for: " + ", ".join(wait_domains) + ".")
-			first = True
-			while wait_until > datetime.datetime.now():
-				if not headless or first:
-					print ("We have to wait", int(round((wait_until - datetime.datetime.now()).total_seconds())), "seconds for the certificate to be issued...")
-				time.sleep(10)
-				first = False

-			continue # Loop!
-
-		if agree_to_tos_url:
-			# The user agrees to the TOS. Loop to try again by agreeing.
-			continue # Loop!
-
-		# Unless we were instructed to wait, or we just agreed to the TOS,
-		# we're done for now.
-		break
-
-	# And finally show the domains with problems.
-	if len(status["problems"]) > 0:
-		print("TLS certificates could not be provisoned for:")
-		for domain in sort_domains(status["problems"], env):
-			print("%s: %s" % (domain, status["problems"][domain]))

 # INSTALLING A NEW CERTIFICATE FROM THE CONTROL PANEL

 def create_csr(domain, ssl_key, country_code, env):
 	return shell("check_output", [
-                "openssl", "req", "-new",
-                "-key", ssl_key,
-                "-sha256",
-                "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (country_code, domain)])
+				"openssl", "req", "-new",
+				"-key", ssl_key,
+				"-sha256",
+				"-subj", "/C=%s/CN=%s" % (country_code, domain)])

 def install_cert(domain, ssl_cert, ssl_chain, env, raw=False):
 	# Write the combined cert+chain to a temporary path and validate that it is OK.
@@ -577,6 +412,16 @@ def install_cert(domain, ssl_cert, ssl_chain, env, raw=False):
 			cert_status += " " + cert_status_details
 		return cert_status

+	# Copy certifiate into ssl directory.
+	install_cert_copy_file(fn, env)
+
+	# Run post-install steps.
+	ret = post_install_func(env)
+	if raw: return ret
+	return "\n".join(ret)
+
+
+def install_cert_copy_file(fn, env):
 	# Where to put it?
 	# Make a unique path for the certificate.
 	from cryptography.hazmat.primitives import hashes
@@ -594,14 +439,26 @@ def install_cert(domain, ssl_cert, ssl_chain, env, raw=False):
 	os.makedirs(os.path.dirname(ssl_certificate), exist_ok=True)
 	shutil.move(fn, ssl_certificate)

-	ret = ["OK"]

-	# When updating the cert for PRIMARY_HOSTNAME, symlink it from the system
+def post_install_func(env):
+	ret = []
+
+	# Get the certificate to use for PRIMARY_HOSTNAME.
+	ssl_certificates = get_ssl_certificates(env)
+	cert = get_domain_ssl_files(env['PRIMARY_HOSTNAME'], ssl_certificates, env, use_main_cert=False)
+	if not cert:
+		# Ruh-row, we don't have any certificate usable
+		# for the primary hostname.
+		ret.append("there is no valid certificate for " + env['PRIMARY_HOSTNAME'])
+
+	# Symlink the best cert for PRIMARY_HOSTNAME to the system
 	# certificate path, which is hard-coded for various purposes, and then
 	# restart postfix and dovecot.
-	if domain == env['PRIMARY_HOSTNAME']:
+	system_ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
+	if cert and os.readlink(system_ssl_certificate) != cert['certificate']:
 		# Update symlink.
-		system_ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
+		ret.append("updating primary certificate")
+		ssl_certificate = cert['certificate']
 		os.unlink(system_ssl_certificate)
 		os.symlink(ssl_certificate, system_ssl_certificate)

@@ -617,12 +474,12 @@ def install_cert(domain, ssl_cert, ssl_chain, env, raw=False):
 	# Update the web configuration so nginx picks up the new certificate file.
 	from web_update import do_web_update
 	ret.append( do_web_update(env) )
-	if raw: return ret
-	return "\n".join(ret)
+
+	return ret

 # VALIDATION OF CERTIFICATES

-def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=True, rounded_time=False, just_check_domain=False):
+def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=10, rounded_time=False, just_check_domain=False):
 	# Check that the ssl_certificate & ssl_private_key files are good
 	# for the provided domain.

@@ -728,7 +585,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 			# We'll renew it with Lets Encrypt.
 			expiry_info = "The certificate expires on %s." % cert_expiration_date.strftime("%x")

-		if ndays <= 10 and warn_if_expiring_soon:
+		if warn_if_expiring_soon and ndays <= warn_if_expiring_soon:
 			# Warn on day 10 to give 4 days for us to automatically renew the
 			# certificate, which occurs on day 14.
 			return ("The certificate is expiring soon: " + expiry_info, None)
--- a/management/status_checks.py
+++ b/management/status_checks.py
@@ -393,7 +393,7 @@ def check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles):

 	# Check that PRIMARY_HOSTNAME resolves to PUBLIC_IP[V6] in public DNS.
 	ipv6 = query_dns(domain, "AAAA") if env.get("PUBLIC_IPV6") else None
-	if ip == env['PUBLIC_IP'] and not (ipv6 and env['PUBLIC_IPV6'] and normalize_ip(ipv6) != normalize_ip(env['PUBLIC_IPV6'])):
+	if ip == env['PUBLIC_IP'] and not (ipv6 and env['PUBLIC_IPV6'] and ipv6 != normalize_ip(env['PUBLIC_IPV6'])):
 		output.print_ok("Domain resolves to box's IP address. [%s ↦ %s]" % (env['PRIMARY_HOSTNAME'], my_ips))
 	else:
 		output.print_error("""This domain must resolve to your box's IP address (%s) in public DNS but it currently resolves
@@ -640,7 +640,7 @@ def check_web_domain(domain, rounded_time, ssl_certificates, env, output):
 		for (rtype, expected) in (("A", env['PUBLIC_IP']), ("AAAA", env.get('PUBLIC_IPV6'))):
 			if not expected: continue # IPv6 is not configured
 			value = query_dns(domain, rtype)
-			if normalize_ip(value) == normalize_ip(expected):
+			if value == normalize_ip(expected):
 				ok_values.append(value)
 			else:
 				output.print_error("""This domain should resolve to your box's IP address (%s %s) if you would like the box to serve
@@ -687,27 +687,17 @@ def query_dns(qname, rtype, nxdomain='[Not Set]', at=None):
 	except dns.exception.Timeout:
 		return "[timeout]"

+	# Normalize IP addresses. IP address --- especially IPv6 addresses --- can
+	# be expressed in equivalent string forms. Canonicalize the form before
+	# returning them. The caller should normalize any IP addresses the result
+	# of this method is compared with.
+	if rtype in ("A", "AAAA"):
+		response = [normalize_ip(str(r)) for r in response]
+
 	# There may be multiple answers; concatenate the response. Remove trailing
 	# periods from responses since that's how qnames are encoded in DNS but is
 	# confusing for us. The order of the answers doesn't matter, so sort so we
 	# can compare to a well known order.
-
-	# Unfortunately, the response.__str__ returns bytes
-	# instead of string, if it resulted from an AAAA-query.
-	# We need to convert manually, until this is fixed:
-	# https://github.com/rthalley/dnspython/issues/204
-	#
-	# BEGIN HOTFIX
-	response_new = []
-	for r in response:
-		s = r.to_text()
-		if isinstance(s, bytes):
-			s = s.decode('utf-8')
-		response_new.append(s)
-	
-	response = response_new
-	# END HOTFIX
-
 	return "; ".join(sorted(str(r).rstrip('.') for r in response))

 def check_ssl_cert(domain, rounded_time, ssl_certificates, env, output):
@@ -805,14 +795,14 @@ def get_latest_miab_version():
 def check_miab_version(env, output):
 	config = load_settings(env)

-	if config.get("privacy", True):
-		output.print_warning("Mail-in-a-Box version check disabled by privacy setting.")
-	else:
-		try:
-			this_ver = what_version_is_this(env)
-		except:
-			this_ver = "Unknown"
+	try:
+		this_ver = what_version_is_this(env)
+	except:
+		this_ver = "Unknown"

+	if config.get("privacy", True):
+		output.print_warning("You are running version Mail-in-a-Box %s. Mail-in-a-Box version check disabled by privacy setting." % this_ver)
+	else:
 		latest_ver = get_latest_miab_version()

 		if this_ver == latest_ver:
@@ -892,7 +882,9 @@ def run_and_output_changes(env, pool):
 		json.dump(cur.buf, f, indent=True)

 def normalize_ip(ip):
-	# Use ipaddress module to normalize the IPv6 notation and ensure we are matching IPv6 addresses written in different representations according to rfc5952.
+	# Use ipaddress module to normalize the IPv6 notation and
+	# ensure we are matching IPv6 addresses written in different
+	# representations according to rfc5952.
 	import ipaddress
 	try:
 		return str(ipaddress.ip_address(ip))
--- a/management/templates/aliases.html
+++ b/management/templates/aliases.html
@@ -39,8 +39,9 @@
    <label for="addaliasForwardsTo" class="col-sm-1 control-label">Forwards To</label>
    <div class="col-sm-10">
      <textarea class="form-control" rows="3" id="addaliasForwardsTo"></textarea>
-      <div style="margin-top: 3px; padding-left: 3px; font-size: 90%" class="text-muted">
-        <span class="domainalias">Enter just the part of an email address starting with the @-sign.</span>
+      <div style="margin-top: 3px; padding-left: 3px; font-size: 90%">
+        <span class="domainalias text-muted">Enter just the part of an email address starting with the @-sign.</span>
+        <span class="text-danger">Only forward mail to addresses handled by this Mail-in-a-Box, since mail forwarded by aliases to other domains may be rejected or filtered by the receiver. To forward mail to other domains, create a mail user and then log into webmail for the user and create a filter rule to forward mail.</span>
      </div>
    </div>
  </div>
--- a/management/templates/mail-guide.html
+++ b/management/templates/mail-guide.html
@@ -59,7 +59,7 @@
 		</div>
 		<div class="panel-body">
 			<h4>Greylisting</h4>
-			<p>Your box using a technique called greylisting to cut down on spam. Greylisting works by delaying mail from people you haven&rsquo;t received mail from before for up to about 10 minutes. The vast majority of spam gets tricked by this. If you are waiting for an email from someone new, such as if you are registering on a new website and are waiting for an email confirmation, please give it up to 10-15 minutes to arrive.</p>
+			<p>Your box uses a technique called greylisting to cut down on spam. Greylisting works by initially rejecting mail from people you haven&rsquo;t received mail from before. Legitimate mail servers will attempt redelivery shortly afterwards, but the vast majority of spam gets tricked by this. If you are waiting for an email from someone new, such as if you are registering on a new website and are waiting for an email confirmation, please be aware there will be a minimum of 3 minutes delay, depending how soon the remote server attempts redelivery.</p>

 			<h4>+tag addresses</h4>
 			<p>Every incoming email address also receives mail for <code>+tag</code> addresses. If your email address is <code>you@yourdomain.com</code>, you&rsquo;ll also automatically get mail sent to <code>you+anythinghere@yourdomain.com</code>. Use this as a fast way to segment incoming mail for your own filtering rules without having to create aliases in this control panel.</p>
--- a/management/templates/ssl.html
+++ b/management/templates/ssl.html
@@ -8,7 +8,7 @@
 <p>You need a TLS certificate for this box&rsquo;s hostname ({{hostname}}) and every other domain name and subdomain that this box is hosting a website for (see the list below).</p>

 <div id="ssl_provision">
-<h3>Provision a certificate</h3>
+<h3>Provision certificates</h3>

 <div id="ssl_provision_p" style="display: none; margin-top: 1.5em">
  <button onclick='return provision_tls_cert();' class='btn btn-primary' style="float: left; margin: 0 1.5em 1em 0;">Provision</button>
@@ -19,21 +19,6 @@
 <div class="clearfix"> </div>

 <div id="ssl_provision_result"></div>
-
-<div id="ssl_provision_problems_div" style="display: none;">
-<p style="margin-bottom: .5em;">Certificates cannot be automatically provisioned for:</p>
-<table id="ssl_provision_problems" style="margin-top: 0;" class="table">
-  <thead>
-    <tr>
-      <th>Domain</th>
-      <th>Problem</th>
-    </tr>
-  </thead>
-  <tbody>
-  </tbody>
-</table>
-<p>Use the <em>Install Certificate</em> button below for these domains.</p>
-</div>
 </div>

 <h3>Certificate status</h3>
@@ -103,24 +88,12 @@ function show_tls(keep_provisioning_shown) {
      // provisioning status

      if (!keep_provisioning_shown)
-        $('#ssl_provision').toggle(res.can_provision.length + res.cant_provision.length > 0)
+        $('#ssl_provision').toggle(res.can_provision.length > 0)

      $('#ssl_provision_p').toggle(res.can_provision.length > 0);
      if (res.can_provision.length > 0)
          $('#ssl_provision_p span').text(res.can_provision.join(", "));

-      $('#ssl_provision_problems_div').toggle(res.cant_provision.length > 0);
-      $('#ssl_provision_problems tbody').text("");
-      for (var i = 0; i < res.cant_provision.length; i++) {
-        var domain = res.cant_provision[i];
-        var row = $("<tr><th class='domain'><a href=''></a></th><td class='status'></td></tr>");
-        $('#ssl_provision_problems tbody').append(row);
-        row.attr('data-domain', domain.domain);
-        row.find('.domain a').text(domain.domain);
-        row.find('.domain a').attr('href', 'https://' + domain.domain);
-        row.find('.status').text(domain.problem);
-      }
-
      // certificate status
      var domains = res.status;
      var tb = $('#ssl_domains tbody');
@@ -159,7 +132,11 @@ function ssl_install(elem) {
 }

 function show_csr() {
+ // Can't show a CSR until both inputs are entered.
 if ($('#ssldomain').val() == "") return;
+ if ($('#sslcc').val() == "") return;
+
+ // Scroll to it and fetch.
 $('#csr_info').slideDown();
 $('#ssl_csr').text('Loading...');
 api(
@@ -192,20 +169,15 @@ function install_cert() {
    });
 }

-var agree_to_tos_url_prompt = null;
-var agree_to_tos_url = null;
 function provision_tls_cert() {
  // Automatically provision any certs.
  $('#ssl_provision_p .btn').attr('disabled', '1'); // prevent double-clicks
  api(
    "/ssl/provision",
    "POST",
-    {
-      agree_to_tos_url: agree_to_tos_url
-    },
+    { },
    function(status) {
      // Clear last attempt.
-      agree_to_tos_url = null;
      $('#ssl_provision_result').text("");
      may_reenable_provision_button = true;

@@ -221,52 +193,33 @@ function provision_tls_cert() {
      for (var i = 0; i < status.requests.length; i++) {
        var r = status.requests[i];

+        if (r.result == "skipped") {
+          // not interested --- this domain wasn't in the table
+          // to begin with
+          continue;
+        }
+
        // create an HTML block to display the results of this request
        var n = $("<div><h4/><p/></div>");
        $('#ssl_provision_result').append(n);

+        // plain log line
+        if (typeof r === "string") {
+          n.find("p").text(r);
+          continue;
+        }
+
        // show a header only to disambiguate request blocks
        if (status.requests.length > 0)
          n.find("h4").text(r.domains.join(", "));

-        if (r.result == "agree-to-tos") {
-          // user needs to agree to Let's Encrypt's TOS
-          agree_to_tos_url_prompt = r.url;
-          $('#ssl_provision_p .btn').attr('disabled', '1');
-          n.find("p").html("Please open and review <a href='" + r.url + "' target='_blank'>Let's Encrypt's terms of service agreement</a>. You must agree to their terms for a certificate to be automatically provisioned from them.");
-          n.append($('<button onclick="agree_to_tos_url = agree_to_tos_url_prompt; return provision_tls_cert();" class="btn btn-success" style="margin-left: 2em">Agree &amp; Try Again</button>'));
-
-          // don't re-enable the Provision button -- user must use the Agree button
-          may_reenable_provision_button = false;
-
-        } else if (r.result == "error") {
+        if (r.result == "error") {
          n.find("p").addClass("text-danger").text(r.message);

-        } else if (r.result == "wait") {
-          // Show a button that counts down to zero, at which point it becomes enabled.
-          n.find("p").text("A certificate is now in the process of being provisioned, but it takes some time. Please wait until the Finish button is enabled, and then click it to acquire the certificate.");
-          var b = $('<button onclick="return provision_tls_cert();" class="btn btn-success" style="margin-left: 2em">Finish</button>');
-          b.attr("disabled", "1");
-          var now = new Date();
-          n.append(b);
-          function ready_to_finish() {
-            var remaining = Math.round(r.seconds - (new Date() - now)/1000);
-            if (remaining > 0) {
-              setTimeout(ready_to_finish, 1000);
-              b.text("Finish (" + remaining + "...)")
-            } else {
-              b.text("Finish (ready)")
-              b.removeAttr("disabled");
-            }
-          }
-          ready_to_finish();
-
-          // don't re-enable the Provision button -- user must use the Retry button when it becomes enabled
-          may_reenable_provision_button = false;
-
        } else if (r.result == "installed") {
          n.find("p").addClass("text-success").text("The TLS certificate was provisioned and installed.");
          setTimeout("show_tls(true)", 1); // update main table of certificate statuses, call with arg keep_provisioning_shown true so that we don't clear what we just outputted
+
        }

        // display the detailed log info in case of problems
@@ -274,7 +227,6 @@ function provision_tls_cert() {
        n.append(trace);
        for (var j = 0; j < r.log.length; j++)
          trace.append($("<div/>").text(r.log[j]));
-
      }

      if (may_reenable_provision_button)
--- a/management/templates/users.html
+++ b/management/templates/users.html
@@ -31,7 +31,7 @@
  <button type="submit" class="btn btn-primary">Add User</button>
 </form>
 <ul style="margin-top: 1em; padding-left: 1.5em; font-size: 90%;">
-  <li>Passwords must be at least eight characters and may not contain spaces. For best results, <a href="#" onclick="return generate_random_password()">generate a random password</a>.</li>
+  <li>Passwords must be at least eight characters consisting of English lettters and numbers only. For best results, <a href="#" onclick="return generate_random_password()">generate a random password</a>.</li>
  <li>Use <a href="#" onclick="return show_panel('aliases')">aliases</a> to create email addresses that forward to existing accounts.</li>
  <li>Administrators get access to this control panel.</li>
  <li>User accounts cannot contain any international (non-ASCII) characters, but <a href="#" onclick="return show_panel('aliases');">aliases</a> can.</li>
@@ -43,7 +43,6 @@
    <tr>
      <th width="50%">Email Address</th>
      <th>Actions</th>
-      <th>Mailbox Size</th>
    </tr>
  </thead>
  <tbody>
@@ -73,8 +72,6 @@
          archive account
        </a>
    </td>
-    <td class='mailboxsize'>
-    </td>
  </tr>
  <tr id="user-extra-template" class="if_inactive">
    <td colspan="3" style="border: 0; padding-top: 0">
@@ -156,7 +153,6 @@ function show_users() {

          n.attr('data-email', user.email);
          n.find('.address').text(user.email)
-          n.find('.mailboxsize').text(nice_size(user.mailbox_size))
          n2.find('.restore_info tt').text(user.mailbox);

          if (user.status == 'inactive') continue;
@@ -213,7 +209,7 @@ function users_set_password(elem) {

  show_modal_confirm(
    "Set Password",
-    $("<p>Set a new password for <b>" + email + "</b>?</p> <p><label for='users_set_password_pw' style='display: block; font-weight: normal'>New Password:</label><input type='password' id='users_set_password_pw'></p><p><small>Passwords must be at least four characters and may not contain spaces.</small>" + yourpw + "</p>"),
+    $("<p>Set a new password for <b>" + email + "</b>?</p> <p><label for='users_set_password_pw' style='display: block; font-weight: normal'>New Password:</label><input type='password' id='users_set_password_pw'></p><p><small>Passwords must be at least eight characters and may not contain spaces.</small>" + yourpw + "</p>"),
    "Set Password",
    function() {
      api(
--- a/management/web_update.py
+++ b/management/web_update.py
@@ -149,7 +149,10 @@ def make_domain_config(domain, templates, ssl_certificates, env):

 			# any proxy or redirect here?
 			for path, url in yaml.get("proxies", {}).items():
-				nginx_conf_extra += "\tlocation %s {\n\t\tproxy_pass %s;\n\t}\n" % (path, url)
+				nginx_conf_extra += "\tlocation %s {" % path
+				nginx_conf_extra += "\n\t\tproxy_pass %s;" % url
+				nginx_conf_extra += "\n\t\tproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;"
+				nginx_conf_extra += "\n\t}\n"
 			for path, url in yaml.get("redirects", {}).items():
 				nginx_conf_extra += "\trewrite %s %s permanent;\n" % (path, url)

@@ -198,8 +201,11 @@ def get_web_domains_info(env):

 	# for the SSL config panel, get cert status
 	def check_cert(domain):
-		tls_cert = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
-		if tls_cert is None: return ("danger", "No Certificate Installed")
+		try:
+			tls_cert = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
+		except OSError: # PRIMARY_HOSTNAME cert is missing
+			tls_cert = None
+		if tls_cert is None: return ("danger", "No certificate installed.")
 		cert_status, cert_status_details = check_certificate(domain, tls_cert["certificate"], tls_cert["private-key"])
 		if cert_status == "OK":
 			return ("success", "Signed & valid. " + cert_status_details)
--- a/setup/bootstrap.sh
+++ b/setup/bootstrap.sh
@@ -7,7 +7,7 @@
 #########################################################

 if [ -z "$TAG" ]; then
-	TAG=v0.26
+	TAG=v0.30
 fi

 # Are we running as root?
--- a/setup/dns.sh
+++ b/setup/dns.sh
@@ -26,6 +26,7 @@ cat > /etc/nsd/nsd.conf << EOF;
 # Do not edit. Overwritten by Mail-in-a-Box setup.
 server:
  hide-version: yes
+  logfile: "/var/log/nsd.log"

  # identify the server (CH TXT ID.SERVER entry).
  identity: ""
@@ -41,6 +42,18 @@ server:

 EOF

+# Add log rotation
+cat > /etc/logrotate.d/nsd <<EOF;
+/var/log/nsd.log {
+  weekly
+  missingok
+  rotate 12
+  compress
+  delaycompress
+  notifempty
+}
+EOF
+
 # Since we have bind9 listening on localhost for locally-generated
 # DNS queries that require a recursive nameserver, and the system
 # might have other network interfaces for e.g. tunnelling, we have
--- a/setup/functions.sh
+++ b/setup/functions.sh
@@ -179,7 +179,7 @@ function wget_verify {
 	DEST=$3
 	CHECKSUM="$HASH  $DEST"
 	rm -f $DEST
-	wget -q -O $DEST $URL || exit 1
+	hide_output wget -O $DEST $URL
 	if ! echo "$CHECKSUM" | sha1sum --check --strict > /dev/null; then
 		echo "------------------------------------------------------------"
 		echo "Download of $URL did not match expected checksum."
--- a/setup/management.sh
+++ b/setup/management.sh
@@ -6,18 +6,32 @@ echo "Installing Mail-in-a-Box system management daemon..."

 # DEPENDENCIES

+# We used to install management daemon-related Python packages
+# directly to /usr/local/lib. We moved to a virtualenv because
+# these packages might conflict with apt-installed packages.
+# We may have a lingering version of acme that conflcits with
+# certbot, which we're about to install below, so remove it
+# first. Once acme is installed by an apt package, this might
+# break the package version and `apt-get install --reinstall python3-acme`
+# might be needed in that case.
+while [ -d /usr/local/lib/python3.4/dist-packages/acme ]; do
+	pip3 uninstall -y acme;
+done
+
 # duplicity is used to make backups of user data. It uses boto
 # (via Python 2) to do backups to AWS S3. boto from the Ubuntu
 # package manager is too out-of-date -- it doesn't support the newer
 # S3 api used in some regions, which breaks backups to those regions.
 # See #627, #653.
-apt_install duplicity python-pip
+#
+# python-virtualenv is used to isolate the Python 3 packages we
+# install via pip from the system-installed packages.
+#
+# certbot installs EFF's certbot which we use to
+# provision free TLS certificates.
+apt_install duplicity python-pip python-virtualenv certbot
 hide_output pip2 install --upgrade boto

-# These are required to build/install the cryptography Python package
-# used by our management daemon.
-apt_install python-virtualenv build-essential libssl-dev libffi-dev python3-dev
-
 # Create a virtualenv for the installation of Python 3 packages
 # used by the management daemon.
 inst_dir=/usr/local/lib/mailinabox
@@ -27,39 +41,16 @@ if [ ! -d $venv ]; then
 	virtualenv -ppython3 $venv
 fi

-# pip<6.1 + setuptools>=34 had a problem with packages that
-# try to update setuptools during installation, like cryptography.
-# See https://github.com/pypa/pip/issues/4253. The Ubuntu 14.04
-# package versions are pip 1.5.4 and setuptools 3.3. When we used to
-# instal cryptography system-wide under those versions, it updated
-# setuptools to version 34, which created the conflict, and
-# then pip gets permanently broken with errors like
-# "ImportError: No module named 'packaging'".
-#
-# Let's test for the error:
-if ! python3 -c "from pkg_resources import load_entry_point" 2&> /dev/null; then
-	# This system seems to be broken already.
-	echo "Fixing broken pip and setuptools..."
-	rm -rf /usr/local/lib/python3.4/dist-packages/{pkg_resources,setuptools}*
-	apt-get install --reinstall python3-setuptools python3-pip python3-pkg-resources
-fi
-#
-# The easiest work-around on systems that aren't already broken is
-# to upgrade pip (to >=9.0.1) and setuptools (to >=34.1) individually
-# before we install any package that tries to update setuptools.
+# Upgrade pip because the Ubuntu-packaged version is out of date.
 hide_output $venv/bin/pip install --upgrade pip
-hide_output $venv/bin/pip install --upgrade setuptools

 # Install other Python 3 packages used by the management daemon.
 # The first line is the packages that Josh maintains himself!
 # NOTE: email_validator is repeated in setup/questions.sh, so please keep the versions synced.
-# Force acme to be updated because it seems to need it after the
-# pip/setuptools breakage (see above) and the ACME protocol may
-# have changed (I got an error on one of my systems).
 hide_output $venv/bin/pip install --upgrade \
-	rtyaml "email_validator>=1.0.0" "free_tls_certificates>=0.1.3" "exclusiveprocess" \
+	rtyaml "email_validator>=1.0.0" "exclusiveprocess" \
 	flask dnspython python-dateutil \
-	"idna>=2.0.0" "cryptography>=1.0.2" acme boto psutil
+	"idna>=2.0.0" "cryptography==2.2.2" boto psutil

 # CONFIGURATION

@@ -100,7 +91,7 @@ rm -f /usr/local/bin/mailinabox-daemon # old path
 cat > $inst_dir/start <<EOF;
 #!/bin/bash
 source $venv/bin/activate
-python `pwd`/management/daemon.py
+exec python `pwd`/management/daemon.py
 EOF
 chmod +x $inst_dir/start
 rm -f /etc/init.d/mailinabox
--- a/setup/migrate.py
+++ b/setup/migrate.py
@@ -137,6 +137,51 @@ def migration_10(env):
 				shutil.move(sslcert, newname)
 				os.rmdir(d)

+def migration_11(env):
+	# Archive the old Let's Encrypt account directory managed by free_tls_certificates
+	# because we'll use that path now for the directory managed by certbot.
+	try:
+		old_path = os.path.join(env["STORAGE_ROOT"], 'ssl', 'lets_encrypt')
+		new_path = os.path.join(env["STORAGE_ROOT"], 'ssl', 'lets_encrypt-old')
+		shutil.move(old_path, new_path)
+	except:
+		# meh
+		pass
+
+def migration_12(env):
+	# Upgrading to Carddav Roundcube plugin to version 3+, it requires the carddav_*
+        # tables to be dropped.
+        # Checking that the roundcube database already exists.
+        if os.path.exists(os.path.join(env["STORAGE_ROOT"], "mail/roundcube/roundcube.sqlite")):
+            import sqlite3
+            conn = sqlite3.connect(os.path.join(env["STORAGE_ROOT"], "mail/roundcube/roundcube.sqlite"))
+            c = conn.cursor()
+            # Get a list of all the tables that begin with 'carddav_'
+            c.execute("SELECT name FROM sqlite_master WHERE type = ? AND name LIKE ?", ('table', 'carddav_%'))
+            carddav_tables = c.fetchall()
+            # If there were tables that begin with 'carddav_', drop them
+            if carddav_tables:
+                for table in carddav_tables:
+                    try:
+                        table = table[0]
+                        c = conn.cursor()
+                        dropcmd = "DROP TABLE %s" % table
+                        c.execute(dropcmd)
+                    except:
+                        print("Failed to drop table", table, e)
+            # Save.
+            conn.commit()
+            conn.close()
+
+            # Delete all sessions, requring users to login again to recreate carddav_*
+            # databases
+            conn = sqlite3.connect(os.path.join(env["STORAGE_ROOT"], "mail/roundcube/roundcube.sqlite"))
+            c = conn.cursor()
+            c.execute("delete from session;")
+            conn.commit()
+            conn.close()
+
+
 def get_current_migration():
 	ver = 0
 	while True:
--- a/setup/munin.sh
+++ b/setup/munin.sh
@@ -38,8 +38,10 @@ chown munin. /var/log/munin/munin-cgi-html.log
 chown munin. /var/log/munin/munin-cgi-graph.log

 # ensure munin-node knows the name of this machine
+# and reduce logging level to warning
 tools/editconf.py /etc/munin/munin-node.conf -s \
-	host_name=$PRIMARY_HOSTNAME
+	host_name=$PRIMARY_HOSTNAME \
+	log_level=1

 # Update the activated plugins through munin's autoconfiguration.
 munin-node-configure --shell --remove-also 2>/dev/null | sh
--- a/setup/owncloud.sh
+++ b/setup/owncloud.sh
@@ -19,7 +19,7 @@ apt-get purge -qq -y owncloud*

 apt_install php7.0 php7.0-fpm \
 	php7.0-cli php7.0-sqlite php7.0-gd php7.0-imap php7.0-curl php-pear php-apc curl \
-        php7.0-dev php7.0-gd memcached php7.0-memcached php7.0-xml php7.0-mbstring php7.0-zip php7.0-apcu
+        php7.0-dev php7.0-gd php7.0-xml php7.0-mbstring php7.0-zip php7.0-apcu php7.0-json php7.0-intl

 # Migrate <= v0.10 setups that stored the ownCloud config.php in /usr/local rather than
 # in STORAGE_ROOT. Move the file to STORAGE_ROOT.
@@ -57,11 +57,11 @@ InstallNextcloud() {
 	# their github repositories.
 	mkdir -p /usr/local/lib/owncloud/apps

-	wget_verify https://github.com/nextcloud/contacts/releases/download/v1.5.3/contacts.tar.gz 78c4d49e73f335084feecd4853bd8234cf32615e /tmp/contacts.tgz
+	wget_verify https://github.com/nextcloud/contacts/releases/download/v2.1.5/contacts.tar.gz b7460d15f1b78d492ed502d778c0c458d503ba17 /tmp/contacts.tgz
 	tar xf /tmp/contacts.tgz -C /usr/local/lib/owncloud/apps/
 	rm /tmp/contacts.tgz

-	wget_verify https://github.com/nextcloud/calendar/releases/download/v1.5.3/calendar.tar.gz b370352d1f280805cc7128f78af4615f623827f8 /tmp/calendar.tgz
+	wget_verify https://github.com/nextcloud/calendar/releases/download/v1.6.1/calendar.tar.gz f93a247cbd18bc624f427ba2a967d93ebb941f21 /tmp/calendar.tgz
 	tar xf /tmp/calendar.tgz -C /usr/local/lib/owncloud/apps/
 	rm /tmp/calendar.tgz

@@ -107,12 +107,12 @@ InstallOwncloud() {
 	rm -rf /usr/local/lib/owncloud

 	# Download and verify
-	wget_verify https://download.owncloud.org/community/owncloud-$version.zip $hash /tmp/owncloud.zip
+	wget_verify https://download.owncloud.org/community/owncloud-$version.tar.bz2 $hash /tmp/owncloud.tar.bz2


 	# Extract ownCloud
-	unzip -q /tmp/owncloud.zip -d /usr/local/lib
-	rm -f /tmp/owncloud.zip
+	tar xjf /tmp/owncloud.tar.bz2 -C /usr/local/lib
+	rm -f /tmp/owncloud.tar.bz2

 	# The two apps we actually want are not in Nextcloud core. Download the releases from
 	# their github repositories.
@@ -154,8 +154,8 @@ InstallOwncloud() {
 	fi
 }

-owncloud_ver=12.0.3
-owncloud_hash=beab41f6a748a43f0accfa6a9808387aef718c61
+owncloud_ver=13.0.6
+owncloud_hash=33e41f476f0e2be5dc7cdb9d496673d9647aa3d6

 # Check if Nextcloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
 if [ ! -d /usr/local/lib/owncloud/ ] \
@@ -183,13 +183,13 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 	# We only need to check if we do upgrades when owncloud/Nextcloud was previously installed
 	if [ -e /usr/local/lib/owncloud/version.php ]; then
 		if grep -q "OC_VersionString = '8\.1\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running 8.1.x, upgrading to 8.2.3 first"
-			InstallOwncloud 8.2.3 bfdf6166fbf6fc5438dc358600e7239d1c970613
+			echo "We are running 8.1.x, upgrading to 8.2.11 first"
+			InstallOwncloud 8.2.11 e4794938fc2f15a095018ba9d6ee18b53f6f299c
 		fi

 		# If we are upgrading from 8.2.x we should go to 9.0 first. Owncloud doesn't support skipping minor versions
 		if grep -q "OC_VersionString = '8\.2\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running version 8.2.x, upgrading to 9.0.2 first"
+			echo "We are running version 8.2.x, upgrading to 9.0.11 first"

 			# We need to disable memcached. The upgrade and install fails
 			# with memcached
@@ -207,8 +207,8 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 EOF
 			chown www-data.www-data $STORAGE_ROOT/owncloud/config.php

-			# We can now install owncloud 9.0.2
-			InstallOwncloud 9.0.2 72a3d15d09f58c06fa8bee48b9e60c9cd356f9c5
+			# We can now install owncloud 9.0.11
+			InstallOwncloud 9.0.11 fc8bad8a62179089bc58c406b28997fb0329337b

 			# The owncloud 9 migration doesn't migrate calendars and contacts
 			# The option to migrate these are removed in 9.1
@@ -224,20 +224,32 @@ EOF

 		# If we are upgrading from 9.0.x we should go to 9.1 first.
 		if grep -q "OC_VersionString = '9\.0\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running ownCloud 9.0.x, upgrading to ownCloud 9.1.4 first"
-			InstallOwncloud 9.1.4 e637cab7b2ca3346164f3506b1a0eb812b4e841a
+			echo "We are running ownCloud 9.0.x, upgrading to ownCloud 9.1.7 first"
+			InstallOwncloud 9.1.7 1307d997d0b23dc42742d315b3e2f11423a9c808
 		fi

-		# If we are upgrading from 9.1.x we should go to Nextcloud 10.0 first.
+		# Newer ownCloud 9.1.x versions cannot be upgraded to Nextcloud 10 and have to be
+		# upgraded to Nextcloud 11 straight away, see:
+		# https://github.com/nextcloud/server/issues/2203
+		# However, for some reason, upgrading to the latest Nextcloud 11.0.7 doesn't
+		# work either. Therefore, we're upgrading to Nextcloud 11.0.0 in the interim.
+		# This should not be a problem since we're upgrading to the latest Nextcloud 12
+		# in the next step.
 		if grep -q "OC_VersionString = '9\.1\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running ownCloud 9.1.x, upgrading to Nextcloud 10.0.5 first"
-			InstallNextcloud 10.0.5 686f6a8e9d7867c32e3bf3ca63b3cc2020564bf6
+			echo "We are running ownCloud 9.1.x, upgrading to Nextcloud 11.0.0 first"
+			InstallNextcloud 11.0.0 e8c9ebe72a4a76c047080de94743c5c11735e72e
 		fi

 		# If we are upgrading from 10.0.x we should go to Nextcloud 11.0 first.
 		if grep -q "OC_VersionString = '10\.0\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running Nextcloud 10.0.x, upgrading to Nextcloud 11.0.3 first"
-			InstallNextcloud 11.0.3 a396aaa1c9f920099a90a86b4a9cd0ec13083c99
+			echo "We are running Nextcloud 10.0.x, upgrading to Nextcloud 11.0.7 first"
+			InstallNextcloud 11.0.7 f936ddcb2ae3dbb66ee4926eb8b2ebbddc3facbe
+		fi
+
+		# If we are upgrading from Nextcloud 11 we should go to Nextcloud 12 first.
+		if grep -q "OC_VersionString = '11\." /usr/local/lib/owncloud/version.php; then
+			echo "We are running Nextcloud 11, upgrading to Nextcloud 12.0.5 first"
+			InstallNextcloud 12.0.5 d25afbac977a4e331f5e38df50aed0844498ca86
 		fi
 	fi

--- a/setup/questions.sh
+++ b/setup/questions.sh
@@ -12,7 +12,9 @@ if [ -z "$NONINTERACTIVE" ]; then
 		apt_get_quiet install dialog python3 python3-pip  || exit 1
 	fi

-	# email_validator is repeated in setup/management.sh
+	# Installing email_validator is repeated in setup/management.sh, but in setup/management.sh
+	# we install it inside a virtualenv. In this script, we don't have the virtualenv yet
+	# so we install the python package globally.
 	hide_output pip3 install "email_validator>=1.0.0" || exit 1

 	message_box "Mail-in-a-Box Installation" \
@@ -49,7 +51,7 @@ you really want.
 			# user hit ESC/cancel
 			exit
 		fi
-		while ! management/mailconfig.py validate-email "$EMAIL_ADDR"
+		while ! python3 management/mailconfig.py validate-email "$EMAIL_ADDR"
 		do
 			input_box "Your Email Address" \
 				"That's not a valid email address.\n\nWhat email address are you setting this box up to manage?" \
--- a/setup/start.sh
+++ b/setup/start.sh
@@ -14,7 +14,7 @@ source setup/preflight.sh
 # Python may not be able to read/write files. This is also
 # in the management daemon startup script and the cron script.

-if [ -z `locale -a | grep en_US.utf8` ]; then
+if ! locale -a | grep en_US.utf8 > /dev/null; then
    # Generate locale if not exists
    hide_output locale-gen en_US.UTF-8
 fi
@@ -127,13 +127,23 @@ tools/web_update
 # fail2ban was first configured, but they should exist now.
 restart_service fail2ban

-# If DNS is already working, try to provision TLS certficates from Let's Encrypt.
-# Suppress extra reasons why domains aren't getting a new certificate.
-management/ssl_certificates.py -q
-
 # If there aren't any mail users yet, create one.
 source setup/firstuser.sh

+# Register with Let's Encrypt, including agreeing to the Terms of Service.
+# We'd let certbot ask the user interactively, but when this script is
+# run in the recommended curl-pipe-to-bash method there is no TTY and
+# certbot will fail if it tries to ask.
+if [ ! -d $STORAGE_ROOT/ssl/lets_encrypt/accounts/acme-v02.api.letsencrypt.org/ ]; then
+echo
+echo "-----------------------------------------------"
+echo "Mail-in-a-Box uses Let's Encrypt to provision free SSL/TLS certificates"
+echo "to enable HTTPS connections to your box. We're automatically"
+echo "agreeing you to their subscriber agreement. See https://letsencrypt.org."
+echo
+certbot register --register-unsafely-without-email --agree-tos --config-dir $STORAGE_ROOT/ssl/lets_encrypt
+fi
+
 # Done.
 echo
 echo "-----------------------------------------------"
--- a/setup/system.sh
+++ b/setup/system.sh
@@ -68,17 +68,10 @@ then
 	fi
 fi

-# ### Add Mail-in-a-Box's PPA.
-
-# We've built several .deb packages on our own that we want to include.
-# One is a replacement for Ubuntu's stock postgrey package that makes
-# some enhancements. The other is dovecot-lucene, a Lucene-based full
-# text search plugin for (and by) dovecot, which is not available in
-# Ubuntu currently.
-#
-# So, first ensure add-apt-repository is installed, then use it to install
-# the [mail-in-a-box ppa](https://launchpad.net/~mail-in-a-box/+archive/ubuntu/ppa).
+# ### Add PPAs.

+# We install some non-standard Ubuntu packages maintained by us and other
+# third-party providers. First ensure add-apt-repository is installed.

 if [ ! -f /usr/bin/add-apt-repository ]; then
 	echo "Installing add-apt-repository..."
@@ -86,11 +79,21 @@ if [ ! -f /usr/bin/add-apt-repository ]; then
 	apt_install software-properties-common
 fi

+# [Main-in-a-Box's own PPA](https://launchpad.net/~mail-in-a-box/+archive/ubuntu/ppa)
+# holds several .deb packages that we built on our own.
+# One is a replacement for Ubuntu's stock postgrey package that makes
+# some enhancements. The other is dovecot-lucene, a Lucene-based full
+# text search plugin for (and by) dovecot, which is not available in
+# Ubuntu currently.
+
 hide_output add-apt-repository -y ppa:mail-in-a-box/ppa
+hide_output add-apt-repository -y ppa:certbot/certbot

 # ### Update Packages

-# Update system packages to make sure we have the latest upstream versions of things from Ubuntu.
+# Update system packages to make sure we have the latest upstream versions
+# of things from Ubuntu, as well as the directory of packages provide by the
+# PPAs so we can install those packages later.

 echo Updating system packages...
 hide_output apt-get update
@@ -123,7 +126,7 @@ echo Installing system packages...
 apt_install python3 python3-dev python3-pip \
 	netcat-openbsd wget curl git sudo coreutils bc \
 	haveged pollinate unzip \
-	unattended-upgrades cron ntp fail2ban
+	unattended-upgrades cron ntp fail2ban rsyslog

 # ### Add PHP7 PPA

--- a/setup/webmail.sh
+++ b/setup/webmail.sh
@@ -22,7 +22,7 @@ source /etc/mailinabox.conf # load global vars
 echo "Installing Roundcube (webmail)..."
 apt_install \
 	dbconfig-common \
-	php7.0-cli php7.0-sqlite php7.0-mcrypt php7.0-intl php7.0-json php7.0-common \
+	php7.0-cli php7.0-sqlite php7.0-mcrypt php7.0-intl php7.0-json php7.0-common php7.0-curl \
 	php7.0-gd php7.0-pspell tinymce libjs-jquery libjs-jquery-mousewheel libmagic1 php7.0-mbstring

 apt_get_quiet remove php-mail-mimedecode # no longer needed since Roundcube 1.1.3
@@ -35,12 +35,12 @@ apt-get purge -qq -y roundcube* #NODOC
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of plugins to track
 # whether we have the latest version of everything.
-VERSION=1.3.3
-HASH=903a4eb1bfc25e9a08d782a7f98502cddfa579de
+VERSION=1.3.8
+HASH=90c7900ccf7b2f46fe49c650d5adb9b85ee9cc22
 PERSISTENT_LOGIN_VERSION=dc5ca3d3f4415cc41edb2fde533c8a8628a94c76
 HTML5_NOTIFIER_VERSION=4b370e3cd60dabd2f428a26f45b677ad1b7118d5
-CARDDAV_VERSION=2.0.4
-CARDDAV_HASH=d93f3cfb3038a519e71c7c3212c1d16f5da609a4
+CARDDAV_VERSION=3.0.3
+CARDDAV_HASH=d1e3b0d851ffa2c6bd42bf0c04f70d0e1d0d78f8

 UPDATE_KEY=$VERSION:$PERSISTENT_LOGIN_VERSION:$HTML5_NOTIFIER_VERSION:$CARDDAV_VERSION

@@ -155,6 +155,7 @@ cat > ${RCM_PLUGIN_DIR}/carddav/config.inc.php <<EOF;
 	 'preemptive_auth' => '1',
 	 'hide'        =>  false,
 );
+?>
 EOF

 # Create writable directories.
--- a/setup/zpush.sh
+++ b/setup/zpush.sh
@@ -22,7 +22,8 @@ apt_install \
 phpenmod -v php7.0 imap

 # Copy Z-Push into place.
-VERSION=2.3.8
+VERSION=2.4.4
+TARGETHASH=104d44426852429dac8ec2783a4e9ad7752d4682
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/z-push/version ]; then
 	needs_update=1 #NODOC
@@ -31,13 +32,14 @@ elif [[ $VERSION != `cat /usr/local/lib/z-push/version` ]]; then
 	needs_update=1 #NODOC
 fi
 if [ $needs_update == 1 ]; then
-	rm -rf /usr/local/lib/z-push
+	# Download
+	wget_verify "https://stash.z-hub.io/rest/api/latest/projects/ZP/repos/z-push/archive?at=refs%2Ftags%2F$VERSION&format=zip" $TARGETHASH /tmp/z-push.zip

-	git_clone https://stash.z-hub.io/scm/zp/z-push.git $VERSION '' /tmp/z-push
-
-	mkdir /usr/local/lib/z-push
-	cp -r /tmp/z-push/src/* /usr/local/lib/z-push
-	rm -rf /tmp/z-push
+	# Extract into place.
+	rm -rf /usr/local/lib/z-push /tmp/z-push
+	unzip -q /tmp/z-push.zip -d /tmp/z-push
+	mv /tmp/z-push/src /usr/local/lib/z-push
+	rm -rf /tmp/z-push.zip /tmp/z-push

 	rm -f /usr/sbin/z-push-{admin,top}
 	ln -s /usr/local/lib/z-push/z-push-admin.php /usr/sbin/z-push-admin
--- a/tools/owncloud-unlockadmin.sh
+++ b/tools/owncloud-unlockadmin.sh
@@ -20,4 +20,4 @@ echo
 echo Press enter to continue.
 read

-sqlite3 $STORAGE_ROOT/owncloud/owncloud.db "INSERT OR IGNORE INTO oc_group_user VALUES ('admin', '$ADMIN')" && echo Done.
+sudo -u www-data php /usr/local/lib/owncloud/occ group:adduser admin $ADMIN && echo Done.
--- a/tools/update-subresource-integrity.py
+++ b/tools/update-subresource-integrity.py
@@ -1,24 +0,0 @@
-#!/usr/bin/python3
-# Updates subresource integrity attributes in management/templates/index.html
-# to prevent CDN-hosted resources from being used as an attack vector. Run this
-# after updating the Bootstrap and jQuery <link> and <script> to compute the
-# appropriate hash and insert it into the template.
-
-import re, urllib.request, hashlib, base64
-
-fn = "management/templates/index.html"
-
-with open(fn, 'r') as f:
-	content = f.read()
-
-def make_integrity(url):
-	resource = urllib.request.urlopen(url).read()
-	return "sha256-" + base64.b64encode(hashlib.sha256(resource).digest()).decode('ascii')
-
-content = re.sub(
-	r'<(link rel="stylesheet" href|script src)="(.*?)" integrity="(.*?)"',
-	lambda m : '<' + m.group(1) + '="' + m.group(2) + '" integrity="' + make_integrity(m.group(2)) + '"',
-	content)
-
-with open(fn, 'w') as f:
-	f.write(content)
Author	SHA1	Message	Date
Joshua Tauberer	7b592b1e99	v0.30 - the last Ubuntu 14.04 release	2019-01-09 06:31:56 -05:00
Joshua Tauberer	a67aa4cfd4	changelog	2019-01-09 06:17:27 -05:00
jvolkenant	b7e9a90005	roundcube: upgrade carddav plugin to 3.0.3 & updated migrate.py (#1479 ) * roundcube: upgrade carddav plugin to 3.0.3 & updated migrate.py * Check for db first and clear sessions to force re-login	2018-12-03 15:33:36 -05:00
Joshua Tauberer	b3b798adf2	changelog entries	2018-12-02 18:03:17 -05:00
Joshua Tauberer	bd54b41041	add missing rsyslog to apt install line see #1438	2018-12-02 18:02:00 -05:00
Joshua Tauberer	a211ad422b	add a note on the aliases page that aliases should not be used to forward to outside domains fixes #1198	2018-12-02 18:02:00 -05:00
Joshua Tauberer	ef28a1defd	show the Mail-in-a-Box version in the system status checks even when the new-version check is disabled fixes #922	2018-12-02 18:02:00 -05:00
Joshua Tauberer	c5c413b447	remove user account mailbox size from the control panel because it takes way too long to compute on very large mailboxes fixes #531	2018-12-02 18:02:00 -05:00
Joshua Tauberer	d2beb3919b	document password character limitation fixes #407	2018-12-02 18:02:00 -05:00
Achilleas Pipinellis	a7dded8182	Add a logfile entry to the NSD conf file (#1434 ) Having a log file can help debugging when something goes wrong and NSD doesn't fail or MiaB doesn't notify you. See https://discourse.mailinabox.email/t/dns-email-domain-becomes-inaccessible-every-few-hours/3770	2018-12-02 18:00:16 -05:00
jeff-h	000363492e	Improve greylisting explanation. (#1447 ) Hopefully this improves the accuracy of the greylisting description.	2018-12-02 17:58:26 -05:00
jeff-h	5be74dec6e	Improve postgrey logging (#1448 ) We can't presume the redelivery timeframe of the sending server. However, we do know the blacklist timeframe within which we will reject a redelivery.	2018-12-02 17:57:37 -05:00
Holger Just	0335595e7e	Update Roundcube to version 1.3.8 (#1475 ) https://github.com/roundcube/roundcubemail/releases/tag/1.3.8	2018-11-25 10:40:21 -05:00
Joshua Tauberer	16f38042ec	v0.29 released, closes #1440	2018-10-24 16:12:25 -04:00
Joshua Tauberer	2f494e9a1c	CHANGELOG fixes/updates	2018-10-24 16:09:59 -04:00
Michael Kroes	6eb9055275	Upgrade NextCloud to 13.06 (#1436 )	2018-10-09 07:09:54 -04:00
Joshua Tauberer	504a9b0abc	certbot uses a new directory path for API v02 accounts and we should check that before creating a new account or else we'll try to create a new account on each setup run (which certbot just fails on)	2018-09-03 13:07:24 -04:00
Joshua Tauberer	842fbb3d72	auto-agree to Let's Encrypt's terms of service during setup fixes #1409 This reverts commit `82844ca651` ("make certbot auto-agree to TOS if NONINTERACTIVE=1 env var is set (#1399)") and instead always auto-agree. If we don't auto-agree, certbot asks the user interactively, but our "curl \| bash" setup line does not permit interactive prompts, so certbot failed to register and all certificate things were broken until the command was re-run interactively.	2018-09-03 13:06:34 -04:00
Joshua Tauberer	a5d5a073c7	update Z-Push to 2.4.4 Starting with 2.4, Z-Push no longer provides tarballs on their download server. The only options are getting the code from their git repository or using one of their distribution packages. Their Ubuntu 18.04 packaes don't seem to actually work in Ubuntu 18.04, so thinking ahead that's currently a bad choice. In `78d1c9be6e` we switched from doing a git clone to using wget on their downloads server because of a problem with something related to stash.z-hub.io's SSL certificate. But wget also seems to work on their source code repository, so we can use that.	2018-09-02 11:29:44 -04:00
Joshua Tauberer	d4b122ee94	update to Nextcloud 13.0.5	2018-08-24 11:11:52 -04:00
Joshua Tauberer	052a1f3b26	update to Roundcube 1.3.7	2018-08-24 10:47:22 -04:00
Joshua Tauberer	180b054dbc	small code cleanup testing if the utf8 locale is installed	2018-08-24 09:49:08 -04:00
Joshua Tauberer	cb162da5fe	Merge pull request #1412 from hlxnd/pr Use ISO 8601 on backups table dates, fixes #1397	2018-08-05 15:16:05 -04:00
hlxnd	de9c556ad7	Add missing PHP end tag	2018-08-05 15:27:35 +02:00
hlxnd	f420294819	Use ISO 8601 on backups table dates.	2018-08-05 15:26:45 +02:00
Joshua Tauberer	738e0a6e17	v0.28 released, closes #1405	2018-07-30 11:14:38 -04:00
Pascal Garber	e0d46d1eb5	Use Nextcloud’s occ command to unlock the admin (#1406 )	2018-07-25 15:37:09 -04:00
Joshua Tauberer	7f37abca05	add php7.0-curl to webmail.sh see `7ee91f6ae6` see #1268 closes #1259	2018-07-22 09:19:36 -04:00
Joshua Tauberer	2f467556bd	new ssl cert provisioning broke if a domain doesnt yet have a cert, fixes #1392	2018-07-19 11:40:49 -04:00
Joshua Tauberer	15583ec10d	updated CHANGELOG	2018-07-19 11:27:37 -04:00
Nils Norman Haukås	78d1c9be6e	failing z-push installation: replace git clone with wget_verify git clone (which uses curl) underneath was failing. Curiously, the same git clone command would work on my macos host machine. From the screenshot it looks like curl was somehow not able to negotiate the connection. Might have been a missing CA certificate for Comodo, but I was not able to determine if that was the issue. fixes #1393 closes #1387 closes #1400	2018-07-19 11:25:57 -04:00
dev9	b0b5d8e792	Fix .mobileconfig so CalDAV calendar works on Mac OS X (#1402 ) The previous CalDAVPrincipalURL "/cloud/remote.php/caldav/calendars/" causes an error in OS X. See: https://discourse.mailinabox.email/t/caldav-with-macos-10-12-2-does-not-work/1649 and other similar issues. The correct CalDAVPrincipalURL: https://discourse.mailinabox.email/t/caldav-with-macos-10-12-2-does-not-work/1649 but it turns out you can just leave the key/value out completely and OS X/iOS are able to auto discover the correct URL.	2018-07-19 11:17:38 -04:00
Nils	82844ca651	make certbot auto-agree to TOS if NONINTERACTIVE=1 env var is set (#1399 )	2018-07-15 11:24:15 -04:00
Joshua Tauberer	2a72c800f6	replace free_tls_certificates with certbot	2018-06-29 16:46:21 -04:00
Joshua Tauberer	8be23d5ef6	ssl_certificates: reuse query_dns function in status_checks and simplify calls by calling normalize_ip within query_dns	2018-06-29 16:46:21 -04:00
Joshua Tauberer	f9a0e39cc9	cryptography is now distributed as a wheel and no longer needs system development packages to be installed or pip/setuptools workarounds	2018-06-29 16:46:21 -04:00
Joshua Tauberer	0c0a079354	v0.27	2018-06-14 07:49:20 -04:00
Joshua Tauberer	42e86610ba	changelog entry	2018-05-12 09:43:41 -04:00
yeah	7c62f4b8e9	Update Roundcube to 1.3.6 (#1376 )	2018-04-17 11:54:24 -04:00
Joshua Tauberer	1eba7b0616	send the mail_log.py report to the box admin every Monday	2018-02-25 11:55:06 -05:00
Joshua Tauberer	9c7820f422	mail_log.py: include sent mail in the logins report in a new smtp column	2018-02-24 09:24:15 -05:00
Joshua Tauberer	87ec4e9f82	mail_log.py: refactor the dovecot login collector	2018-02-24 09:24:14 -05:00
Joshua Tauberer	08becf7fa3	the hidden feature for proxying web requests now sets X-Forwarded-For	2018-02-24 09:24:14 -05:00
Joshua Tauberer	5eb4a53de1	remove old tools/update-subresource-integrity.py script which isn't used now that we download all admin page remote assets during setup	2018-02-24 09:24:14 -05:00
Joshua Tauberer	598ade3f7a	changelog entry	2018-02-24 09:24:09 -05:00
xetorixik	8f399df5bb	Update Roundcube to 1.3.4 and Z-push to 2.3.9 (#1354 )	2018-02-21 08:22:57 -05:00
Joshua Tauberer	ae73dc5d30	v0.26c	2018-02-13 10:46:02 -05:00
Joshua Tauberer	c409b2efd0	CHANGELOG entries	2018-02-13 10:44:07 -05:00
Joshua Tauberer	6961840c0e	wrap wget in hide_output so that wget errors are shown Our wget_verify function uses wget to download a file and then check the file's hash. If wget fails, i.e. because of a 404 or other HTTP or network error, we exited setup without displaying any output because normally there are no errors and -q keeps the setup output clean. Wrapping wget with our hide_output function, and dropping -q, captures wget's output and shows it and exits setup just if wget fails. see #1297	2018-02-13 10:38:10 -05:00
yeah	6162a9637c	Add some development instructions to CONTRIBUTING.md (#1348 )	2018-02-05 08:41:19 -05:00
Jan Schulz-Hofen	47c968e71b	Upgrade Nextcloud from 12.0.3 to 12.0.5	2018-02-04 10:13:30 -05:00
Jan Schulz-Hofen	ed3e2aa712	Use new .tar.bz2 source files for ownCloud and fix upgrade paths	2018-02-04 10:13:30 -05:00
NatCC	fe597da7aa	Update users.html (#1345 ) Passwords must be eight characters long; when passwords are changed via the users page the dialog states that passwords need to be at least four characters but only eight or more are acceptable.	2018-02-03 17:49:11 -05:00
Joshua Tauberer	61e9888a85	Cdon't try to generate a CSR in the control panel until both the domain and country are selected Fixes #1338. See 0e9680fda63c33ace3f34ca7126617fb0efe8ffc, `a52c56e571`.	2018-01-28 09:08:24 -05:00
Joshua Tauberer	35fed8606e	only spawn one process for the management daemon In `0088fb4553` I changed the management daemon's startup script from a symlink to a Python script to a bash script that activated the new virtualenv and then launched Python. As a result, the init.d script that starts the daemon would write the pid of bash to the pidfile, and when trying to kill it, it would kill bash but not the Python process. Using exec to start Python fixes this problem by making the Python process have the pid that the init.d script knows about. fixes #1339	2018-01-28 09:08:19 -05:00
Joshua Tauberer	ef6f121491	when generating a CSR in the control panel, don't set empty attributes Same as in `a52c56e571`. Fixes #1338.	2018-01-28 09:07:54 -05:00
Joshua Tauberer	ec3aab0eaa	v0.26b	2018-01-25 09:27:17 -05:00
Joshua Tauberer	8c69b9e261	update CHANGELOG	2018-01-25 09:23:04 -05:00
Joshua Tauberer	e7150e3bc6	pin acme to v0.20, which is the last version compatible with free_tls_certificates free_tls_certificates uses acme.jose, which in acme v0.21 was moved to a new Python package. See #1328	2018-01-20 11:23:45 -05:00
Joshua Tauberer	8d6d84d87f	run mailconfig.py's email address validator outside of the virtualenv during questions.sh We don't have the virtualenv this early in setup. Broken by `0088fb4553`. Fixes #1326. See https://discourse.mailinabox.email/t/that-is-not-a-valid-email-error-during-mailinabox-installation/2793.	2018-01-20 10:59:37 -05:00
barrybingo	a6a1cc7ae0	Reduce munin-node log level to warning (#1330 )	2018-01-19 12:00:44 -05:00