v0.27

Our wget_verify function uses wget to download a file and then check
the file's hash. If wget fails, i.e. because of a 404 or other HTTP
or network error, we exited setup without displaying any output because
normally there are no errors and -q keeps the setup output clean.

Wrapping wget with our hide_output function, and dropping -q, captures
wget's output and shows it and exits setup just if wget fails.

see #1297

CHANGELOG.md

@@ -1,6 +1,38 @@
 CHANGELOG
 =========
 
+v0.27 (June 14, 2018)
+---------------------
+
+* A report of box activity, including sent/received mail totals and logins by user, is now emailed to the box's administrator user each week.
+* Update Roundcube to version 1.3.6 and Z-Push to version 2.3.9.
+* The undocumented feature for proxying web requests to another server now sets X-Forwarded-For.
+
+v0.26c (February 13, 2018)
+--------------------------
+
+Setup:
+
+* Upgrades from v0.21c (February 1, 2017) or earlier were broken because the intermediate versions of ownCloud used in setup were no longer available from ownCloud.
+* Some download errors had no output --- there is more output on error now.
+
+Control Panel:
+
+* The background service for the control panel was not restarting on updates, leaving the old version running. This was broken in v0.26 and is now fixed.
+* Installing your own TLS/SSL certificate had been broken since v0.24 because the new version of openssl became stricter about CSR generation parameters.
+* Fixed password length help text.
+
+Contacts/Calendar:
+
+* Upgraded Nextcloud from 12.0.3 to 12.0.5.
+
+v0.26b (January 25, 2018)
+-------------------------
+
+* Fix new installations which broke at the step of asking for the user's desired email address, which was broken by v0.26's changes related to the control panel.
+* Fix the provisioning of TLS certificates by pinning a Python package we rely on (acme) to an earlier version because our code isn't yet compatible with its current version.
+* Reduce munin's log_level from debug to warning to prevent massive log files.
+
 v0.26 (January 18, 2018)
 ------------------------
 
@@ -24,7 +56,7 @@ Installer:
 * We now run `apt-get autoremove` at the start of setup to clear out old packages, especially old kernels that take up a lot of space. On the first run, this step may take a long time.
 * We now fetch Z-Push from its tagged git repository, fixing an installation problem.
 * Some old PHP5 packages are removed from setup, fixing an installation bug where Apache would get installed.
-* Python 3 packages for the control panel are now installed using a virtualenv to prevent installation errors.
+* Python 3 packages for the control panel are now installed using a virtualenv to prevent installation errors due to conflicts in the cryptography/openssl packages between OS-installed packages and pip-installed packages.
 
 v0.25 (November 15, 2017)
 -------------------------

CONTRIBUTING.md

@@ -1,3 +1,50 @@
+# Contributing
+
+Mail-in-a-Box is an open source project. Your contributions and pull requests are welcome.
+
+## Development
+
+To start developing Mail-in-a-Box, [clone the repository](https://github.com/mail-in-a-box/mailinabox) and familiarize yourself with the code.
+
+    $ git clone https://github.com/mail-in-a-box/mailinabox
+
+### Vagrant and VirtualBox
+
+We recommend you use [Vagrant](https://www.vagrantup.com/intro/getting-started/install.html) and [VirtualBox](https://www.virtualbox.org/wiki/Downloads) for development. Please install them first.
+
+With Vagrant set up, the following should boot up Mail-in-a-Box inside a virtual machine:
+
+    $ vagrant up --provision
+
+_If you're seeing an error message about your *IP address being listed in the Spamhaus Block List*, simply uncomment the `export SKIP_NETWORK_CHECKS=1` line in `Vagrantfile`. It's normal, you're probably using a dynamic IP address assigned by your Internet provider–they're almost all listed._
+
+### Modifying your `hosts` file
+
+After a while, Mail-in-a-Box will be available at `192.168.50.4` (unless you changed that in your `Vagrantfile`). To be able to use the web-based bits, we recommend to add a hostname to your `hosts` file:
+
+    $ echo "192.168.50.4 mailinabox.lan" | sudo tee -a /etc/hosts
+
+You should now be able to navigate to https://mailinabox.lan/admin using your browser. There should be an initial admin user with the name `me@mailinabox.lan` and the password `12345678`.
+
+### Making changes
+
+Your working copy of Mail-in-a-Box will be mounted inside your VM at `/vagrant`. Any change you make locally will appear inside your VM automatically.
+
+Running `vagrant up --provision` again will repeat the installation with your modifications.
+
+Alternatively, you can also ssh into the VM using:
+
+    $ vagrant ssh
+
+Once inside the VM, you can re-run individual parts of the setup like in this example:
+
+    vm$ cd /vagrant
+    vm$ sudo setup/owncloud.sh # replace with script you'd like to re-run
+
+### Tests
+
+Mail-in-a-Box needs more tests. If you're still looking for a way to help out, writing and contributing tests would be a great start!
+
 ## Public domain
 
 This project is in the public domain. Copyright and related rights in the work worldwide are waived through the [CC0 1.0 Universal public domain dedication][CC0]. See the LICENSE file in this directory.

README.md

@@ -59,7 +59,7 @@ by me:
 	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
 	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported
 
-	$ git verify-tag v0.26
+	$ git verify-tag v0.27
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
@@ -72,7 +72,7 @@ and on my [personal homepage](https://razor.occams.info/). (Of course, if this r
 
 Checkout the tag corresponding to the most recent release:
 
-	$ git checkout v0.26
+	$ git checkout v0.27
 
 Begin the installation.
 
@@ -82,6 +82,12 @@ For help, DO NOT contact me directly --- I don't do tech support by email or twe
 
 Post your question on the [discussion forum](https://discourse.mailinabox.email/) instead, where me and other Mail-in-a-Box users may be able to help you.
 
+Contributing and Development
+----------------------------
+
+Mail-in-a-Box is an open source project. Your contributions and pull requests are welcome. See [CONTRIBUTING](CONTRIBUTING.md) to get started. 
+
+
 The Acknowledgements
 --------------------
 

management/daily_tasks.sh

@@ -9,6 +9,12 @@ export LC_ALL=en_US.UTF-8
 export LANG=en_US.UTF-8
 export LC_TYPE=en_US.UTF-8
 
+# On Mondays, i.e. once a week, send the administrator a report of total emails
+# sent and received so the admin might notice server abuse.
+if [ `date "+%u"` -eq 1 ]; then
+    management/mail_log.py -t week | management/email_administrator.py "Mail-in-a-Box Usage Report"
+fi
+
 # Take a backup.
 management/backup.py | management/email_administrator.py "Backup Status"
 

management/email_administrator.py

@@ -4,8 +4,14 @@
 
 import sys
 
+import html
 import smtplib
-from email.message import Message
+
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+
+# In Python 3.6:
+#from email.message import Message
 
 from utils import load_environment
 
@@ -26,11 +32,23 @@ if content == "":
 	sys.exit(0)
 
 # create MIME message
-msg = Message()
+msg = MIMEMultipart('alternative')
+
+# In Python 3.6:
+#msg = Message()
+
 msg['From'] = "\"%s\" <%s>" % (env['PRIMARY_HOSTNAME'], admin_addr)
 msg['To'] = admin_addr
 msg['Subject'] = "[%s] %s" % (env['PRIMARY_HOSTNAME'], subject)
-msg.set_payload(content, "UTF-8")
+
+content_html = "<html><body><pre>{}</pre></body></html>".format(html.escape(content))
+
+msg.attach(MIMEText(content, 'plain'))
+msg.attach(MIMEText(content_html, 'html'))
+
+# In Python 3.6:
+#msg.set_content(content)
+#msg.add_alternative(content_html, "html")
 
 # send
 smtpclient = smtplib.SMTP('127.0.0.1', 25)

management/mail_log.py

@@ -53,10 +53,10 @@ VERBOSE = False
 # List of strings to filter users with
 FILTERS = None
 
-# What to show by default
+# What to show (with defaults)
 SCAN_OUT = True  # Outgoing email
 SCAN_IN = True  # Incoming email
-SCAN_CONN = False  # IMAP and POP3 logins
+SCAN_DOVECOT_LOGIN = True  # Dovecot Logins
 SCAN_GREY = False  # Greylisted email
 SCAN_BLOCKED = False  # Rejected email
 
@@ -76,6 +76,7 @@ def scan_files(collector):
             tmp_file = tempfile.NamedTemporaryFile()
             shutil.copyfileobj(gzip.open(fn), tmp_file)
 
+        if VERBOSE:
             print("Processing file", fn, "...")
         fn = tmp_file.name if tmp_file else fn
 
@@ -105,7 +106,7 @@ def scan_mail_log(env):
         "scan_time": time.time(),  # The time in seconds the scan took
         "sent_mail": OrderedDict(),  # Data about email sent by users
         "received_mail": OrderedDict(),  # Data about email received by users
-        "dovecot": OrderedDict(),  # Data about Dovecot activity
+        "logins": OrderedDict(),  # Data about login activity
         "postgrey": {},  # Data about greylisting of email addresses
         "rejected": OrderedDict(),  # Emails that were blocked
         "known_addresses": None,  # Addresses handled by the Miab installation
@@ -119,8 +120,8 @@ def scan_mail_log(env):
     except ImportError:
         pass
 
-    print("Scanning from {:%Y-%m-%d %H:%M:%S} back to {:%Y-%m-%d %H:%M:%S}".format(
+    print("Scanning logs from {:%Y-%m-%d %H:%M:%S} to {:%Y-%m-%d %H:%M:%S}".format(
-        START_DATE, END_DATE)
+        END_DATE, START_DATE)
     )
 
     # Scan the lines in the log files until the date goes out of range
@@ -138,8 +139,8 @@ def scan_mail_log(env):
     # Print Sent Mail report
 
     if collector["sent_mail"]:
-        msg = "Sent email between {:%Y-%m-%d %H:%M:%S} and {:%Y-%m-%d %H:%M:%S}"
+        msg = "Sent email"
-        print_header(msg.format(END_DATE, START_DATE))
+        print_header(msg)
 
         data = OrderedDict(sorted(collector["sent_mail"].items(), key=email_sort))
 
@@ -173,8 +174,8 @@ def scan_mail_log(env):
     # Print Received Mail report
 
     if collector["received_mail"]:
-        msg = "Received email between {:%Y-%m-%d %H:%M:%S} and {:%Y-%m-%d %H:%M:%S}"
+        msg = "Received email"
-        print_header(msg.format(END_DATE, START_DATE))
+        print_header(msg)
 
         data = OrderedDict(sorted(collector["received_mail"].items(), key=email_sort))
 
@@ -199,43 +200,55 @@ def scan_mail_log(env):
             [accum]
         )
 
-    # Print Dovecot report
+    # Print login report
 
-    if collector["dovecot"]:
+    if collector["logins"]:
-        msg = "Email client logins between {:%Y-%m-%d %H:%M:%S} and {:%Y-%m-%d %H:%M:%S}"
+        msg = "User logins per hour"
-        print_header(msg.format(END_DATE, START_DATE))
+        print_header(msg)
 
-        data = OrderedDict(sorted(collector["dovecot"].items(), key=email_sort))
+        data = OrderedDict(sorted(collector["logins"].items(), key=email_sort))
+
+        # Get a list of all of the protocols seen in the logs in reverse count order.
+        all_protocols = defaultdict(int)
+        for u in data.values():
+            for protocol_name, count in u["totals_by_protocol"].items():
+                all_protocols[protocol_name] += count
+        all_protocols = [k for k, v in sorted(all_protocols.items(), key=lambda kv : -kv[1])]
 
         print_user_table(
             data.keys(),
             data=[
-                ("imap", [u["imap"] for u in data.values()]),
+                (protocol_name, [
-                ("pop3", [u["pop3"] for u in data.values()]),
+                    round(u["totals_by_protocol"][protocol_name] / (u["latest"]-u["earliest"]).total_seconds() * 60*60, 1)
+                    if (u["latest"]-u["earliest"]).total_seconds() > 0
+                    else 0 # prevent division by zero
+                  for u in data.values()])
+                for protocol_name in all_protocols
             ],
             sub_data=[
-                ("IMAP IP addresses", [[k + " (%d)" % v for k, v in u["imap-logins"].items()]
+                ("Protocol and Source", [[
-                                       for u in data.values()]),
+                    "{} {}: {} times".format(protocol_name, host, count)
-                ("POP3 IP addresses", [[k + " (%d)" % v for k, v in u["pop3-logins"].items()]
+                    for (protocol_name, host), count
-                                       for u in data.values()]),
+                    in sorted(u["totals_by_protocol_and_host"].items(), key=lambda kv:-kv[1])
+                  ] for u in data.values()])
             ],
             activity=[
-                ("imap", [u["activity-by-hour"]["imap"] for u in data.values()]),
+                (protocol_name, [u["activity-by-hour"][protocol_name] for u in data.values()])
-                ("pop3", [u["activity-by-hour"]["pop3"] for u in data.values()]),
+                for protocol_name in all_protocols
             ],
             earliest=[u["earliest"] for u in data.values()],
             latest=[u["latest"] for u in data.values()],
+            numstr=lambda n : str(round(n, 1)),
         )
 
-        accum = {"imap": defaultdict(int), "pop3": defaultdict(int), "both": defaultdict(int)}
+        accum = { protocol_name: defaultdict(int) for protocol_name in all_protocols }
         for h in range(24):
-            accum["imap"][h] = sum(d["activity-by-hour"]["imap"][h] for d in data.values())
+            for protocol_name in all_protocols:
-            accum["pop3"][h] = sum(d["activity-by-hour"]["pop3"][h] for d in data.values())
+              accum[protocol_name][h] = sum(d["activity-by-hour"][protocol_name][h] for d in data.values())
-            accum["both"][h] = accum["imap"][h] + accum["pop3"][h]
 
         print_time_table(
-            ["imap", "pop3", "   +"],
+            all_protocols,
-            [accum["imap"], accum["pop3"], accum["both"]]
+            [accum[protocol_name] for protocol_name in all_protocols]
         )
 
     if collector["postgrey"]:
@@ -348,9 +361,9 @@ def scan_mail_log_line(line, collector):
     elif service == "postfix/lmtp":
         if SCAN_IN:
             scan_postfix_lmtp_line(date, log, collector)
-    elif service in ("imap-login", "pop3-login"):
+    elif service.endswith("-login"):
-        if SCAN_CONN:
+        if SCAN_DOVECOT_LOGIN:
-            scan_dovecot_line(date, log, collector, service[:4])
+            scan_dovecot_login_line(date, log, collector, service[:4])
     elif service == "postgrey":
         if SCAN_GREY:
             scan_postgrey_line(date, log, collector)
@@ -448,44 +461,43 @@ def scan_postfix_smtpd_line(date, log, collector):
                 collector["rejected"][user] = data
 
 
-def scan_dovecot_line(date, log, collector, prot):
+def scan_dovecot_login_line(date, log, collector, protocol_name):
-    """ Scan a dovecot log line and extract interesting data """
+    """ Scan a dovecot login log line and extract interesting data """
 
     m = re.match("Info: Login: user=<(.*?)>, method=PLAIN, rip=(.*?),", log)
 
     if m:
         # TODO: CHECK DIT
-        user, rip = m.groups()
+        user, host = m.groups()
 
         if user_match(user):
+            add_login(user, date, protocol_name, host, collector)
+
+
+def add_login(user, date, protocol_name, host, collector):
             # Get the user data, or create it if the user is new
-            data = collector["dovecot"].get(
+            data = collector["logins"].get(
                 user,
                 {
-                    "imap": 0,
-                    "pop3": 0,
                     "earliest": None,
                     "latest": None,
-                    "imap-logins": defaultdict(int),
+                    "totals_by_protocol": defaultdict(int),
-                    "pop3-logins": defaultdict(int),
+                    "totals_by_protocol_and_host": defaultdict(int),
-                    "activity-by-hour": {
+                    "activity-by-hour": defaultdict(lambda : defaultdict(int)),
-                        "imap": defaultdict(int),
-                        "pop3": defaultdict(int),
-                    },
                 }
             )
 
-            data[prot] += 1
-            data["activity-by-hour"][prot][date.hour] += 1
-
             if data["latest"] is None:
                 data["latest"] = date
             data["earliest"] = date
 
-            if rip not in ("127.0.0.1", "::1") or True:
+            data["totals_by_protocol"][protocol_name] += 1
-                data["%s-logins" % prot][rip] += 1
+            data["totals_by_protocol_and_host"][(protocol_name, host)] += 1
 
-            collector["dovecot"][user] = data
+            if host not in ("127.0.0.1", "::1") or True:
+                data["activity-by-hour"][protocol_name][date.hour] += 1
+
+            collector["logins"][user] = data
 
 
 def scan_postfix_lmtp_line(date, log, collector):
@@ -561,6 +573,8 @@ def scan_postfix_submission_line(date, log, collector):
 
             collector["sent_mail"][user] = data
 
+            # Also log this as a login.
+            add_login(user, date, "smtp", client, collector)
 
 # Utility functions
 
@@ -640,7 +654,7 @@ def print_time_table(labels, data, do_print=True):
         for i, d in enumerate(data):
             lines[i] += base.format(d[h])
 
-    lines.insert(0, "┬")
+    lines.insert(0, "┬ totals by time of day:")
     lines.append("└" + (len(lines[-1]) - 2) * "─")
 
     if do_print:
@@ -650,7 +664,7 @@ def print_time_table(labels, data, do_print=True):
 
 
 def print_user_table(users, data=None, sub_data=None, activity=None, latest=None, earliest=None,
-                     delimit=False):
+                     delimit=False, numstr=str):
     str_temp = "{:<32} "
     lines = []
     data = data or []
@@ -764,7 +778,7 @@ def print_user_table(users, data=None, sub_data=None, activity=None, latest=None
 
     # Print totals
 
-    data_accum = [str(a) for a in data_accum]
+    data_accum = [numstr(a) for a in data_accum]
     footer = str_temp.format("Totals:" if do_accum else " ")
     for row, (l, _) in enumerate(data):
         temp = "{:>%d}" % max(5, len(l) + 1)
@@ -818,7 +832,7 @@ if __name__ == "__main__":
                         action="store_true")
     parser.add_argument("-s", "--sent", help="Scan for sent emails.",
                         action="store_true")
-    parser.add_argument("-l", "--logins", help="Scan for IMAP/POP logins.",
+    parser.add_argument("-l", "--logins", help="Scan for user logins to IMAP/POP3.",
                         action="store_true")
     parser.add_argument("-g", "--grey", help="Scan for greylisted emails.",
                         action="store_true")
@@ -863,8 +877,8 @@ if __name__ == "__main__":
         if not SCAN_OUT:
             print("Ignoring sent emails")
 
-        SCAN_CONN = args.logins
+        SCAN_DOVECOT_LOGIN = args.logins
-        if not SCAN_CONN:
+        if not SCAN_DOVECOT_LOGIN:
             print("Ignoring logins")
 
         SCAN_GREY = args.grey

management/mailconfig.py

@@ -1,5 +1,14 @@
 #!/usr/local/lib/mailinabox/env/bin/python
 
+# NOTE:
+# This script is run both using the system-wide Python 3
+# interpreter (/usr/bin/python3) as well as through the
+# virtualenv (/usr/local/lib/mailinabox/env). So only
+# import packages at the top level of this script that
+# are installed in *both* contexts. We use the system-wide
+# Python 3 in setup/questions.sh to validate the email
+# address entered by the user.
+
 import subprocess, shutil, os, sqlite3, re
 import utils
 from email_validator import validate_email as validate_email_, EmailNotValidError

management/ssl_certificates.py

@@ -556,7 +556,7 @@ def create_csr(domain, ssl_key, country_code, env):
                 "openssl", "req", "-new",
                 "-key", ssl_key,
                 "-sha256",
-                "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (country_code, domain)])
+                "-subj", "/C=%s/CN=%s" % (country_code, domain)])
 
 def install_cert(domain, ssl_cert, ssl_chain, env, raw=False):
 	# Write the combined cert+chain to a temporary path and validate that it is OK.

management/templates/ssl.html

@@ -159,7 +159,11 @@ function ssl_install(elem) {
 }
 
 function show_csr() {
+ // Can't show a CSR until both inputs are entered.
  if ($('#ssldomain').val() == "") return;
+ if ($('#sslcc').val() == "") return;
+
+ // Scroll to it and fetch.
  $('#csr_info').slideDown();
  $('#ssl_csr').text('Loading...');
  api(

management/templates/users.html

@@ -213,7 +213,7 @@ function users_set_password(elem) {
 
   show_modal_confirm(
     "Set Password",
-    $("<p>Set a new password for <b>" + email + "</b>?</p> <p><label for='users_set_password_pw' style='display: block; font-weight: normal'>New Password:</label><input type='password' id='users_set_password_pw'></p><p><small>Passwords must be at least four characters and may not contain spaces.</small>" + yourpw + "</p>"),
+    $("<p>Set a new password for <b>" + email + "</b>?</p> <p><label for='users_set_password_pw' style='display: block; font-weight: normal'>New Password:</label><input type='password' id='users_set_password_pw'></p><p><small>Passwords must be at least eight characters and may not contain spaces.</small>" + yourpw + "</p>"),
     "Set Password",
     function() {
       api(

management/web_update.py

@@ -149,7 +149,10 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 
 			# any proxy or redirect here?
 			for path, url in yaml.get("proxies", {}).items():
-				nginx_conf_extra += "\tlocation %s {\n\t\tproxy_pass %s;\n\t}\n" % (path, url)
+				nginx_conf_extra += "\tlocation %s {" % path
+				nginx_conf_extra += "\n\t\tproxy_pass %s;" % url
+				nginx_conf_extra += "\n\t\tproxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;"
+				nginx_conf_extra += "\n\t}\n"
 			for path, url in yaml.get("redirects", {}).items():
 				nginx_conf_extra += "\trewrite %s %s permanent;\n" % (path, url)
 

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.26
+	TAG=v0.27
 fi
 
 # Are we running as root?

setup/functions.sh

@@ -179,7 +179,7 @@ function wget_verify {
 	DEST=$3
 	CHECKSUM="$HASH  $DEST"
 	rm -f $DEST
-	wget -q -O $DEST $URL || exit 1
+	hide_output wget -O $DEST $URL
 	if ! echo "$CHECKSUM" | sha1sum --check --strict > /dev/null; then
 		echo "------------------------------------------------------------"
 		echo "Download of $URL did not match expected checksum."

setup/management.sh

@@ -59,7 +59,7 @@ hide_output $venv/bin/pip install --upgrade setuptools
 hide_output $venv/bin/pip install --upgrade \
 	rtyaml "email_validator>=1.0.0" "free_tls_certificates>=0.1.3" "exclusiveprocess" \
 	flask dnspython python-dateutil \
-	"idna>=2.0.0" "cryptography>=1.0.2" acme boto psutil
+	"idna>=2.0.0" "cryptography>=1.0.2" "acme==0.20.0" boto psutil
 
 # CONFIGURATION
 
@@ -100,7 +100,7 @@ rm -f /usr/local/bin/mailinabox-daemon # old path
 cat > $inst_dir/start <<EOF;
 #!/bin/bash
 source $venv/bin/activate
-python `pwd`/management/daemon.py
+exec python `pwd`/management/daemon.py
 EOF
 chmod +x $inst_dir/start
 rm -f /etc/init.d/mailinabox

setup/munin.sh

@@ -38,8 +38,10 @@ chown munin. /var/log/munin/munin-cgi-html.log
 chown munin. /var/log/munin/munin-cgi-graph.log
 
 # ensure munin-node knows the name of this machine
+# and reduce logging level to warning
 tools/editconf.py /etc/munin/munin-node.conf -s \
-	host_name=$PRIMARY_HOSTNAME
+	host_name=$PRIMARY_HOSTNAME \
+	log_level=1
 
 # Update the activated plugins through munin's autoconfiguration.
 munin-node-configure --shell --remove-also 2>/dev/null | sh

setup/owncloud.sh

@@ -107,12 +107,12 @@ InstallOwncloud() {
 	rm -rf /usr/local/lib/owncloud
 
 	# Download and verify
-	wget_verify https://download.owncloud.org/community/owncloud-$version.zip $hash /tmp/owncloud.zip
+	wget_verify https://download.owncloud.org/community/owncloud-$version.tar.bz2 $hash /tmp/owncloud.tar.bz2
 
 
 	# Extract ownCloud
-	unzip -q /tmp/owncloud.zip -d /usr/local/lib
+	tar xjf /tmp/owncloud.tar.bz2 -C /usr/local/lib
-	rm -f /tmp/owncloud.zip
+	rm -f /tmp/owncloud.tar.bz2
 
 	# The two apps we actually want are not in Nextcloud core. Download the releases from
 	# their github repositories.
@@ -154,8 +154,8 @@ InstallOwncloud() {
 	fi
 }
 
-owncloud_ver=12.0.3
+owncloud_ver=12.0.5
-owncloud_hash=beab41f6a748a43f0accfa6a9808387aef718c61
+owncloud_hash=d25afbac977a4e331f5e38df50aed0844498ca86
 
 # Check if Nextcloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
 if [ ! -d /usr/local/lib/owncloud/ ] \
@@ -183,13 +183,13 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 	# We only need to check if we do upgrades when owncloud/Nextcloud was previously installed
 	if [ -e /usr/local/lib/owncloud/version.php ]; then
 		if grep -q "OC_VersionString = '8\.1\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running 8.1.x, upgrading to 8.2.3 first"
+			echo "We are running 8.1.x, upgrading to 8.2.11 first"
-			InstallOwncloud 8.2.3 bfdf6166fbf6fc5438dc358600e7239d1c970613
+			InstallOwncloud 8.2.11 e4794938fc2f15a095018ba9d6ee18b53f6f299c
 		fi
 
 		# If we are upgrading from 8.2.x we should go to 9.0 first. Owncloud doesn't support skipping minor versions
 		if grep -q "OC_VersionString = '8\.2\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running version 8.2.x, upgrading to 9.0.2 first"
+			echo "We are running version 8.2.x, upgrading to 9.0.11 first"
 
 			# We need to disable memcached. The upgrade and install fails
 			# with memcached
@@ -207,8 +207,8 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 EOF
 			chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
 
-			# We can now install owncloud 9.0.2
+			# We can now install owncloud 9.0.11
-			InstallOwncloud 9.0.2 72a3d15d09f58c06fa8bee48b9e60c9cd356f9c5
+			InstallOwncloud 9.0.11 fc8bad8a62179089bc58c406b28997fb0329337b
 
 			# The owncloud 9 migration doesn't migrate calendars and contacts
 			# The option to migrate these are removed in 9.1
@@ -224,20 +224,26 @@ EOF
 
 		# If we are upgrading from 9.0.x we should go to 9.1 first.
 		if grep -q "OC_VersionString = '9\.0\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running ownCloud 9.0.x, upgrading to ownCloud 9.1.4 first"
+			echo "We are running ownCloud 9.0.x, upgrading to ownCloud 9.1.7 first"
-			InstallOwncloud 9.1.4 e637cab7b2ca3346164f3506b1a0eb812b4e841a
+			InstallOwncloud 9.1.7 1307d997d0b23dc42742d315b3e2f11423a9c808
 		fi
 
-		# If we are upgrading from 9.1.x we should go to Nextcloud 10.0 first.
+		# Newer ownCloud 9.1.x versions cannot be upgraded to Nextcloud 10 and have to be
+		# upgraded to Nextcloud 11 straight away, see:
+		# https://github.com/nextcloud/server/issues/2203
+		# However, for some reason, upgrading to the latest Nextcloud 11.0.7 doesn't
+		# work either. Therefore, we're upgrading to Nextcloud 11.0.0 in the interim.
+		# This should not be a problem since we're upgrading to the latest Nextcloud 12
+		# in the next step.
 		if grep -q "OC_VersionString = '9\.1\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running ownCloud 9.1.x, upgrading to Nextcloud 10.0.5 first"
+			echo "We are running ownCloud 9.1.x, upgrading to Nextcloud 11.0.0 first"
-			InstallNextcloud 10.0.5 686f6a8e9d7867c32e3bf3ca63b3cc2020564bf6
+			InstallNextcloud 11.0.0 e8c9ebe72a4a76c047080de94743c5c11735e72e
 		fi
 
 		# If we are upgrading from 10.0.x we should go to Nextcloud 11.0 first.
 		if grep -q "OC_VersionString = '10\.0\.[0-9]" /usr/local/lib/owncloud/version.php; then
-			echo "We are running Nextcloud 10.0.x, upgrading to Nextcloud 11.0.3 first"
+			echo "We are running Nextcloud 10.0.x, upgrading to Nextcloud 11.0.7 first"
-			InstallNextcloud 11.0.3 a396aaa1c9f920099a90a86b4a9cd0ec13083c99
+			InstallNextcloud 11.0.7 f936ddcb2ae3dbb66ee4926eb8b2ebbddc3facbe
 		fi
 	fi
 

setup/questions.sh

@@ -12,7 +12,9 @@ if [ -z "$NONINTERACTIVE" ]; then
 		apt_get_quiet install dialog python3 python3-pip  || exit 1
 	fi
 
-	# email_validator is repeated in setup/management.sh
+	# Installing email_validator is repeated in setup/management.sh, but in setup/management.sh
+	# we install it inside a virtualenv. In this script, we don't have the virtualenv yet
+	# so we install the python package globally.
 	hide_output pip3 install "email_validator>=1.0.0" || exit 1
 
 	message_box "Mail-in-a-Box Installation" \
@@ -49,7 +51,7 @@ you really want.
 			# user hit ESC/cancel
 			exit
 		fi
-		while ! management/mailconfig.py validate-email "$EMAIL_ADDR"
+		while ! python3 management/mailconfig.py validate-email "$EMAIL_ADDR"
 		do
 			input_box "Your Email Address" \
 				"That's not a valid email address.\n\nWhat email address are you setting this box up to manage?" \

setup/webmail.sh

@@ -35,8 +35,8 @@ apt-get purge -qq -y roundcube* #NODOC
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of plugins to track
 # whether we have the latest version of everything.
-VERSION=1.3.3
+VERSION=1.3.6
-HASH=903a4eb1bfc25e9a08d782a7f98502cddfa579de
+HASH=ece5cfc9c7af0cbe90c0065ef33e85ed42991830
 PERSISTENT_LOGIN_VERSION=dc5ca3d3f4415cc41edb2fde533c8a8628a94c76
 HTML5_NOTIFIER_VERSION=4b370e3cd60dabd2f428a26f45b677ad1b7118d5
 CARDDAV_VERSION=2.0.4

setup/zpush.sh

@@ -22,7 +22,7 @@ apt_install \
 phpenmod -v php7.0 imap
 
 # Copy Z-Push into place.
-VERSION=2.3.8
+VERSION=2.3.9
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/z-push/version ]; then
 	needs_update=1 #NODOC

tools/update-subresource-integrity.py

@@ -1,24 +0,0 @@
-#!/usr/bin/python3
-# Updates subresource integrity attributes in management/templates/index.html
-# to prevent CDN-hosted resources from being used as an attack vector. Run this
-# after updating the Bootstrap and jQuery <link> and <script> to compute the
-# appropriate hash and insert it into the template.
-
-import re, urllib.request, hashlib, base64
-
-fn = "management/templates/index.html"
-
-with open(fn, 'r') as f:
-	content = f.read()
-
-def make_integrity(url):
-	resource = urllib.request.urlopen(url).read()
-	return "sha256-" + base64.b64encode(hashlib.sha256(resource).digest()).decode('ascii')
-
-content = re.sub(
-	r'<(link rel="stylesheet" href|script src)="(.*?)" integrity="(.*?)"',
-	lambda m : '<' + m.group(1) + '="' + m.group(2) + '" integrity="' + make_integrity(m.group(2)) + '"',
-	content)
-
-with open(fn, 'w') as f:
-	f.write(content)