more dockerization work

[From @joshdata: This is part of @toilal's work in #377 and 1eb77c332b. The changes are:

* Separates out the runit configuration from starting Mail-in-a-Box setup so that Mail-in-a-Box setup does not block the starting of runit services and we can assume that runit is running during setup (i.e. we can restart services).
* Adds a SKIP_INSTALL flag so that the container can be restarted without re-running the whole Mail-in-a-Box setup.
* Made containers/docker/run more flexible.
* I'm also adding some "|| exit 0"s to the run script to stop if there are any docker errors.
* I'm also adding the prereqs installs from questions.sh into Dockerfile so we don't have to reinstall each time.

]

CHANGELOG.md

@@ -1,42 +1,18 @@
 CHANGELOG
 =========
 
-v0.12 (July 3, 2015)
---------------------
-
-This is a minor update to v0.11, which was a major update. Please read v0.11's advisories.
-
-* The administrator@ alias was incorrectly created starting with v0.11. If your first install was v0.11, check that the administrator@ alias forwards mail to you.
-* Intrusion detection rules (fail2ban) are relaxed (i.e. less is blocked).
-* SSL certificates could not be installed for the new automatic 'www.' redirect domains.
-* PHP's default character encoding is changed from no default to UTF8. The effect of this change is unclear but should prevent possible future text conversion issues.
-* User-installed SSL private keys in the BEGIN PRIVATE KEY format were not accepted.
-* SSL certificates with SAN domains with IDNA encoding were broken in v0.11.
-* Some IDNA functionality was using IDNA 2003 rather than IDNA 2008.
-
-
-v0.11 (June 29, 2015)
----------------------
-
-Advisories:
-* Users can no longer spoof arbitrary email addresses in outbound mail. When sending mail, the email address configured in your mail client must match the SMTP login username being used, or the email address must be an alias with the SMTP login username listed as one of the alias's targets.
-* This update replaces your DKIM signing key with a stronger key. Because of DNS caching/propagation, mail sent within a few hours after this update could be marked as spam by recipients. If you use External DNS, you will need to update your DNS records.
-* The box will now install software from a new Mail-in-a-Box PPA on Launchpad.net, where we are distributing two of our own packages: a patched postgrey and dovecot-lucene.
-* v0.11b was posted shortly after the initial posting of v0.11 correcting a missing dependency for the new PPA.
+In Development
+--------------
 
 Mail:
 * Greylisting will now let some reputable senders pass through immediately.
 * Searching mail (via IMAP) will now be much faster using the dovecot lucene full text search plugin.
-* Users can no longer spoof arbitrary email addresses in outbound mail (see above).
 * Fix for deleting admin@ and postmaster@ addresses.
 * Roundcube is updated to version 1.1.2, plugins updated.
-* Exchange/ActiveSync autoconfiguration was not working on all devices (e.g. iPhone) because of a case-sensitive URL.
-* The DKIM signing key has been increased to 2048 bits, from 1024, replacing the existing key.
 
 Web:
 * 'www' subdomains now automatically redirect to their parent domain (but you'll need to install an SSL certificate).
 * OCSP no longer uses Google Public DNS.
-* The installed PHP version is no longer exposed through HTTP response headers, for better security.
 
 DNS:
 * Default IPv6 AAAA records were missing since version 0.09.
@@ -44,13 +20,10 @@ DNS:
 Control panel:
 * Resetting a user's password now forces them to log in again everywhere.
 * Status checks were not working if an ssh server was not installed.
-* SSL certificate validation now uses the Python cryptography module in some places where openssl was used.
-* There is a new tab to show the installed version of Mail-in-a-Box and to fetch the latest released version.
 
 System:
 * The munin system monitoring tool is now installed and accessible at /admin/munin.
-* ownCloud updated to version 8.0.4. The ownCloud installation step now is reslient to download problems. The ownCloud configuration file is now stored in STORAGE_ROOT to fix loss of data when moving STORAGE_ROOT to a new machine.
-* The setup scripts now run `apt-get update` prior to installing anything to ensure the apt database is in sync with the packages actually available.
+* ownCloud updated to version 8.0.4.
 
 
 v0.10 (June 1, 2015)

Dockerfile

@@ -0,0 +1,57 @@
+# Mail-in-a-Box Dockerfile
+###########################
+#
+# This file lets Mail-in-a-Box run inside of Docker (https://docker.io),
+# a virtualization/containerization manager.
+#
+# Run:
+#   $ containers/docker/run.sh
+# to build the image, launch a storage container, and launch a Mail-in-a-Box
+# container.
+#
+###########################################
+
+# We need a better starting image than docker's ubuntu image because that
+# base image doesn't provide enough to run most Ubuntu services. See
+# http://phusion.github.io/baseimage-docker/ for an explanation.
+
+FROM phusion/baseimage:0.9.16
+
+# Dockerfile metadata.
+MAINTAINER Joshua Tauberer (http://razor.occams.info)
+EXPOSE 25 53/udp 53/tcp 80 443 587 993 4190
+VOLUME /home/user-data
+
+# Use baseimage's init system. A correct init process is required for
+# process #1 in order to have a functioning Linux system.
+CMD ["/sbin/my_init"]
+
+# Create the user-data user, so the start script doesn't have to.
+RUN useradd -m user-data
+
+# Docker has a beautiful way to cache images after each step. The next few
+# steps of installing system packages are very intensive, so we take care
+# of them early and let docker cache the image after that, before doing
+# any Mail-in-a-Box specific system configuration. That makes rebuilds
+# of the image extremely fast.
+
+# Update system packages.
+RUN apt-get update
+RUN DEBIAN_FRONTEND=noninteractive apt-get upgrade -y
+
+# Install packages needed by Mail-in-a-Box.
+ADD containers/docker/apt_package_list.txt /tmp/mailinabox_apt_package_list.txt
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y $(cat /tmp/mailinabox_apt_package_list.txt)
+
+# from questions.sh -- needs merging into the above line
+RUN DEBIAN_FRONTEND=noninteractive apt-get install -y dialog python3 python3-pip
+RUN pip3 install "email_validator==0.1.0-rc4"
+
+# Now add Mail-in-a-Box to the system.
+ADD . /usr/local/mailinabox
+
+# Configure runit services.
+RUN /usr/local/mailinabox/containers/docker/tools/configure_services.sh
+
+# Add my_init scripts
+ADD containers/docker/my_init.d/* /etc/my_init.d/

README.md

@@ -57,16 +57,13 @@ I sign the release tags on git. To verify that a tag is signed by me, you can pe
 	$ cd mailinabox
 
 	# Verify the tag.
-	$ git verify-tag v0.12
+	$ git verify-tag v0.10
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
 	gpg:          There is no indication that the signature belongs to the owner.
 	Primary key fingerprint: 5F4C 0E73 13CC D744 693B  2AEA B920 41F4 C10B DD81
 
-	# Check out the tag.
-	$ git checkout v0.12
-
 The key ID and fingerprint above should match my [Keybase.io key](https://keybase.io/joshdata) and the fingerprint I publish on [my homepage](https://razor.occams.info/).
 
 The Acknowledgements
@@ -83,4 +80,3 @@ The History
 * In August 2013 I began Mail-in-a-Box by combining my own mail server configuration with the setup in ["NSA-proof your email in 2 hours"](http://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/) and making the setup steps reproducible with bash scripts.
 * Mail-in-a-Box was a semifinalist in the 2014 [Knight News Challenge](https://www.newschallenge.org/challenge/2014/submissions/mail-in-a-box), but it was not selected as a winner.
 * Mail-in-a-Box hit the front page of Hacker News in [April](https://news.ycombinator.com/item?id=7634514) 2014, [September](https://news.ycombinator.com/item?id=8276171) 2014, and [May](https://news.ycombinator.com/item?id=9624267) 2015.
-* FastCompany mentioned Mail-in-a-Box a [roundup of privacy projects](http://www.fastcompany.com/3047645/your-own-private-cloud) on June 26, 2015.

conf/fail2ban/jail.local

@@ -1,19 +1,34 @@
-# Fail2Ban configuration file for Mail-in-a-Box
-
+# Fail2Ban configuration file.
+# For Mail-in-a-Box
 [DEFAULT]
 
+# bantime in seconds
+bantime  = 60
+
 # This should ban dumb brute-force attacks, not oblivious users.
 findtime = 30
 maxretry = 20
 
+#
 # JAILS
+#
+
+[ssh]
+
+enabled  = true
+logpath  = /var/log/auth.log
+maxretry = 20
 
 [ssh-ddos]
+
 enabled  = true
+maxretry = 20
 
 [sasl]
+
 enabled  = true
 
 [dovecot]
+
 enabled = true
 filter  = dovecotimap

conf/nginx-alldomains.conf

@@ -55,7 +55,7 @@
 		# file upload limit to match the corresponding Postfix limit.
 		client_max_body_size 128M;
 	}
-	location ~* ^/autodiscover/autodiscover.xml$ {
+	location /autodiscover/autodiscover.xml {
 		include fastcgi_params;
 		fastcgi_param SCRIPT_FILENAME /usr/local/lib/z-push/autodiscover/autodiscover.php;
 		fastcgi_param PHP_VALUE "include_path=.:/usr/share/php:/usr/share/pear:/usr/share/awl/inc";

conf/nginx-ssl.conf

@@ -69,6 +69,6 @@ ssl_dhparam STORAGE_ROOT/ssl/dh2048.pem;
 # 8.8.8.8 and 8.8.4.4 below are Google's public IPv4 DNS servers. 
 # nginx will use them to talk to the CA.
 ssl_stapling on;
-ssl_stapling_verify on;
+ssl_stapling_verify off;
 resolver 127.0.0.1 valid=86400;
 resolver_timeout 10;

containers/docker/apt_package_list.txt

@@ -0,0 +1,86 @@
+bc
+bind9
+ca-certificates
+coreutils
+cron
+curl
+dbconfig-common
+dovecot-antispam
+dovecot-core
+dovecot-imapd
+dovecot-lmtpd
+dovecot-lucene
+dovecot-managesieved
+dovecot-pop3d
+dovecot-sieve
+dovecot-sqlite
+duplicity
+fail2ban
+git
+haveged
+ldnsutils
+libapr1
+libawl-php
+libcurl4-openssl-dev
+libjs-jquery
+libjs-jquery-mousewheel
+libmagic1
+libtool
+libyaml-dev
+links
+memcached
+munin
+munin-node
+nginx
+nsd
+ntp
+opendkim
+opendkim-tools
+opendmarc
+openssh-client
+openssl
+php-apc
+php-auth
+php-crypt-gpg
+php-mail-mime
+php-net-sieve
+php-net-smtp
+php-net-socket
+php-pear
+php-soap
+php-xml-parser
+php5
+php5-cli
+php5-common
+php5-curl
+php5-dev
+php5-fpm
+php5-gd
+php5-imap
+php5-intl
+php5-json
+php5-mcrypt
+php5-memcache
+php5-pspell
+php5-sqlite
+php5-xsl
+postfix
+postfix-pcre
+postgrey
+python3
+python3-dateutil
+python3-dev
+python3-dnspython
+python3-flask
+python3-pip
+pyzor
+razor
+resolvconf
+spampd
+sqlite3
+sudo
+tinymce
+ufw
+unattended-upgrades
+unzip
+wget

containers/docker/my_init.d/10-mailinabox.sh

@@ -0,0 +1,58 @@
+#!/bin/bash
+
+# This script is used within containers to turn it into a Mail-in-a-Box.
+# It is referenced by the Dockerfile. You should not run it directly.
+########################################################################
+
+# Local configuration details were not known at the time the Docker
+# image was created, so all setup is defered until the container
+# is started. That's when this script runs.
+
+# If we're not in an interactive shell, set defaults.
+if [ ! -t 0 ]; then
+	echo '*** Non interactive shell detected...'
+	export PUBLIC_IP=auto
+	export PUBLIC_IPV6=auto
+	export PRIMARY_HOSTNAME=auto
+	export CSR_COUNTRY=US
+	export NONINTERACTIVE=1
+fi
+
+if ([ -z "$FORCE_INSTALL" ] && [ -f /var/lib/mailinabox/api.key ]); then
+	# Mailinabox is already installed and we don't want to reinstall
+	export SKIP_INSTALL=1
+fi
+
+# If we are skipping install, reload from /etc/mailinabox.conf if exists
+if ([ -f /var/lib/mailinabox/api.key ] && [ ! -z "$SKIP_INSTALL" ]); then
+	echo '*** Loading variables from "/etc/mailinabox.conf"...'
+
+	source /etc/mailinabox.conf
+	unset PRIVATE_IP
+	unset PRIVATE_IPV6
+	export SKIP_NETWORK_CHECKS=1
+	export NONINTERACTIVE=1
+fi
+
+export DISABLE_FIREWALL=1
+cd /usr/local/mailinabox
+
+if [ -z "$SKIP_INSTALL" ]; then
+	echo "*** Starting mailinabox installation..."
+	# Run in background to avoid blocking runit initialization while installing.
+	source setup/start.sh &
+else
+	echo "*** Configuring mailinabox..."
+	# Run in foreground for services to be started after configuration is re-written.
+	source setup/questions.sh
+	cat > /etc/mailinabox.conf << EOF;
+STORAGE_USER=$STORAGE_USER
+STORAGE_ROOT=$STORAGE_ROOT
+PRIMARY_HOSTNAME=$PRIMARY_HOSTNAME
+PUBLIC_IP=$PUBLIC_IP
+PUBLIC_IPV6=$PUBLIC_IPV6
+PRIVATE_IP=$PRIVATE_IP
+PRIVATE_IPV6=$PRIVATE_IPV6
+CSR_COUNTRY=$CSR_COUNTRY
+EOF
+fi

containers/docker/run

@@ -0,0 +1,114 @@
+#!/bin/bash
+# Use this script to launch Mail-in-a-Box within a docker container.
+# ==================================================================
+#
+# Run this script from the base directory of the Mail-in-a-Box
+# repository (i.e. run as './containers/docker/run').
+#
+# Set these optional environment variables as needed:
+# * HOST_HTTP_PORT: Host http: port to bind (default: 80).
+# * HOST_HTTPS_PORT: Host https: port to bind (default: 443).
+# * SKIP_BUILD: Skip the build of docker image (default: unset).
+# * NODNS: Skip mapping of DNS ports (53 tcp/upd). They are not always available on host, as another DNS server can be running (default: unset).
+# * CONTAINER_NAME: Name of the main container (default: mailinabox).
+# * CONTAINER_DATA_NAME: Name of the data container (default: mailinabox-data).
+# * NONINTERACTIVE: Use this when mailinabox is already installed on the volume container. Else, it's not recommanded (default: unset).
+#
+# A base image is created first. The base image installs Ubuntu
+# packages and pulls in the Mail-in-a-Box source code. This is
+# defined in Dockerfile at the root of this repository.
+#
+# A mailinabox-data container is created next. This container
+# contains nothing but a shared volume for storing user data.
+# It is segregated from the rest of the live system to make backups
+# easier.
+#
+# The mailinabox container is started last. It is the
+# real thing: it runs the mailinabox image. This container will
+# initialize itself and will initialize the mailinabox-data
+# volume if the volume is new.
+
+# Build or rebuild the image.
+# Rebuilds are very fast.
+
+HOST_HTTP_PORT=${HOST_HTTP_PORT:-80}
+HOST_HTTPS_PORT=${HOST_HTTPS_PORT:-443}
+CONTAINER_NAME=${CONTAINER_NAME:-mailinabox}
+CONTAINER_DATA_NAME=${CONTAINER_DATA_NAME:-${CONTAINER_NAME}-data}
+
+if [ -z "$SKIP_BUILD" ]; then
+	tput setaf 2
+	echo "Building/updating base image (mailinabox)..."
+	tput setaf 7
+
+	docker build -q -t mailinabox . || exit 1
+fi;
+
+if ! docker inspect ${CONTAINER_DATA_NAME} > /dev/null; then
+	tput setaf 2
+	echo
+	echo "Creating a new container for your data (${CONTAINER_DATA_NAME})..."
+	tput setaf 7
+
+	docker create \
+		--name ${CONTAINER_DATA_NAME} \
+		-v /home/user-data \
+		phusion/baseimage:0.9.16 || exit 1
+else
+	tput setaf 2
+	echo
+	echo "Using existing container ${CONTAINER_DATA_NAME} for your data."
+	tput setaf 7
+fi
+
+# End a running container.
+if docker inspect ${CONTAINER_NAME} > /dev/null; then
+	tput setaf 2
+	echo
+	echo "Destroying ${CONTAINER_NAME} container..."
+	tput setaf 7
+
+	docker rm -f ${CONTAINER_NAME}
+fi
+
+# Start container.
+tput setaf 2
+echo
+echo "Starting new container (${CONTAINER_NAME})..."
+tput setaf 7
+
+# Run the services container
+# detached if NONINTERACTIVE is set,
+# interactively if NONINTERACTIVE is not set,
+# Notes:
+# * Passing through SKIP_NETWORK_CHECKS makes it easier to do testing
+#   on a residential network.
+# * --privileged flag cause an issue with bind9/named failing to start in this case
+#   see docker/docker#7318
+docker run \
+	-v /dev/urandom:/dev/random \
+	-p 25:25 \
+	$([ -z "$NODNS" ] && echo "-p 53:53/udp -p 53:53/tcp") \
+	-p $HOST_HTTP_PORT:80 \
+	-p $HOST_HTTPS_PORT:443 \
+	-p 587:587 \
+	-p 993:993 \
+	-p 4190:4190 \
+	--name ${CONTAINER_NAME} \
+	--volumes-from ${CONTAINER_DATA_NAME} \
+	--restart always \
+	$([ ! -z "$NONINTERACTIVE" ] && echo "-d") \
+	-it \
+	-e "IS_DOCKER=1" \
+	-e "SKIP_NETWORK_CHECKS=$SKIP_NETWORK_CHECKS" \
+	mailinabox \
+	|| exit 1
+
+if [ -z "$NONINTERACTIVE" ]; then
+	tput setaf 2
+	echo
+	echo "Restarting container ${CONTAINER_NAME}..."
+	tput setaf 7
+
+	docker restart ${CONTAINER_NAME} || exit 1
+fi

containers/docker/runit/dovecot.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+/usr/sbin/dovecot -F -c /etc/dovecot/dovecot.conf &> /var/log/dovecot.log

containers/docker/tools/configure_services.sh

@@ -0,0 +1,103 @@
+#!/bin/bash
+
+# The phusion/baseimage base image we use for a working Ubuntu
+# replaces the normal Upstart system service management with
+# a ligher-weight service management system called runit that
+# requires a different configuration. We need to create service
+# run files that do not daemonize.
+
+# This removes /etc/init.d service if service exists in runit.
+# It also creates a symlink from /usr/bin/sv to /etc/init.d/$service
+# to support SysV syntax: service $service <command> or /etc/init.d/$service <command>
+SERVICES=/etc/service/*
+for f in $SERVICES
+do
+	service=$(basename "$f")
+	if [ -d /etc/service/$service ]; then
+		if [ -f /etc/init.d/$service ]; then
+			mv /etc/init.d/$service /etc/init.d/$service.lsb
+			chmod -x /etc/init.d/$service.lsb
+		fi
+		ln -s /usr/bin/sv /etc/init.d/$service
+	fi
+done
+
+# Create runit services from sysv services. For most of the services,
+# there is a common pattern we can use: execute the init.d script that
+# the Ubuntu package installs, and then poll for the termination of
+# the daemon.
+function make_runit_service {
+	INITD_NAME=$1
+	WAIT_ON_PROCESS_NAME=$2
+	mkdir -p /etc/service/$INITD_NAME
+	cat > /etc/service/$INITD_NAME/run <<EOF;
+#!/bin/bash
+source /usr/local/mailinabox/setup/functions.sh
+hide_output /etc/init.d/$INITD_NAME restart
+while [ \`ps a -C $WAIT_ON_PROCESS_NAME -o pid= | wc -l\` -gt 0 ]; do
+       sleep 30
+done
+echo $WAIT_ON_PROCESS_NAME died.
+sleep 20
+EOF
+	chmod +x /etc/service/$INITD_NAME/run
+}
+make_runit_service bind9 named
+make_runit_service resolvconf resolvconf
+make_runit_service fail2ban fail2ban
+make_runit_service mailinabox mailinabox-daemon
+make_runit_service memcached memcached
+make_runit_service nginx nginx
+make_runit_service nsd nsd
+make_runit_service opendkim opendkim
+make_runit_service opendmarc opendmarc
+make_runit_service php5-fpm php5-fpm
+make_runit_service postfix postfix
+make_runit_service postgrey postgrey
+make_runit_service spampd spampd
+
+# Dovecot doesn't provide an init.d script, but it does provide
+# a way to launch without daemonization. We wrote a script for
+# that specifically.
+for service in dovecot; do
+	mkdir -p /etc/service/$service
+	cp /usr/local/mailinabox/containers/docker/runit/$service.sh /etc/service/$service/run
+	chmod +x /etc/service/$service/run
+done
+
+# This adds a log/run file on each runit service directory.
+# This file make services stdout/stderr output to svlogd log
+# directory located in /var/log/runit/$service.
+SERVICES=/etc/service/*
+for f in $SERVICES
+do
+	service=$(basename "$f")
+	if [ -d /etc/service/$service ]; then
+		mkdir -p /etc/service/$service/log
+		cat > /etc/service/$service/log/run <<EOF;
+#!/bin/bash
+mkdir -p /var/log/runit
+chmod o-wrx /var/log/runit
+mkdir -p /var/log/runit/$service
+chmod o-wrx /var/log/runit/$service
+exec svlogd -tt /var/log/runit/$service/
+EOF
+		chmod +x /etc/service/$service/log/run
+	fi
+done
+
+# Disable services for now. Until Mail-in-a-Box is installed the
+# services won't be configured right and there would be errors if
+# they got run prematurely.
+SERVICES=/etc/service/*
+for f in $SERVICES
+do
+	service=$(basename "$f")
+	if [ "$service" = "syslog-ng" ]; then continue; fi;
+	if [ "$service" = "syslog-forwarder" ]; then continue; fi;
+	if [ "$service" = "ssh" ]; then continue; fi;
+	if [ "$service" = "cron" ]; then continue; fi;
+	if ([ -d /etc/service/$service ] && [ ! -f /etc/service/$service/down ]); then
+		touch /etc/service/$service/down
+	fi
+done

management/daemon.py

@@ -340,24 +340,6 @@ def web_update():
 
 # System
 
-@app.route('/system/version', methods=["GET"])
-@authorized_personnel_only
-def system_version():
-	from status_checks import what_version_is_this
-	try:
-		return what_version_is_this(env)
-	except Exception as e:
-		return (str(e), 500)
-
-@app.route('/system/latest-upstream-version', methods=["POST"])
-@authorized_personnel_only
-def system_latest_upstream_version():
-	from status_checks import get_latest_miab_version
-	try:
-		return get_latest_miab_version()
-	except Exception as e:
-		return (str(e), 500)
-
 @app.route('/system/status', methods=["POST"])
 @authorized_personnel_only
 def system_status():

management/dns_update.py

@@ -250,8 +250,8 @@ def build_zone(domain, all_domains, additional_records, www_redirect_domains, en
 	# Skip if the user has set a DKIM record already.
 	opendkim_record_file = os.path.join(env['STORAGE_ROOT'], 'mail/dkim/mail.txt')
 	with open(opendkim_record_file) as orf:
-		m = re.match(r'(\S+)\s+IN\s+TXT\s+\( ((?:"[^"]+"\s+)+)\)', orf.read(), re.S)
-		val = "".join(re.findall(r'"([^"]+)"', m.group(2)))
+		m = re.match(r'(\S+)\s+IN\s+TXT\s+\( "([^"]+)"\s+"([^"]+)"\s*\)', orf.read(), re.S)
+		val = m.group(2) + m.group(3)
 		if not has_rec(m.group(1), "TXT", prefix="v=DKIM1; "):
 			records.append((m.group(1), "TXT", val, "Recommended. Provides a way for recipients to verify that this machine sent @%s mail." % domain))
 
@@ -373,16 +373,9 @@ $TTL 1800           ; default time to live
 			zone += subdomain
 		zone += "\tIN\t" + querytype + "\t"
 		if querytype == "TXT":
-			# Divide into 255-byte max substrings.
-			v2 = ""
-			while len(value) > 0:
-				s = value[0:255]
-				value = value[255:]
-				s = s.replace('\\', '\\\\') # escape backslashes
-				s = s.replace('"', '\\"') # escape quotes
-				s = '"' + s + '"' # wrap in quotes
-				v2 += s + " "
-			value = v2
+			value = value.replace('\\', '\\\\') # escape backslashes
+			value = value.replace('"', '\\"') # escape quotes
+			value = '"' + value + '"' # wrap in quotes
 		zone += value + "\n"
 
 	# DNSSEC requires re-signing a zone periodically. That requires

management/mailconfig.py

@@ -3,7 +3,6 @@
 import subprocess, shutil, os, sqlite3, re
 import utils
 from email_validator import validate_email as validate_email_, EmailNotValidError
-import idna
 
 def validate_email(email, mode=None):
 	# Checks that an email address is syntactically valid. Returns True/False.
@@ -53,9 +52,9 @@ def sanitize_idn_email_address(email):
 	# to the underlying protocols.
 	try:
 		localpart, domainpart = email.split("@")
-		domainpart = idna.encode(domainpart).decode('ascii')
+		domainpart = domainpart.encode("idna").decode('ascii')
 		return localpart + "@" + domainpart
-	except idna.IDNAError:
+	except:
 		# Domain part is not IDNA-valid, so leave unchanged. If there
 		# are non-ASCII characters it will be filtered out by
 		# validate_email.
@@ -66,9 +65,9 @@ def prettify_idn_email_address(email):
 	# names in IDNA in the database, but we want to show Unicode to the user.
 	try:
 		localpart, domainpart = email.split("@")
-		domainpart = idna.decode(domainpart.encode("ascii"))
+		domainpart = domainpart.encode("ascii").decode('idna')
 		return localpart + "@" + domainpart
-	except (UnicodeError, idna.IDNAError):
+	except:
 		# Failed to decode IDNA. Should never happen.
 		return email
 
@@ -239,7 +238,7 @@ def get_domain(emailaddr, as_unicode=True):
 	# Gets the domain part of an email address. Turns IDNA
 	# back to Unicode for display.
 	ret = emailaddr.split('@', 1)[1]
-	if as_unicode: ret = idna.decode(ret.encode('ascii'))
+	if as_unicode: ret = ret.encode('ascii').decode('idna')
 	return ret
 
 def get_mail_domains(env, filter_aliases=lambda alias : True):
@@ -544,7 +543,6 @@ def kick(env, mail_result=None):
 
 		# Doesn't exist.
 		administrator = get_system_administrator(env)
-		if source == administrator: return # don't make an alias from the administrator to itself --- this alias must be created manually
 		add_mail_alias(source, administrator, env, do_kick=False)
 		results.append("added alias %s (=> %s)\n" % (source, administrator))
 

management/status_checks.py

@@ -10,7 +10,6 @@ import sys, os, os.path, re, subprocess, datetime, multiprocessing.pool
 
 import dns.reversename, dns.resolver
 import dateutil.parser, dateutil.tz
-import idna
 
 from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config, get_secondary_dns
 from web_update import get_web_domains, get_default_www_redirects, get_domain_ssl_files
@@ -260,7 +259,7 @@ def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zone
 	output = BufferedOutput()
 
 	# The domain is IDNA-encoded, but for display use Unicode.
-	output.add_heading(idna.decode(domain.encode('ascii')))
+	output.add_heading(domain.encode('ascii').decode('idna'))
 
 	if domain == env["PRIMARY_HOSTNAME"]:
 		check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles)
@@ -606,115 +605,103 @@ def check_ssl_cert(domain, rounded_time, env, output):
 			output.print_line(cert_status_details)
 			output.print_line("")
 
-def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=True, rounded_time=False, just_check_domain=False):
-	# Check that the ssl_certificate & ssl_private_key files are good
-	# for the provided domain.
+def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=True, rounded_time=False):
+	# Use openssl verify to check the status of a certificate.
 
-	from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
-	from cryptography.x509 import Certificate, DNSName, ExtensionNotFound, OID_COMMON_NAME, OID_SUBJECT_ALTERNATIVE_NAME
-	import idna
+	# First check that the certificate is for the right domain. The domain
+	# must be found in the Subject Common Name (CN) or be one of the
+	# Subject Alternative Names. A wildcard might also appear as the CN
+	# or in the SAN list, so check for that tool.
+	retcode, cert_dump = shell('check_output', [
+		"openssl", "x509",
+		"-in", ssl_certificate,
+		"-noout", "-text", "-nameopt", "rfc2253",
+		], trap=True)
 
-	# The ssl_certificate file may contain a chain of certificates. We'll
-	# need to split that up before we can pass anything to openssl or
-	# parse them in Python. Parse it with the cryptography library.
-	try:
-		ssl_cert_chain = load_cert_chain(ssl_certificate)
-		cert = load_pem(ssl_cert_chain[0])
-		if not isinstance(cert, Certificate): raise ValueError("This is not a certificate file.")
-	except ValueError as e:
-		return ("There is a problem with the certificate file: %s" % str(e), None)
+	# If the certificate is catastrophically bad, catch that now and report it.
+	# More information was probably written to stderr (which we aren't capturing),
+	# but it is probably not helpful to the user anyway.
+	if retcode != 0:
+		return ("The SSL certificate appears to be corrupted or not a PEM-formatted SSL certificate file. (%s)" % ssl_certificate, None)
 
-	# First check that the domain name is one of the names allowed by
-	# the certificate.
-	if domain is not None:
-		# The domain may be found in the Subject Common Name (CN). This comes back as an IDNA (ASCII)
-		# string, which is the format we store domains in - so good.
-		certificate_names = set()
-		try:
-			certificate_names.add(
-				cert.subject.get_attributes_for_oid(OID_COMMON_NAME)[0].value
-				)
-		except IndexError:
-			# No common name? Certificate is probably generated incorrectly.
-			# But we'll let it error-out when it doesn't find the domain.
-			pass
+	cert_dump = cert_dump.split("\n")
+	certificate_names = set()
+	cert_expiration_date = None
+	while len(cert_dump) > 0:
+		line = cert_dump.pop(0)
 
-		# ... or be one of the Subject Alternative Names. The cryptography library handily IDNA-decodes
-		# the names for us. We must encode back to ASCII, but wildcard certificates can't pass through
-		# IDNA encoding/decoding so we must special-case. See https://github.com/pyca/cryptography/pull/2071.
-		def idna_decode_dns_name(dns_name):
-			if dns_name.startswith("*."):
-				return "*." + idna.encode(dns_name[2:]).decode('ascii')
-			else:
-				return idna.encode(dns_name).decode('ascii')
+		# Grab from the Subject Common Name. We include the indentation
+		# at the start of the line in case maybe the cert includes the
+		# common name of some other referenced entity (which would be
+		# indented, I hope).
+		m = re.match("        Subject: CN=([^,]+)", line)
+		if m:
+			certificate_names.add(m.group(1))
 	
-		try:
-			sans = cert.extensions.get_extension_for_oid(OID_SUBJECT_ALTERNATIVE_NAME).value.get_values_for_type(DNSName)
-			for san in sans:
-				certificate_names.add(idna_decode_dns_name(san))
-		except ExtensionNotFound:
-			pass
+		# Grab from the Subject Alternative Name, which is a comma-delim
+		# list of names, like DNS:mydomain.com, DNS:otherdomain.com.
+		m = re.match("            X509v3 Subject Alternative Name:", line)
+		if m:
+			names = re.split(",\s*", cert_dump.pop(0).strip())
+			for n in names:
+				m = re.match("DNS:(.*)", n)
+				if m:
+					certificate_names.add(m.group(1))
 
-		# Check that the domain appears among the acceptable names, or a wildcard
-		# form of the domain name (which is a stricter check than the specs but
-		# should work in normal cases).
-		wildcard_domain = re.sub("^[^\.]+", "*", domain)
-		if domain not in certificate_names and wildcard_domain not in certificate_names:
-			return ("The certificate is for the wrong domain name. It is for %s."
-				% ", ".join(sorted(certificate_names)), None)
+		# Grab the expiration date for testing later.
+		m = re.match("            Not After : (.*)", line)
+		if m:
+			cert_expiration_date = dateutil.parser.parse(m.group(1))
 
-	# Second, check that the certificate matches the private key.
+	wildcard_domain = re.sub("^[^\.]+", "*", domain)
+	if domain is not None and domain not in certificate_names and wildcard_domain not in certificate_names:
+		return ("The certificate is for the wrong domain name. It is for %s."
+			% ", ".join(sorted(certificate_names)), None)
+
+	# Second, check that the certificate matches the private key. Get the modulus of the
+	# private key and of the public key in the certificate. They should match. The output
+	# of each command looks like "Modulus=XXXXX".
 	if ssl_private_key is not None:
-		try:
-			priv_key = load_pem(open(ssl_private_key, 'rb').read())
-		except ValueError as e:
-			return ("The private key file %s is not a private key file: %s" % (ssl_private_key, str(e)), None)
-
-		if not isinstance(priv_key, RSAPrivateKey):
-			return ("The private key file %s is not a private key file." % ssl_private_key, None)
-
-		if priv_key.public_key().public_numbers() != cert.public_key().public_numbers():
-			return ("The certificate does not correspond to the private key at %s." % ssl_private_key, None)
-
-		# We could also use the openssl command line tool to get the modulus
-		# listed in each file. The output of each command below looks like "Modulus=XXXXX".
-		# $ openssl rsa -inform PEM -noout -modulus -in ssl_private_key
-		# $ openssl x509 -in ssl_certificate -noout -modulus
-
-	# Third, check if the certificate is self-signed. Return a special flag string.
-	if cert.issuer == cert.subject:
-		return ("SELF-SIGNED", None)
-
-	# When selecting which certificate to use for non-primary domains, we check if the primary
-	# certificate or a www-parent-domain certificate is good for the domain. There's no need
-	# to run extra checks beyond this point.
-	if just_check_domain:
-		return ("OK", None)
-
-	# Check that the certificate hasn't expired. The datetimes returned by the
-	# certificate are 'naive' and in UTC. We need to get the current time in UTC.
-	now = datetime.datetime.utcnow()
-	if not(cert.not_valid_before <= now <= cert.not_valid_after):
-		return ("The certificate has expired or is not yet valid. It is valid from %s to %s." % (cert.not_valid_before, cert.not_valid_after), None)
+		private_key_modulus = shell('check_output', [
+			"openssl", "rsa",
+			"-inform", "PEM",
+			"-noout", "-modulus",
+			"-in", ssl_private_key])
+		cert_key_modulus = shell('check_output', [
+			"openssl", "x509",
+			"-in", ssl_certificate,
+			"-noout", "-modulus"])
+		if private_key_modulus != cert_key_modulus:
+			return ("The certificate installed at %s does not correspond to the private key at %s." % (ssl_certificate, ssl_private_key), None)
 
 	# Next validate that the certificate is valid. This checks whether the certificate
 	# is self-signed, that the chain of trust makes sense, that it is signed by a CA
 	# that Ubuntu has installed on this machine's list of CAs, and I think that it hasn't
 	# expired.
 
-	# The certificate chain has to be passed separately and is given via STDIN.
+	# In order to verify with openssl, we need to split out any
+	# intermediary certificates in the chain (if any) from our
+	# certificate (at the top). They need to be passed separately.
+
+	cert = open(ssl_certificate).read()
+	m = re.match(r'(-*BEGIN CERTIFICATE-*.*?-*END CERTIFICATE-*)(.*)', cert, re.S)
+	if m == None:
+		return ("The certificate file is an invalid PEM certificate.", None)
+	mycert, chaincerts = m.groups()
+
 	# This command returns a non-zero exit status in most cases, so trap errors.
+
 	retcode, verifyoutput = shell('check_output', [
 		"openssl",
 		"verify", "-verbose",
 		"-purpose", "sslserver", "-policy_check",]
-		+ ([] if len(ssl_cert_chain) == 1 else ["-untrusted", "/dev/stdin"])
+		+ ([] if chaincerts.strip() == "" else ["-untrusted", "/dev/stdin"])
 		+ [ssl_certificate],
-		input=b"\n\n".join(ssl_cert_chain[1:]),
+		input=chaincerts.encode('ascii'),
 		trap=True)
 
 	if "self signed" in verifyoutput:
-		# Certificate is self-signed. Probably we detected this above.
+		# Certificate is self-signed.
 		return ("SELF-SIGNED", None)
 
 	elif retcode != 0:
@@ -729,7 +716,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 		# good.
 
 		# But is it expiring soon?
-		cert_expiration_date = cert.not_valid_after
+		now = datetime.datetime.now(dateutil.tz.tzlocal())
 		ndays = (cert_expiration_date-now).days
 		if not rounded_time or ndays < 7:
 			expiry_info = "The certificate expires in %d days on %s." % (ndays, cert_expiration_date.strftime("%x"))
@@ -746,33 +733,6 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 		# Return the special OK code.
 		return ("OK", expiry_info)
 
-def load_cert_chain(pemfile):
-	# A certificate .pem file may contain a chain of certificates.
-	# Load the file and split them apart.
-	re_pem = rb"(-+BEGIN (?:.+)-+[\r\n](?:[A-Za-z0-9+/=]{1,64}[\r\n])+-+END (?:.+)-+[\r\n])"
-	with open(pemfile, "rb") as f:
-		pem = f.read() + b"\n" # ensure trailing newline
-		pemblocks = re.findall(re_pem, pem)
-		if len(pemblocks) == 0:
-			raise ValueError("File does not contain valid PEM data.")
-		return pemblocks
-
-def load_pem(pem):
-	# Parse a "---BEGIN .... END---" PEM string and return a Python object for it
-	# using classes from the cryptography package.
-	from cryptography.x509 import load_pem_x509_certificate
-	from cryptography.hazmat.primitives import serialization
-	from cryptography.hazmat.backends import default_backend
-	pem_type = re.match(b"-+BEGIN (.*?)-+\n", pem)
-	if pem_type is None:
-		raise ValueError("File is not a valid PEM-formatted file.")
-	pem_type = pem_type.group(1)
-	if pem_type in (b"RSA PRIVATE KEY", b"PRIVATE KEY"):
-		return serialization.load_pem_private_key(pem, password=None, backend=default_backend())
-	if pem_type == b"CERTIFICATE":
-		return load_pem_x509_certificate(pem, default_backend())
-	raise ValueError("Unsupported PEM object type: " + pem_type.decode("ascii", "replace"))
-
 _apt_updates = None
 def list_apt_updates(apt_update=True):
 	# See if we have this information cached recently.
@@ -807,20 +767,6 @@ def list_apt_updates(apt_update=True):
 
 	return pkgs
 
-def what_version_is_this(env):
-	# This function runs `git describe` on the Mail-in-a-Box installation directory.
-	# Git may not be installed and Mail-in-a-Box may not have been cloned from github,
-	# so this function may raise all sorts of exceptions.
-	miab_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-	tag = shell("check_output", ["/usr/bin/git", "describe"], env={"GIT_DIR": os.path.join(miab_dir, '.git')}).strip()
-	return tag
-
-def get_latest_miab_version():
-	# This pings https://mailinabox.email/bootstrap.sh and extracts the tag named in
-	# the script to determine the current product version.
-	import urllib.request
-	return re.search(b'TAG=(.*)', urllib.request.urlopen("https://mailinabox.email/bootstrap.sh?ping=1").read()).group(1).decode("utf8")
-
 def run_and_output_changes(env, pool, send_via_email):
 	import json
 	from difflib import SequenceMatcher
@@ -1001,6 +947,3 @@ if __name__ == "__main__":
 		if cert_status != "OK":
 			sys.exit(1)
 		sys.exit(0)
-
-	elif sys.argv[1] == "--version":
-		print(what_version_is_this(env))

management/templates/index.html

@@ -114,7 +114,6 @@
             </li>
             <li><a href="#sync_guide" onclick="return show_panel(this);">Contacts/Calendar</a></li>
             <li><a href="#web" onclick="return show_panel(this);">Web</a></li>
-            <li><a href="#version" onclick="return show_panel(this);">Version</a></li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
             <li><a href="#" onclick="do_logout(); return false;" style="color: white">Log out?</a></li>
@@ -168,10 +167,6 @@
       {% include "ssl.html" %}
       </div>
 
-      <div id="panel_version" class="admin_panel">
-      {% include "version.html" %}
-      </div>
-
       <hr>
 
       <footer>

management/templates/version.html

@@ -1,36 +0,0 @@
-<style>
-</style>
-
-<h2>Mail-in-a-Box Version</h2>
-
-<p>You are running Mail-in-a-Box version <span id="miab-version" style="font-weight: bold">...</span>.</p>
-
-<p>The latest version of Mail-in-a-Box is <button id="miab-get-latest-upstream" onclick="check_latest_version()">Check</button>.</p>
-
-<p>To find the latest version and for upgrade instructions, see <a href="https://mailinabox.email/">https://mailinabox.email/</a>, <a href="https://github.com/mail-in-a-box/mailinabox/blob/master/CHANGELOG.md">release notes</a>, and <a href="https://mailinabox.email/maintenance.html#updating-mail-in-a-box">upgrade instructions</a>.</p>
-
-<script>
-function show_version() {
- $('#miab-version').text('loading...');
- api(
-    "/system/version",
-    "GET",
-    {
-    },
-    function(version) {
-      $('#miab-version').text(version);
-    });
-}
-
-function check_latest_version() {
-  $('#miab-get-latest-upstream').text('loading...');
-  api(
-   "/system/latest-upstream-version",
-   "POST",
-   {
-   },
-   function(version) {
-      $('#miab-get-latest-upstream').text(version);
-   });
-}
-</script>

management/web_update.py

@@ -201,14 +201,14 @@ def get_domain_ssl_files(domain, env, allow_shared_cert=True):
 		# the user has uploaded a different private key for this domain.
 		if not ssl_key_is_alt and allow_shared_cert:
 			from status_checks import check_certificate
-			if check_certificate(domain, ssl_certificate_primary, None, just_check_domain=True)[0] == "OK":
+			if check_certificate(domain, ssl_certificate_primary, None)[0] == "OK":
 				ssl_certificate = ssl_certificate_primary
 				ssl_via = "Using multi/wildcard certificate of %s." % env['PRIMARY_HOSTNAME']
 
 			# For a 'www.' domain, see if we can reuse the cert of the parent.
 			elif domain.startswith('www.'):
 				ssl_certificate_parent = os.path.join(env["STORAGE_ROOT"], 'ssl/%s/ssl_certificate.pem' % safe_domain_name(domain[4:]))
-				if os.path.exists(ssl_certificate_parent) and check_certificate(domain, ssl_certificate_parent, None, just_check_domain=True)[0] == "OK":
+				if os.path.exists(ssl_certificate_parent) and check_certificate(domain, ssl_certificate_parent, None)[0] == "OK":
 					ssl_certificate = ssl_certificate_parent
 					ssl_via = "Using multi/wildcard certificate of %s." % domain[4:]
 
@@ -254,7 +254,7 @@ def create_csr(domain, ssl_key, env):
                 "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain)])
 
 def install_cert(domain, ssl_cert, ssl_chain, env):
-	if domain not in get_web_domains(env) + get_default_www_redirects(env):
+	if domain not in get_web_domains(env):
 		return "Invalid domain name."
 
 	# Write the combined cert+chain to a temporary path and validate that it is OK.

security.md

@@ -85,11 +85,7 @@ If the recipient's domain name supports DNSSEC and has published a [DANE TLSA](h
 
 ### Domain Policy Records
 
-Domain policy records allow recipient MTAs to detect when the _domain_ part of of the sender address in incoming mail has been spoofed. All outbound mail is signed with [DKIM](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) and "quarantine" [DMARC](https://en.wikipedia.org/wiki/DMARC) records are automatically set in DNS. Receiving MTAs that implement DMARC will automatically quarantine mail that is "From:" a domain hosted by the box but which was not sent by the box. (Strong [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records are also automatically set in DNS.) ([source](management/dns_update.py))
-
-### User Policy
-
-While domain policy records prevent other servers from sending mail with a "From:" header that matches a domain hosted on the box (see above), those policy records do not guarnatee that the user portion of the sender email address matches the actual sender. In enterprise environments where the box may host the mail of untrusted users, it is important to guard against users impersonating other users. The box restricts the envelope sender address that users may put into outbound mail to either a) their own email address (their SMTP login username) or b) any alias that they are listed as a direct recipient of. Note that the envelope sender address is not the same as the "From:" header.
+Domain policy records allow recipient MTAs to detect when the _domain_ part of incoming mail has been spoofed. All outbound mail is signed with [DKIM](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) and "quarantine" [DMARC](https://en.wikipedia.org/wiki/DMARC) records are automatically set in DNS. Receiving MTAs that implement DMARC will automatically quarantine mail that is "From:" a domain hosted by the box but which was not sent by the box. (Strong [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records are also automatically set in DNS.) ([source](management/dns_update.py))
 
 Incoming Mail
 -------------

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.12
+	TAG=v0.10
 fi
 
 # Are we running as root?

setup/dkim.sh

@@ -35,18 +35,13 @@ RequireSafeKeys         false
 EOF
 fi
 
-# Create a new DKIM key. This creates mail.private and mail.txt
-# in $STORAGE_ROOT/mail/dkim. The former is the private key and
-# the latter is the suggested DNS TXT entry which we'll include
-# in our DNS setup. Note tha the files are named after the
-# 'selector' of the key, which we can change later on to support
-# key rotation.
-#
-# A 1024-bit key is seen as a minimum standard by several providers
-# such as Google. But they and others use a 2048 bit key, so we'll
-# do the same. Keys beyond 2048 bits may exceed DNS record limits.
+# Create a new DKIM key. This creates
+# mail.private and mail.txt in $STORAGE_ROOT/mail/dkim. The former
+# is the actual private key and the latter is the suggested DNS TXT
+# entry which we'll want to include in our DNS setup.
 if [ ! -f "$STORAGE_ROOT/mail/dkim/mail.private" ]; then
-	opendkim-genkey -b 2048 -r -s mail -D $STORAGE_ROOT/mail/dkim
+	# Should we specify -h rsa-sha256?
+	opendkim-genkey -r -s mail -D $STORAGE_ROOT/mail/dkim
 fi
 
 # Ensure files are owned by the opendkim user and are private otherwise.

setup/dns.sh

@@ -127,6 +127,8 @@ EOF
 chmod +x /etc/cron.daily/mailinabox-dnssec
 
 # Permit DNS queries on TCP/UDP in the firewall.
-
 ufw_allow domain
 
+# Start nsd. None of the zones are configured until the management daemon is
+# run later, though.
+restart_service nsd

setup/functions.sh

@@ -39,8 +39,14 @@ function apt_get_quiet {
 }
 
 function apt_install {
-	# Report any packages already installed.
 	PACKAGES=$@
+
+	if [ ! -z "$IS_DOCKER" ]; then
+		# Speed things up because packages are already installed by the image.
+		PACKAGES=""
+	fi
+			
+	# Report any packages already installed.
 	TO_INSTALL=""
 	ALREADY_INSTALLED=""
 	for pkg in $PACKAGES; do
@@ -72,12 +78,20 @@ function get_default_hostname {
 	# Guess the machine's hostname. It should be a fully qualified
 	# domain name suitable for DNS. None of these calls may provide
 	# the right value, but it's the best guess we can make.
-	set -- $(hostname --fqdn      2>/dev/null ||
-                 hostname --all-fqdns 2>/dev/null ||
-                 hostname             2>/dev/null)
+	set -- $(
+		get_hostname_from_reversedns ||
+		hostname --fqdn      2>/dev/null ||
+		hostname --all-fqdns 2>/dev/null ||
+		hostname             2>/dev/null)
 	printf '%s\n' "$1" # return this value
 }
 
+function get_hostname_from_reversedns {
+	# Do a reverse DNS lookup on our public IPv4 address. The output of
+	# `host` is complex -- use sed to get the FDQN.
+	host $(get_publicip_from_web_service 4) | sed "s/.*pointer \(.*\)\./\1/"
+}
+
 function get_publicip_from_web_service {
 	# This seems to be the most reliable way to determine the
 	# machine's public IP address: asking a very nice web API
@@ -153,7 +167,17 @@ function ufw_allow {
 }
 
 function restart_service {
-	hide_output service $1 restart
+	if [ -z "$IS_DOCKER" ]; then
+		# Restart the service.
+		hide_output service $1 restart
+
+	else
+		# In Docker, make sure the service is not disabled by a down file.
+		if [ -f /etc/service/$1/down ]; then
+			rm /etc/service/$1/down
+		fi
+		sv restart $1
+	fi
 }
 
 ## Dialog Functions ##

setup/mail-postfix.sh

@@ -160,7 +160,6 @@ tools/editconf.py /etc/postfix/main.cf virtual_transport=lmtp:[127.0.0.1]:10025
 #
 # * `reject_non_fqdn_sender`: Reject not-nice-looking return paths.
 # * `reject_unknown_sender_domain`: Reject return paths with invalid domains.
-# * `reject_authenticated_sender_login_mismatch`: Reject if mail FROM address does not match the client SASL login
 # * `reject_rhsbl_sender`: Reject return paths that use blacklisted domains.
 # * `permit_sasl_authenticated`: Authenticated users (i.e. on port 587) can skip further checks.
 # * `permit_mynetworks`: Mail that originates locally can skip further checks.
@@ -174,7 +173,7 @@ tools/editconf.py /etc/postfix/main.cf virtual_transport=lmtp:[127.0.0.1]:10025
 # whitelisted) then postfix does a DEFER_IF_REJECT, which results in all "unknown user" sorts of messages turning into #NODOC
 # "450 4.7.1 Client host rejected: Service unavailable". This is a retry code, so the mail doesn't properly bounce. #NODOC
 tools/editconf.py /etc/postfix/main.cf \
-	smtpd_sender_restrictions="reject_non_fqdn_sender,reject_unknown_sender_domain,reject_authenticated_sender_login_mismatch,reject_rhsbl_sender dbl.spamhaus.org" \
+	smtpd_sender_restrictions="reject_non_fqdn_sender,reject_unknown_sender_domain,reject_rhsbl_sender dbl.spamhaus.org" \
 	smtpd_recipient_restrictions=permit_sasl_authenticated,permit_mynetworks,"reject_rbl_client zen.spamhaus.org",reject_unlisted_recipient,"check_policy_service inet:127.0.0.1:10023"
 
 # Postfix connects to Postgrey on the 127.0.0.1 interface specifically. Ensure that

setup/mail-users.sh

@@ -69,22 +69,6 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtpd_sasl_path=private/auth \
 	smtpd_sasl_auth_enable=yes
 
-# ### Sender Validation
-
-# Use a Sqlite3 database to set login maps. This is used with
-# reject_authenticated_sender_login_mismatch to see if user is
-# allowed to send mail using FROM field specified in the request.
-tools/editconf.py /etc/postfix/main.cf \
-	smtpd_sender_login_maps=sqlite:/etc/postfix/sender-login-maps.cf
-
-# SQL statement to set login map which includes the case when user is
-# sending email using a valid alias.
-# This is the same as virtual-alias-maps.cf, See below
-cat > /etc/postfix/sender-login-maps.cf << EOF;
-dbpath=$db_path
-query = SELECT destination from (SELECT destination, 0 as priority FROM aliases WHERE source='%s' UNION SELECT email as destination, 1 as priority FROM users WHERE email='%s') ORDER BY priority LIMIT 1;
-EOF
-
 # ### Destination Validation
 
 # Use a Sqlite3 database to check whether a destination email address exists,
@@ -108,25 +92,13 @@ query = SELECT 1 FROM users WHERE email='%s'
 EOF
 
 # SQL statement to rewrite an email address if an alias is present.
-#
-# Postfix makes multiple queries for each incoming mail. It first
-# queries the whole email address, then just the user part in certain
-# locally-directed cases (but we don't use this), then just `@`+the
-# domain part. The first query that returns something wins. See
-# http://www.postfix.org/virtual.5.html.
-#
-# virtual-alias-maps has precedence over virtual-mailbox-maps, but
-# we don't want catch-alls and domain aliases to catch mail for users
-# that have been defined on those domains. To fix this, we not only
-# query the aliases table but also the users table when resolving
-# aliases, i.e. we turn users into aliases from themselves to
-# themselves. That means users will match in postfix's first query
-# before postfix gets to the third query for catch-alls/domain alises.
-#
+# Aliases have precedence over users, but that's counter-intuitive for
+# catch-all aliases ("@domain.com") which should *not* catch mail users.
+# To fix this, not only query the aliases table but also the users
+# table, i.e. turn users into aliases from themselves to themselves.
 # If there is both an alias and a user for the same address either
 # might be returned by the UNION, so the whole query is wrapped in
-# another select that prioritizes the alias definition to preserve
-# postfix's preference for aliases for whole email addresses.
+# another select that prioritizes the alias definition.
 cat > /etc/postfix/virtual-alias-maps.cf << EOF;
 dbpath=$db_path
 query = SELECT destination from (SELECT destination, 0 as priority FROM aliases WHERE source='%s' UNION SELECT email as destination, 1 as priority FROM users WHERE email='%s') ORDER BY priority LIMIT 1;

setup/management.sh

@@ -2,10 +2,8 @@
 
 source setup/functions.sh
 
-# build-essential libssl-dev libffi-dev python3-dev: Required to pip install cryptography.
-apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil \
-	build-essential libssl-dev libffi-dev python3-dev
-hide_output pip3 install --upgrade rtyaml email_validator idna cryptography
+apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil
+hide_output pip3 install rtyaml "email_validator==0.1.0-rc5"
 	# email_validator is repeated in setup/questions.sh
 
 # Create a backup directory and a random key for encrypting backups.

setup/migrate.py

@@ -95,11 +95,6 @@ def migration_7(env):
 	# Save.
 	conn.commit()
 
-def migration_8(env):
-	# Delete DKIM keys. We had generated 1024-bit DKIM keys.
-	# By deleting the key file we'll automatically generate
-	# a new key, which will be 2048 bits.
-	os.unlink(os.path.join(env['STORAGE_ROOT'], 'mail/dkim/mail.private'))
 
 def get_current_migration():
 	ver = 0

setup/owncloud.sh

@@ -34,17 +34,16 @@ fi
 if [ ! -d /usr/local/lib/owncloud/ ] \
 	|| ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then
 
-	# Download and verify
-	echo "installing ownCloud..."
-	wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
-
 	# Clear out the existing ownCloud.
-	if [ -d /usr/local/lib/owncloud/ ]; then
+	if [ ! -d /usr/local/lib/owncloud/ ]; then
+		echo installing ownCloud...
+	else
 		echo "upgrading ownCloud to $owncloud_ver (backing up existing ownCloud directory to /tmp/owncloud-backup-$$)..."
 		mv /usr/local/lib/owncloud /tmp/owncloud-backup-$$
 	fi
 
-	# Extract ownCloud
+	# Download and extract ownCloud.
+	wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
 	unzip -u -o -q /tmp/owncloud.zip -d /usr/local/lib #either extracts new or replaces current files
 	rm -f /tmp/owncloud.zip
 
@@ -185,4 +184,5 @@ chmod +x /etc/cron.hourly/mailinabox-owncloud
 
 # Enable PHP modules and restart PHP.
 php5enmod imap
+restart_service memcached
 restart_service php5-fpm

setup/preflight.sh

@@ -4,7 +4,7 @@ if [[ $EUID -ne 0 ]]; then
 	echo
 	echo "sudo $0"
 	echo
-	exit
+	exit 1
 fi
 
 # Check that we are running on Ubuntu 14.04 LTS (or 14.04.xx).
@@ -14,7 +14,7 @@ if [ "`lsb_release -d | sed 's/.*:\s*//' | sed 's/14\.04\.[0-9]/14.04/' `" != "U
 	lsb_release -d | sed 's/.*:\s*//'
 	echo
 	echo "We can't write scripts that run on every possible setup, sorry."
-	exit
+	exit 1
 fi
 
 # Check that we have enough memory.
@@ -30,6 +30,6 @@ if [ ! -d /vagrant ]; then
 	echo "Your Mail-in-a-Box needs more memory (RAM) to function properly."
 	echo "Please provision a machine with at least 768 MB, 1 GB recommended."
 	echo "This machine has $TOTAL_PHYSICAL_MEM MB memory."
-	exit
+	exit 1
 fi
 fi

setup/questions.sh

@@ -12,8 +12,10 @@ if [ -z "$NONINTERACTIVE" ]; then
 		apt_get_quiet install dialog python3 python3-pip  || exit 1
 	fi
 
+	if [ -z "$IS_DOCKER" ]; then
 	# email_validator is repeated in setup/management.sh
-	hide_output pip3 install email_validator || exit 1
+	hide_output pip3 install "email_validator==0.1.0-rc5" || exit 1
+	fi
 
 	message_box "Mail-in-a-Box Installation" \
 		"Hello and thanks for deploying a Mail-in-a-Box!

setup/start.sh

@@ -69,7 +69,7 @@ if [ ! -d $STORAGE_ROOT ]; then
 fi
 if [ ! -f $STORAGE_ROOT/mailinabox.version ]; then
 	echo $(setup/migrate.py --current) > $STORAGE_ROOT/mailinabox.version
-	chown $STORAGE_USER.$STORAGE_USER $STORAGE_ROOT/mailinabox.version
+	chown $STORAGE_USER:$STORAGE_USER $STORAGE_ROOT/mailinabox.version
 fi
 
 

setup/system.sh

@@ -11,21 +11,12 @@ source setup/functions.sh # load our functions
 # text search plugin for (and by) dovecot, which is not available in
 # Ubuntu currently.
 #
-# Add that to the system's list of repositories using add-apt-repository.
-# But add-apt-repository may not be installed. If it's not available,
-# then install it. But we have to run apt-get update before we try to
-# install anything so the package index is up to date. After adding the
-# PPA, we have to run apt-get update *again* to load the PPA's index,
-# so this must precede the apt-get update line below.
-
-if [ ! -f /usr/bin/add-apt-repository ]; then
-	echo "Installing add-apt-repository..."
-	hide_output apt-get update
-	apt_install software-properties-common
-fi
+# Add that to the system's list of repositories:
 
 hide_output add-apt-repository -y ppa:mail-in-a-box/ppa
 
+# The apt-get update in the next step will pull in the PPA's index.
+
 # ### Update Packages
 
 # Update system packages to make sure we have the latest upstream versions of things from Ubuntu.

setup/web.sh

@@ -36,10 +36,6 @@ tools/editconf.py /etc/nginx/nginx.conf -s \
 tools/editconf.py /etc/php5/fpm/php.ini -c ';' \
 	expose_php=Off
 
-# Set PHPs default charset to UTF-8, since we use it. See #367.
-tools/editconf.py /etc/php5/fpm/php.ini -c ';' \
-        default_charset="UTF-8"
-
 # Bump up PHP's max_children to support more concurrent connections
 tools/editconf.py /etc/php5/fpm/pool.d/www.conf -c ';' \
 	pm.max_children=8

setup/webmail.sh

@@ -126,7 +126,7 @@ EOF
 
 # Create writable directories.
 mkdir -p /var/log/roundcubemail /tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
-chown -R www-data.www-data /var/log/roundcubemail /tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
+chown -R www-data:www-data /var/log/roundcubemail /tmp/roundcubemail $STORAGE_ROOT/mail/roundcube
 
 # Password changing plugin settings
 # The config comes empty by default, so we need the settings 
@@ -147,9 +147,9 @@ usermod -a -G dovecot www-data
 
 # set permissions so that PHP can use users.sqlite
 # could use dovecot instead of www-data, but not sure it matters
-chown root.www-data $STORAGE_ROOT/mail
+chown root:www-data $STORAGE_ROOT/mail
 chmod 775 $STORAGE_ROOT/mail
-chown root.www-data $STORAGE_ROOT/mail/users.sqlite 
+chown root:www-data $STORAGE_ROOT/mail/users.sqlite
 chmod 664 $STORAGE_ROOT/mail/users.sqlite 
 
 # Enable PHP modules.

tools/editconf.py

@@ -14,8 +14,6 @@
 #
 # NAME VALUE
 #
-# If the -c option is given, then the supplied character becomes the comment character
-#
 # If the -w option is given, then setting lines continue onto following
 # lines while the lines start with whitespace, e.g.:
 #
@@ -26,7 +24,7 @@ import sys, re
 
 # sanity check
 if len(sys.argv) < 3:
-	print("usage: python3 editconf.py /etc/file.conf [-s] [-w] [-c <CHARACTER>] [-t] NAME=VAL [NAME=VAL ...]")
+	print("usage: python3 editconf.py /etc/file.conf [-s] [-w] [-t] NAME=VAL [NAME=VAL ...]")
 	sys.exit(1)
 
 # parse command line arguments

tools/list_all_packages.py

@@ -0,0 +1,23 @@
+#!/usr/bin/python3
+
+import os.path, glob, re
+
+packages = set()
+
+def add(line):
+	global packages
+	if line.endswith("\\"): line = line[:-1]
+	packages |= set(p for p in line.split(" ") if p not in("", "apt_install"))
+
+for fn in glob.glob(os.path.join(os.path.dirname(__file__), "../setup/*.sh")):
+	with open(fn) as f:
+		in_apt_install = False
+		for line in f:
+			line = line.strip()
+			if line.startswith("apt_install "):
+				in_apt_install = True
+			if in_apt_install:
+				add(line)
+			in_apt_install = in_apt_install and line.endswith("\\")
+
+print("\n".join(sorted(packages)))

tools/parse-nginx-log-bootstrap-accesses.py

@@ -25,7 +25,6 @@ for fn in glob.glob("/var/log/nginx/access.log*"):
 	with f:
 		for line in f:
 			# Find lines that are GETs on /bootstrap.sh by either curl or wget.
-			# (Note that we purposely skip ...?ping=1 requests which is the admin panel querying us for updates.)
 			m = re.match(rb"(?P<ip>\S+) - - \[(?P<date>.*?)\] \"GET /bootstrap.sh HTTP/.*\" 200 \d+ .* \"(?:curl|wget)", line, re.I)
 			if m:
 				date, time = m.group("date").decode("ascii").split(":", 1)