bump bootstrap.sh to incoming 0.03 tag

fixes #195

README.md

@@ -1,49 +1,32 @@
 Mail-in-a-Box
 =============
 
-By [@JoshData](https://github.com/JoshData) and contributors.
+By [@JoshData](https://github.com/JoshData) and [contributors](https://github.com/mail-in-a-box/mailinabox/graphs/contributors).
 
 Mail-in-a-Box helps individuals take back control of their email by defining a one-click, easy-to-deploy SMTP+everything else server: a mail server in a box.
 
-**This is a work in progress. I work on this in my limited free time.**
+**Please see [https://mailinabox.email](https://mailinabox.email) for the project's website and setup guide!**
 
-Why build this? Mass electronic surveillance by governments revealed over the last year has spurred a new movement to [re-decentralize](http://redecentralize.org/) the web, that is, to empower netizens to be their own service providers again. SMTP, the protocol of email, is decentralized in principle but highly centralized in practice due to the high cost of implementing all of the modern protocols that surround it. As a result, most individuals trade their independence for access to a “free” email service.
+* * *
 
+I am trying to:
+
+* Make deploying a good mail server easy.
+* Promote [decentralization](http://redecentralize.org/), innovation, and privacy on the web.
+* Have automated, auditable, and [idempotent](http://sharknet.us/2014/02/01/automated-configuration-management-challenges-with-idempotency/) configuration.
+* **Not** be a mail server that the NSA cannot hack.
+* **Not** be customizable by power users.
+
+The long-term goal is to have this be a one-click email appliance with *no* user-configurable setup options.
+
+For more background, see [The Rationale](https://github.com/mail-in-a-box/mailinabox/wiki).
+
+This setup is what has been powering my own personal email since September 2013.
 
 The Box
 -------
 
-Mail-in-a-Box turns a fresh Ubuntu 14.04 LTS 64-bit machine into a working mail server, including SMTP ([postfix](http://www.postfix.org/)), IMAP ([dovecot](http://dovecot.org/)), Exchange ActiveSync ([z-push](https://github.com/fmbiete/Z-Push-contrib)), webmail ([Roundcube](http://roundcube.net/)), spam filtering ([spamassassin](https://spamassassin.apache.org/)), greylisting ([postgrey](http://postgrey.schweikert.ch/)), CardDAV/CalDAV ([ownCloud](http://owncloud.org/)), DNS, [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework), DKIM ([OpenDKIM](http://www.opendkim.org/)), [DMARC](https://en.wikipedia.org/wiki/DMARC), [DNSSEC](https://en.wikipedia.org/wiki/DNSSEC), [DANE TLSA](https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities), and basic system services like a firewall, intrusion protection, and setting the system clock.
-
-This setup is what has been powering my own personal email since September 2013.
-
-Please see [mailinabox.email](https://mailinabox.email) for more information and how to set up a Mail-in-a-Box.
-
-In short, it's like this:
-
-	# do this on a fresh install of Ubuntu 14.04 only!
-	sudo apt-get install -y git
-	git clone https://github.com/mail-in-a-box/mailinabox
-	cd mailinabox
-	sudo setup/start.sh
-
-Congratulations! You should now have a working setup. You'll be given the address of the administrative interface for further instructions.
-
-**Status**: This is a work in progress. It works for what it is, but it is missing such things as quotas, backup/restore, etc.
-
-The Goals
----------
-
-* Create a push-button "Email Appliance" for everyday users.
-* Promote decentralization, innovation, and privacy on the web.
-* Have automated, auditable, and [idempotent](http://sharknet.us/2014/02/01/automated-configuration-management-challenges-with-idempotency/) configuration.
-
-For more background, see [The Rationale](https://github.com/mail-in-a-box/mailinabox/wiki).
-
-What I am not trying to do:
-
-* **Not** to be a mail server that the NSA cannot hack.
-* **Not** to be customizable by power users.
+Mail-in-a-Box turns a fresh Ubuntu 14.04 LTS 64-bit machine into a working mail server, including SMTP ([postfix](http://www.postfix.org/)), IMAP ([dovecot](http://dovecot.org/)), Exchange ActiveSync ([z-push](https://github.com/fmbiete/Z-Push-contrib)), webmail ([Roundcube](http://roundcube.net/)), spam filtering ([spamassassin](https://spamassassin.apache.org/)), greylisting ([postgrey](http://postgrey.schweikert.ch/)), CardDAV/CalDAV ([ownCloud](http://owncloud.org/)), DNS, [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework), DKIM ([OpenDKIM](http://www.opendkim.org/)), [DMARC](https://en.wikipedia.org/wiki/DMARC), [DNSSEC](https://en.wikipedia.org/wiki/DNSSEC), [DANE TLSA](https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities), [SSHFP](https://tools.ietf.org/html/rfc4255), and basic system services like a firewall, intrusion protection, and setting the system clock.
 
 The Acknowledgements
 --------------------
@@ -57,3 +40,4 @@ The History
 
 * In 2007 I wrote a relatively popular Mozilla Thunderbird extension that added client-side SPF and DKIM checks to mail to warn users about possible phishing: [add-on page](https://addons.mozilla.org/en-us/thunderbird/addon/sender-verification-anti-phish/), [source](https://github.com/JoshData/thunderbird-spf).
 * Mail-in-a-Box was a semifinalist in the 2014 [Knight News Challenge](https://www.newschallenge.org/challenge/2014/submissions/mail-in-a-box), but it was not selected as a winner.
+* Mail-in-a-Box hit the front page of Hacker News in [April](https://news.ycombinator.com/item?id=7634514) and [September](https://news.ycombinator.com/item?id=8276171) 2014.

Vagrantfile

@@ -18,6 +18,7 @@ Vagrant.configure("2") do |config|
 	# machine figure out its own public IP and it'll take a
 	# subdomain on our justtesting.email domain so we can get
 	# started quickly.
+    export NONINTERACTIVE=1
     export PUBLIC_IP=auto
     export PUBLIC_IPV6=auto
     export PRIMARY_HOSTNAME=auto-easy

conf/management-initscript

@@ -12,7 +12,7 @@
 # Adapted from http://blog.codefront.net/2007/06/11/nginx-php-and-a-php-fastcgi-daemon-init-script/
 
 PATH=/sbin:/usr/sbin:/bin:/usr/bin
-DESC="Mail-in-a-Box Mgmt"
+DESC="Mail-in-a-Box Management Daemon"
 NAME=mailinabox
 DAEMON=/usr/local/bin/mailinabox-daemon
 PIDFILE=/var/run/$NAME.pid

conf/nginx-primaryonly.conf

@@ -1,6 +1,14 @@
+	# Control Panel
+	rewrite ^/admin$ /admin/;
+	location /admin/ {
+		proxy_pass http://localhost:10222/;
+		proxy_set_header X-Forwarded-For $remote_addr;
+	}
+
 	# ownCloud configuration.
 	rewrite ^/cloud$ /cloud/ redirect;
 	rewrite ^/cloud/$ /cloud/index.php;
+	rewrite ^/cloud/(contacts|calendar|files)$ /cloud/index.php/apps/$1/ redirect;
 	rewrite ^(/cloud/core/doc/[^\/]+/)$ $1/index.html;
 	location /cloud/ {
 		alias /usr/local/lib/owncloud/;

conf/nginx-top.conf

@@ -1,6 +1,10 @@
 ## NOTE: This file is automatically generated by Mail-in-a-Box.
-##       Do not edit this file. It will be replaced each time
-##       Mail-in-a-Box needs to update the web configuration.
+##       Do not edit this file. It is continually updated by
+##       Mail-in-a-Box and your changes will be lost.
+##
+##       Mail-in-a-Box machines are not meant to be modified.
+##       If you modify any system configuration you are on
+##       your own --- please do not ask for help from us.
 
 upstream php-fpm {
 	server unix:/var/run/php5-fpm.sock;

conf/nginx.conf

@@ -24,12 +24,6 @@ server {
 	root $ROOT;
 	index index.html index.htm;
 
-	# Control Panel
-	rewrite ^/admin$ /admin/;
-	location /admin/ {
-		proxy_pass http://localhost:10222/;
-	}
-
 	# Roundcube Webmail configuration.
 	rewrite ^/mail$ /mail/ redirect;
 	rewrite ^/mail/$ /mail/index.php;

management/backup.py

@@ -2,119 +2,207 @@
 
 # This script performs a backup of all user data:
 # 1) System services are stopped while a copy of user data is made.
-# 2) An incremental backup is made using rdiff-backup into the
-#    directory STORAGE_ROOT/backup/rdiff-history. This directory
-#    will contain the latest files plus a complete history for
-#    all prior backups.
+# 2) An incremental backup is made using duplicity into the
+#    directory STORAGE_ROOT/backup/duplicity.
 # 3) The stopped services are restarted.
-# 4) The backup directory is compressed into a single file using tar.
-# 5) That file is encrypted with a long password stored in backup/secret_key.txt.
+# 4) The backup files are encrypted with a long password (stored in
+#    backup/secret_key.txt) to STORAGE_ROOT/backup/encrypted.
+# 5) STORAGE_ROOT/backup/after-backup is executd if it exists.
 
-import sys, os, os.path, shutil, glob
+import os, os.path, shutil, glob, re, datetime
+import dateutil.parser, dateutil.relativedelta, dateutil.tz
 
 from utils import exclusive_process, load_environment, shell
 
-# settings
-full_backup = "--full" in sys.argv
-keep_backups_for = "31D" # destroy backups older than 31 days except the most recent full backup
+# destroy backups when the most recent increment in the chain
+# that depends on it is this many days old.
+keep_backups_for_days = 14
 
-env = load_environment()
+def backup_status(env):
+	# What is the current status of backups?
+	# Loop through all of the files in STORAGE_ROOT/backup/duplicity to
+	# get a list of all of the backups taken and sum up file sizes to
+	# see how large the storage is.
 
-exclusive_process("backup")
+	now = datetime.datetime.now(dateutil.tz.tzlocal())
+	def reldate(date, ref):
+		rd = dateutil.relativedelta.relativedelta(ref, date)
+		if rd.days >= 7: return "%d days" % rd.days
+		if rd.days > 1: return "%d days, %d hours" % (rd.days, rd.hours)
+		if rd.days == 1: return "%d day, %d hours" % (rd.days, rd.hours)
+		return "%d hours, %d minutes" % (rd.hours, rd.minutes)
 
-# Ensure the backup directory exists.
-backup_dir = os.path.join(env["STORAGE_ROOT"], 'backup')
-backup_duplicity_dir = os.path.join(backup_dir, 'duplicity')
-os.makedirs(backup_dir, exist_ok=True)
+	backups = { }
+	basedir = os.path.join(env['STORAGE_ROOT'], 'backup/duplicity/')
+	encdir = os.path.join(env['STORAGE_ROOT'], 'backup/encrypted/')
+	for fn in os.listdir(basedir):
+		m = re.match(r"duplicity-(full|full-signatures|(inc|new-signatures)\.(?P<incbase>\d+T\d+Z)\.to)\.(?P<date>\d+T\d+Z)\.", fn)
+		if not m: raise ValueError(fn)
 
-# On the first run, always do a full backup. Incremental
-# will fail.
-if len(os.listdir(backup_duplicity_dir)) == 0:
-	full_backup = True
-else:
-	# When the size of incremental backups exceeds the size of existing
-	# full backups, take a new full backup. We want to avoid full backups
-	# because they are costly to synchronize off-site.
-	full_sz = sum(os.path.getsize(f) for f in glob.glob(backup_duplicity_dir + '/*-full.*'))
-	inc_sz = sum(os.path.getsize(f) for f in glob.glob(backup_duplicity_dir + '/*-inc.*'))
-	# (n.b. not counting size of new-signatures files because they are relatively small)
-	if inc_sz > full_sz * 1.5:
-		full_backup = True
+		key = m.group("date")
+		if key not in backups:
+			date = dateutil.parser.parse(m.group("date"))
+			backups[key] = {
+				"date": m.group("date"),
+				"date_str": date.strftime("%x %X"),
+				"date_delta": reldate(date, now),
+				"full": m.group("incbase") is None,
+				"previous": m.group("incbase"),
+				"size": 0,
+				"encsize": 0,
+			}
 
-# Stop services.
-shell('check_call', ["/usr/sbin/service", "dovecot", "stop"])
-shell('check_call', ["/usr/sbin/service", "postfix", "stop"])
+		backups[key]["size"] += os.path.getsize(os.path.join(basedir, fn))
 
-# Update the backup mirror directory which mirrors the current
-# STORAGE_ROOT (but excluding the backups themselves!).
-try:
+		# Also check encrypted size.
+		encfn = os.path.join(encdir, fn + ".enc")
+		if os.path.exists(encfn):
+			backups[key]["encsize"] += os.path.getsize(encfn)
+
+	# Ensure the rows are sorted reverse chronologically.
+	# This is relied on by should_force_full() and the next step.
+	backups = sorted(backups.values(), key = lambda b : b["date"], reverse=True)
+
+	# When will a backup be deleted?
+	saw_full = False
+	deleted_in = None
+	days_ago = now - datetime.timedelta(days=keep_backups_for_days)
+	for bak in backups:
+		if deleted_in:
+			# Subsequent backups are deleted when the most recent increment
+			# in the chain would be deleted.
+			bak["deleted_in"] = deleted_in
+		if bak["full"]:
+			# Reset when we get to a full backup. A new chain start next.
+			saw_full = True
+			deleted_in = None
+		elif saw_full and not deleted_in:
+			# Mark deleted_in only on the first increment after a full backup.
+			deleted_in = reldate(days_ago, dateutil.parser.parse(bak["date"]))
+			bak["deleted_in"] = deleted_in
+
+	return {
+		"directory": basedir,
+		"encpwfile": os.path.join(env['STORAGE_ROOT'], 'backup/secret_key.txt'),
+		"encdirectory": encdir,
+		"tz": now.tzname(),
+		"backups": backups,
+	}
+
+def should_force_full(env):
+	# Force a full backup when the total size of the increments
+	# since the last full backup is greater than half the size
+	# of that full backup.
+	inc_size = 0
+	for bak in backup_status(env)["backups"]:
+		if not bak["full"]:
+			# Scan through the incremental backups cumulating
+			# size...
+			inc_size += bak["size"]
+		else:
+			# ...until we reach the most recent full backup.
+			# Return if we should to a full backup.
+			return inc_size > .5*bak["size"]
+	else:
+		# If we got here there are no (full) backups, so make one.
+		# (I love for/else blocks. Here it's just to show off.)
+		return True
+
+def perform_backup(full_backup):
+	env = load_environment()
+
+	exclusive_process("backup")
+
+	# Ensure the backup directory exists.
+	backup_dir = os.path.join(env["STORAGE_ROOT"], 'backup')
+	backup_duplicity_dir = os.path.join(backup_dir, 'duplicity')
+	os.makedirs(backup_duplicity_dir, exist_ok=True)
+
+	# On the first run, always do a full backup. Incremental
+	# will fail. Otherwise do a full backup when the size of
+	# the increments since the most recent full backup are
+	# large.
+	full_backup = full_backup or should_force_full(env)
+
+	# Stop services.
+	shell('check_call', ["/usr/sbin/service", "dovecot", "stop"])
+	shell('check_call', ["/usr/sbin/service", "postfix", "stop"])
+
+	# Update the backup mirror directory which mirrors the current
+	# STORAGE_ROOT (but excluding the backups themselves!).
+	try:
+		shell('check_call', [
+			"/usr/bin/duplicity",
+			"full" if full_backup else "incr",
+			"--no-encryption",
+			"--archive-dir", "/tmp/duplicity-archive-dir",
+			"--name", "mailinabox",
+			"--exclude", backup_dir,
+			"--volsize", "100",
+			"--verbosity", "warning",
+			env["STORAGE_ROOT"],
+			"file://" + backup_duplicity_dir
+			])
+	finally:
+		# Start services again.
+		shell('check_call', ["/usr/sbin/service", "dovecot", "start"])
+		shell('check_call', ["/usr/sbin/service", "postfix", "start"])
+
+	# Remove old backups. This deletes all backup data no longer needed
+	# from more than 31 days ago. Must do this before destroying the
+	# cache directory or else this command will re-create it.
 	shell('check_call', [
 		"/usr/bin/duplicity",
-		"full" if full_backup else "incr",
-		"--no-encryption",
+		"remove-older-than",
+		"%dD" % keep_backups_for_days,
 		"--archive-dir", "/tmp/duplicity-archive-dir",
 		"--name", "mailinabox",
-		"--exclude", backup_dir,
-		"--volsize", "100",
+		"--force",
 		"--verbosity", "warning",
-		env["STORAGE_ROOT"],
 		"file://" + backup_duplicity_dir
 		])
-finally:
-	# Start services again.
-	shell('check_call', ["/usr/sbin/service", "dovecot", "start"])
-	shell('check_call', ["/usr/sbin/service", "postfix", "start"])
 
-# Remove old backups. This deletes all backup data no longer needed
-# from more than 31 days ago. Must do this before destroying the
-# cache directory or else this command will re-create it.
-shell('check_call', [
-	"/usr/bin/duplicity",
-	"remove-older-than",
-	keep_backups_for,
-	"--archive-dir", "/tmp/duplicity-archive-dir",
-	"--name", "mailinabox",
-	"--force",
-	"--verbosity", "warning",
-	"file://" + backup_duplicity_dir
-	])
+	# Remove duplicity's cache directory because it's redundant with our backup directory.
+	shutil.rmtree("/tmp/duplicity-archive-dir")
 
-# Remove duplicity's cache directory because it's redundant with our backup directory.
-shutil.rmtree("/tmp/duplicity-archive-dir")
+	# Encrypt all of the new files.
+	backup_encrypted_dir = os.path.join(backup_dir, 'encrypted')
+	os.makedirs(backup_encrypted_dir, exist_ok=True)
+	for fn in os.listdir(backup_duplicity_dir):
+		fn2 = os.path.join(backup_encrypted_dir, fn) + ".enc"
+		if os.path.exists(fn2): continue
 
-# Encrypt all of the new files.
-backup_encrypted_dir = os.path.join(backup_dir, 'encrypted')
-os.makedirs(backup_encrypted_dir, exist_ok=True)
-for fn in os.listdir(backup_duplicity_dir):
-	fn2 = os.path.join(backup_encrypted_dir, fn) + ".enc"
-	if os.path.exists(fn2): continue
+		# Encrypt the backup using the backup private key.
+		shell('check_call', [
+			"/usr/bin/openssl",
+			"enc",
+			"-aes-256-cbc",
+			"-a",
+			"-salt",
+			"-in", os.path.join(backup_duplicity_dir, fn),
+			"-out", fn2,
+			"-pass", "file:%s" % os.path.join(backup_dir, "secret_key.txt"),
+			])
 
-	# Encrypt the backup using the backup private key.
-	shell('check_call', [
-		"/usr/bin/openssl",
-		"enc",
-		"-aes-256-cbc",
-		"-a",
-		"-salt",
-		"-in", os.path.join(backup_duplicity_dir, fn),
-		"-out", fn2,
-		"-pass", "file:%s" % os.path.join(backup_dir, "secret_key.txt"),
-		])
+		# The backup can be decrypted with:
+		# openssl enc -d -aes-256-cbc -a -in latest.tgz.enc -out /dev/stdout -pass file:secret_key.txt | tar -z
 
-	# The backup can be decrypted with:
-	# openssl enc -d -aes-256-cbc -a -in latest.tgz.enc -out /dev/stdout -pass file:secret_key.txt | tar -z
+	# Remove encrypted backups that are no longer needed.
+	for fn in os.listdir(backup_encrypted_dir):
+		fn2 = os.path.join(backup_duplicity_dir, fn.replace(".enc", ""))
+		if os.path.exists(fn2): continue
+		os.unlink(os.path.join(backup_encrypted_dir, fn))
 
-# Remove encrypted backups that are no longer needed.
-for fn in os.listdir(backup_encrypted_dir):
-	fn2 = os.path.join(backup_duplicity_dir, fn.replace(".enc", ""))
-	if os.path.exists(fn2): continue
-	os.unlink(os.path.join(backup_encrypted_dir, fn))
+	# Execute a post-backup script that does the copying to a remote server.
+	# Run as the STORAGE_USER user, not as root. Pass our settings in
+	# environment variables so the script has access to STORAGE_ROOT.
+	post_script = os.path.join(backup_dir, 'after-backup')
+	if os.path.exists(post_script):
+		shell('check_call',
+			['su', env['STORAGE_USER'], '-c', post_script],
+			env=env)
 
-# Execute a post-backup script that does the copying to a remote server.
-# Run as the STORAGE_USER user, not as root. Pass our settings in
-# environment variables so the script has access to STORAGE_ROOT.
-post_script = os.path.join(backup_dir, 'after-backup')
-if os.path.exists(post_script):
-	shell('check_call',
-		['su', env['STORAGE_USER'], '-c', post_script],
-		env=env)
+if __name__ == "__main__":
+	import sys
+	full_backup = "--full" in sys.argv
+	perform_backup(full_backup)

management/buy_certificate.py

@@ -14,7 +14,7 @@ import rtyaml
 
 from utils import load_environment, shell
 from web_update import get_web_domains, get_domain_ssl_files, get_web_root
-from whats_next import check_certificate
+from status_checks import check_certificate
 
 def buy_ssl_certificate(api_key, domain, command, env):
 	if domain != env['PRIMARY_HOSTNAME'] \

management/daemon.py

@@ -71,8 +71,10 @@ def json_response(data):
 def index():
 	# Render the control panel. This route does not require user authentication
 	# so it must be safe!
+	no_admins_exist = (len([user for user in get_mail_users(env, as_json=True) if "admin" in user['privileges']]) == 0)
 	return render_template('index.html',
 		hostname=env['PRIMARY_HOSTNAME'],
+		no_admins_exist=no_admins_exist,
 	)
 
 @app.route('/me')
@@ -102,12 +104,18 @@ def mail_users():
 @app.route('/mail/users/add', methods=['POST'])
 @authorized_personnel_only
 def mail_users_add():
-	return add_mail_user(request.form.get('email', ''), request.form.get('password', ''), request.form.get('privileges', ''), env)
+	try:
+		return add_mail_user(request.form.get('email', ''), request.form.get('password', ''), request.form.get('privileges', ''), env)
+	except ValueError as e:
+		return (str(e), 400)
 
 @app.route('/mail/users/password', methods=['POST'])
 @authorized_personnel_only
 def mail_users_password():
-	return set_mail_password(request.form.get('email', ''), request.form.get('password', ''), env)
+	try:
+		return set_mail_password(request.form.get('email', ''), request.form.get('password', ''), env)
+	except ValueError as e:
+		return (str(e), 400)
 
 @app.route('/mail/users/remove', methods=['POST'])
 @authorized_personnel_only
@@ -172,6 +180,29 @@ def dns_update():
 	except Exception as e:
 		return (str(e), 500)
 
+@app.route('/dns/set/<qname>', methods=['POST'])
+@app.route('/dns/set/<qname>/<rtype>', methods=['POST'])
+@app.route('/dns/set/<qname>/<rtype>/<value>', methods=['POST'])
+@authorized_personnel_only
+def dns_set_record(qname, rtype="A", value=None):
+	from dns_update import do_dns_update, set_custom_dns_record
+	try:
+		# Get the value from the URL, then the POST parameters, or if it is not set then
+		# use the remote IP address of the request --- makes dynamic DNS easy. To clear a
+		# value, '' must be explicitly passed.
+		if value is None:
+			value = request.form.get("value")
+		if value is None:
+			value = request.environ.get("HTTP_X_FORWARDED_FOR") # normally REMOTE_ADDR but we're behind nginx as a reverse proxy
+		if value == '' or value == '__delete__':
+			# request deletion
+			value = None
+		if set_custom_dns_record(qname, rtype, value, env):
+			return do_dns_update(env)
+		return "OK"
+	except ValueError as e:
+		return (str(e), 400)
+
 @app.route('/dns/dump')
 @authorized_personnel_only
 def dns_get_dump():
@@ -191,7 +222,7 @@ def web_update():
 @app.route('/system/status', methods=["POST"])
 @authorized_personnel_only
 def system_status():
-	from whats_next import run_checks
+	from status_checks import run_checks
 	class WebOutput:
 		def __init__(self):
 			self.items = []
@@ -210,14 +241,11 @@ def system_status():
 @app.route('/system/updates')
 @authorized_personnel_only
 def show_updates():
-	utils.shell("check_call", ["/usr/bin/apt-get", "-qq", "update"])
-	simulated_install = utils.shell("check_output", ["/usr/bin/apt-get", "-qq", "-s", "upgrade"])
-	pkgs = []
-	for line in simulated_install.split('\n'):
-		if re.match(r'^Conf .*', line): continue # remove these lines, not informative
-		line = re.sub(r'^Inst (.*) \[(.*)\] \((\S*).*', r'Updated Package Available: \1 (\3)', line) # make these lines prettier
-		pkgs.append(line)
-	return "\n".join(pkgs)
+	from status_checks import list_apt_updates
+	return "".join(
+		"%s (%s)\n"
+		% (p["package"], p["version"])
+		for p in list_apt_updates())
 
 @app.route('/system/update-packages', methods=["POST"])
 @authorized_personnel_only
@@ -227,6 +255,12 @@ def do_updates():
 		"DEBIAN_FRONTEND": "noninteractive"
 	})
 
+@app.route('/system/backup/status')
+@authorized_personnel_only
+def backup_status():
+	from backup import backup_status
+	return json_response(backup_status(env))
+
 # APP
 
 if __name__ == '__main__':

management/dns_update.py

@@ -4,7 +4,8 @@
 # and mail aliases and restarts nsd.
 ########################################################################
 
-import os, os.path, urllib.parse, datetime, re, hashlib
+import os, os.path, urllib.parse, datetime, re, hashlib, base64
+import ipaddress
 import rtyaml
 
 from mailconfig import get_mail_domains
@@ -159,12 +160,16 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 		# Add a DANE TLSA record for SMTP.
 		records.append(("_25._tcp", "TLSA", build_tlsa_record(env), "Recommended when DNSSEC is enabled. Advertises to mail servers connecting to the box that mandatory encryption should be used."))
 
+		# Add a SSHFP records to help SSH key validation. One per available SSH key on this system.
+		for value in build_sshfp_records():
+			records.append((None, "SSHFP", value, "Optional. Provides an out-of-band method for verifying an SSH key before connecting. Use 'VerifyHostKeyDNS yes' (or 'VerifyHostKeyDNS ask') when connecting with ssh."))
+
 	# The MX record says where email for the domain should be delivered: Here!
 	records.append((None,  "MX",  "10 %s." % env["PRIMARY_HOSTNAME"], "Required. Specifies the hostname (and priority) of the machine that handles @%s mail." % domain))
 
 	# SPF record: Permit the box ('mx', see above) to send mail on behalf of
 	# the domain, and no one else.
-	records.append((None,  "TXT", '"v=spf1 mx -all"', "Recommended. Specifies that only the box is permitted to send @%s mail." % domain))
+	records.append((None,  "TXT", 'v=spf1 mx -all', "Recommended. Specifies that only the box is permitted to send @%s mail." % domain))
 
 	# Add DNS records for any subdomains of this domain. We should not have a zone for
 	# both a domain and one of its subdomains.
@@ -192,7 +197,7 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 
 	# Add defaults if not overridden by the user's custom settings.
 	defaults = [
-		(None,  "A",    env["PUBLIC_IP"],       "Optional. Sets the IP address that %s resolves to, e.g. for web hosting. (It is not necessary for receiving mail on this domain.)" % domain),
+		(None,  "A",    env["PUBLIC_IP"],       "Required. May have a different value. Sets the IP address that %s resolves to for web hosting and other services besides mail. The A record must be present but its value does not affect mail delivery." % domain),
 		("www", "A",    env["PUBLIC_IP"],       "Optional. Sets the IP address that www.%s resolves to, e.g. for web hosting." % domain),
 		(None,  "AAAA", env.get('PUBLIC_IPV6'), "Optional. Sets the IPv6 address that %s resolves to, e.g. for web hosting. (It is not necessary for receiving mail on this domain.)" % domain),
 		("www", "AAAA", env.get('PUBLIC_IPV6'), "Optional. Sets the IPv6 address that www.%s resolves to, e.g. for web hosting." % domain),
@@ -208,11 +213,12 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 	if os.path.exists(opendkim_record_file):
 		# Append the DKIM TXT record to the zone as generated by OpenDKIM, after string formatting above.
 		with open(opendkim_record_file) as orf:
-			m = re.match(r"(\S+)\s+IN\s+TXT\s+(\(.*\))\s*;", orf.read(), re.S)
-			records.append((m.group(1), "TXT", m.group(2), "Recommended. Provides a way for recipients to verify that this machine sent @%s mail." % domain))
+			m = re.match(r'(\S+)\s+IN\s+TXT\s+\( "([^"]+)"\s+"([^"]+)"\s*\)', orf.read(), re.S)
+			val = m.group(2) + m.group(3)
+			records.append((m.group(1), "TXT", val, "Recommended. Provides a way for recipients to verify that this machine sent @%s mail." % domain))
 
 		# Append a DMARC record.
-		records.append(("_dmarc", "TXT", '"v=DMARC1; p=quarantine"', "Optional. Specifies that mail that does not originate from the box but claims to be from @%s is suspect and should be quarantined by the recipient's mail system." % domain))
+		records.append(("_dmarc", "TXT", 'v=DMARC1; p=quarantine', "Optional. Specifies that mail that does not originate from the box but claims to be from @%s is suspect and should be quarantined by the recipient's mail system." % domain))
 
 	# Sort the records. The None records *must* go first in the nsd zone file. Otherwise it doesn't matter.
 	records.sort(key = lambda rec : list(reversed(rec[0].split(".")) if rec[0] is not None else ""))
@@ -238,7 +244,7 @@ def get_custom_records(domain, additional_records, env):
 		if isinstance(value, str):
 			values = [("A", value)]
 			if value == "local" and env.get("PUBLIC_IPV6"):
-				values.appnd( ("AAAA", value) )
+				values.append( ("AAAA", value) )
 
 		# A mapping creates multiple records.
 		elif isinstance(value, dict):
@@ -256,11 +262,6 @@ def get_custom_records(domain, additional_records, env):
 			if rtype == "AAAA" and value2 == "local":
 				if "PUBLIC_IPV6" not in env: continue # no IPv6 address is available so don't set anything
 				value2 = env["PUBLIC_IPV6"]
-
-			# For typical zone file output, quote a text record.
-			if rtype == "TXT":
-				value2 = "\"" + value2 + "\""
-
 			yield (qname, rtype, value2)
 
 ########################################################################
@@ -288,26 +289,65 @@ def build_tlsa_record(env):
 	# 1: The certificate is SHA256'd here.
 	return "3 0 1 " + certhash
 
+def build_sshfp_records():
+	# The SSHFP record is a way for us to embed this server's SSH public
+	# key fingerprint into the DNS so that remote hosts have an out-of-band
+	# method to confirm the fingerprint. See RFC 4255 and RFC 6594. This
+	# depends on DNSSEC.
+	#
+	# On the client side, set SSH's VerifyHostKeyDNS option to 'ask' to
+	# include this info in the key verification prompt or 'yes' to trust
+	# the SSHFP record.
+	#
+	# See https://github.com/xelerance/sshfp for inspiriation.
+
+	algorithm_number = {
+		"ssh-rsa": 1,
+		"ssh-dss": 2,
+		"ecdsa-sha2-nistp256": 3,
+	}
+
+	# Get our local fingerprints by running ssh-keyscan. The output looks
+	# like the known_hosts file: hostname, keytype, fingerprint.
+	keys = shell("check_output", ["ssh-keyscan", "localhost"])
+	for key in keys.split("\n"):
+		if key.strip() == "" or key[0] == "#": continue
+		try:
+			host, keytype, pubkey = key.split(" ")
+			yield "%d %d ( %s )" % (
+				algorithm_number[keytype],
+				2, # specifies we are using SHA-256 on next line
+				hashlib.sha256(base64.b64decode(pubkey)).hexdigest().upper(),
+				)
+		except:
+			# Lots of things can go wrong. Don't let it disturb the DNS
+			# zone.
+			pass
+	
 ########################################################################
 
 def write_nsd_zone(domain, zonefile, records, env, force):
-	# We set the administrative email address for every domain to domain_contact@[domain.com].
-	# You should probably create an alias to your email address.
-
 	# On the $ORIGIN line, there's typically a ';' comment at the end explaining
 	# what the $ORIGIN line does. Any further data after the domain confuses
 	# ldns-signzone, however. It used to say '; default zone domain'.
 
+	# The SOA contact address for all of the domains on this system is hostmaster
+	# @ the PRIMARY_HOSTNAME. Hopefully that's legit.
+
+	# For the refresh through TTL fields, a good reference is:
+	# http://www.peerwisdom.org/2013/05/15/dns-understanding-the-soa-record/
+
+
 	zone = """
 $ORIGIN {domain}.
-$TTL 86400           ; default time to live
+$TTL 1800           ; default time to live
 
 @ IN SOA ns1.{primary_domain}. hostmaster.{primary_domain}. (
            __SERIAL__     ; serial number
-           28800       ; Refresh
-           7200        ; Retry
-           864000      ; Expire
-           86400       ; Min TTL
+           7200     ; Refresh (secondary nameserver update interval)
+           1800     ; Retry (when refresh fails, how often to try again)
+           1209600  ; Expire (when refresh fails, how long secondary nameserver will keep records around anyway)
+           1800     ; Negative TTL (how long negative responses are cached)
            )
 """
 
@@ -319,6 +359,10 @@ $TTL 86400           ; default time to live
 		if subdomain:
 			zone += subdomain
 		zone += "\tIN\t" + querytype + "\t"
+		if querytype == "TXT":
+			value = value.replace('\\', '\\\\') # escape backslashes
+			value = value.replace('"', '\\"') # escape quotes
+			value = '"' + value + '"' # wrap in quotes
 		zone += value + "\n"
 
 	# DNSSEC requires re-signing a zone periodically. That requires
@@ -496,7 +540,7 @@ def sign_zone(domain, zonefile, env):
 	# Remove our temporary file.
 	for fn in files_to_kill:
 		os.unlink(fn)
-	
+
 ########################################################################
 
 def write_opendkim_tables(zonefiles, env):
@@ -551,6 +595,89 @@ def write_opendkim_tables(zonefiles, env):
 
 ########################################################################
 
+def set_custom_dns_record(qname, rtype, value, env):
+	# validate qname
+	for zone, fn in get_dns_zones(env):
+		# It must match a zone apex or be a subdomain of a zone
+		# that we are otherwise hosting.
+		if qname == zone or qname.endswith("."+zone):
+			break
+	else:
+		# No match.
+		raise ValueError("%s is not a domain name or a subdomain of a domain name managed by this box." % qname)
+
+	# validate rtype
+	rtype = rtype.upper()
+	if value is not None:
+		if rtype in ("A", "AAAA"):
+			v = ipaddress.ip_address(value)
+			if rtype == "A" and not isinstance(v, ipaddress.IPv4Address): raise ValueError("That's an IPv6 address.")
+			if rtype == "AAAA" and not isinstance(v, ipaddress.IPv6Address): raise ValueError("That's an IPv4 address.")
+		elif rtype in ("CNAME", "TXT"):
+			# anything goes
+			pass
+		else:
+			raise ValueError("Unknown record type '%s'." % rtype)
+
+	# load existing config
+	config = get_custom_dns_config(env)
+
+	# update
+	if qname not in config:
+		if value is None:
+			# Is asking to delete a record that does not exist.
+			return False
+		elif rtype == "A":
+			# Add this record using the short form 'qname: value'.
+			config[qname] = value
+		else:
+			# Add this record. This is the qname's first record.
+			config[qname] = { rtype: value }
+	else:
+		if isinstance(config[qname], str):
+			# This is a short-form 'qname: value' implicit-A record.
+			if value is None and rtype != "A":
+				# Is asking to delete a record that doesn't exist.
+				return False
+			elif value is None and rtype == "A":
+				# Delete record.
+				del config[qname]
+			elif rtype == "A":
+				# Update, keeping short form.
+				if config[qname] == "value":
+					# No change.
+					return False
+				config[qname] = value
+			else:
+				# Expand short form so we can add a new record type.
+				config[qname] = { "A": config[qname], rtype: value }
+		else:
+			# This is the qname: { ... } (dict) format.
+			if value is None:
+				if rtype not in config[qname]:
+					# Is asking to delete a record that doesn't exist.
+					return False
+				else:
+					# Delete the record. If it's the last record, delete the domain.
+					del config[qname][rtype]
+					if len(config[qname]) == 0:
+						del config[qname]
+			else:
+				# Update the record.
+				if config[qname].get(rtype) == "value":
+					# No change.
+					return False
+				config[qname][rtype] = value
+
+	# serialize & save
+	config_yaml = rtyaml.dump(config)
+	with open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml'), "w") as f:
+		f.write(config_yaml)
+
+	return True
+
+########################################################################
+
 def justtestingdotemail(domain, records):
 	# If the domain is a subdomain of justtesting.email, which we own,
 	# automatically populate the zone where it is set up on dns4e.com.
@@ -596,8 +723,9 @@ def build_recommended_dns(env):
 	ret = []
 	domains = get_dns_domains(env)
 	zonefiles = get_dns_zones(env)
+	additional_records = get_custom_dns_config(env)
 	for domain, zonefile in zonefiles:
-		records = build_zone(domain, domains, {}, env)
+		records = build_zone(domain, domains, additional_records, env)
 
 		# remove records that we don't dislay
 		records = [r for r in records if r[3] is not False]

management/mailconfig.py

@@ -139,13 +139,7 @@ def add_mail_user(email, pw, privs, env):
 	if not validate_email(email, mode='user'):
 		return ("Invalid email address.", 400)
 
-	# validate password
-	if pw.strip() == "":
-		return ("No password provided.", 400)
-	if re.search(r"[\s]", pw):
-		return ("Passwords cannot contain spaces.", 400)
-	if len(pw) < 4:
-		return ("Passwords must be at least four characters.", 400)
+	validate_password(pw)
 
 	# validate privileges
 	if privs is None or privs.strip() == "":
@@ -193,6 +187,8 @@ def add_mail_user(email, pw, privs, env):
 	return kick(env, "mail user added")
 
 def set_mail_password(email, pw, env):
+	validate_password(pw)
+	
 	# hash the password
 	pw = utils.shell('check_output', ["/usr/bin/doveadm", "pw", "-s", "SHA512-CRYPT", "-p", pw]).strip()
 
@@ -312,7 +308,7 @@ def get_required_aliases(env):
 	# These are the aliases that must exist.
 	aliases = set()
 
-	# The hostmaster aliase is exposed in the DNS SOA for each zone.
+	# The hostmaster alias is exposed in the DNS SOA for each zone.
 	aliases.add("hostmaster@" + env['PRIMARY_HOSTNAME'])
 
 	# Get a list of domains we serve mail for, except ones for which the only
@@ -344,10 +340,15 @@ def kick(env, mail_result=None):
 
 	# Ensure every required alias exists.
 
+	existing_users = get_mail_users(env)
 	existing_aliases = get_mail_aliases(env)
 	required_aliases = get_required_aliases(env)
 
 	def ensure_admin_alias_exists(source):
+		# If a user account exists with that address, we're good.
+		if source in existing_users:
+			return
+
 		# Does this alias exists?
 		for s, t in existing_aliases:
 			if s == source:
@@ -381,6 +382,16 @@ def kick(env, mail_result=None):
 
 	return "".join(s for s in results if s != "")
 
+def validate_password(pw):
+	# validate password
+	if pw.strip() == "":
+		raise ValueError("No password provided.")
+	if re.search(r"[\s]", pw):
+		raise ValueError("Passwords cannot contain spaces.")
+	if len(pw) < 4:
+		raise ValueError("Passwords must be at least four characters.")
+
+
 if __name__ == "__main__":
 	import sys
 	if len(sys.argv) > 2 and sys.argv[1] == "validate-email":

management/status_checks.py

@@ -6,9 +6,10 @@
 
 __ALL__ = ['check_certificate']
 
-import os, os.path, re, subprocess
+import os, os.path, re, subprocess, datetime
 
 import dns.reversename, dns.resolver
+import dateutil.parser, dateutil.tz
 
 from dns_update import get_dns_zones, build_tlsa_record
 from web_update import get_web_domains, get_domain_ssl_files
@@ -36,6 +37,17 @@ def run_system_checks(env):
 	else:
 		env['out'].print_ok("SSH disallows password-based login.")
 
+	# Check for any software package updates.
+	pkgs = list_apt_updates(apt_update=False)
+	if os.path.exists("/var/run/reboot-required"):
+		env['out'].print_error("System updates have been installed and a reboot of the machine is required.")
+	elif len(pkgs) == 0:
+		env['out'].print_ok("System software is up to date.")
+	else:
+		env['out'].print_error("There are %d software packages that can be updated." % len(pkgs))
+		for p in pkgs:
+			env['out'].print_line("%s (%s)" % (p["package"], p["version"]))
+
 	# Check that the administrator alias exists since that's where all
 	# admin email is automatically directed.
 	check_alias_exists("administrator@" + env['PRIMARY_HOSTNAME'], env)
@@ -57,17 +69,18 @@ def run_network_checks(env):
 			machines from being able to send spam. A quick connection test to Google's mail server on port 25
 			failed.""")
 
-	# Stop if the IPv4 address is listed in the ZEN Spamhouse Block List.
+	# Stop if the IPv4 address is listed in the ZEN Spamhaus Block List.
 	# The user might have ended up on an IP address that was previously in use
 	# by a spammer, or the user may be deploying on a residential network. We
 	# will not be able to reliably send mail in these cases.
 	rev_ip4 = ".".join(reversed(env['PUBLIC_IP'].split('.')))
-	if not query_dns(rev_ip4+'.zen.spamhaus.org', 'A', nxdomain=None):
+	zen = query_dns(rev_ip4+'.zen.spamhaus.org', 'A', nxdomain=None)
+	if zen is None:
 		env['out'].print_ok("IP address is not blacklisted by zen.spamhaus.org.")
 	else:
-		env['out'].print_error("""The IP address of this machine %s is listed in the Spamhaus Block List,
+		env['out'].print_error("""The IP address of this machine %s is listed in the Spamhaus Block List (code %s),
 			which may prevent recipients from receiving your email. See http://www.spamhaus.org/query/ip/%s."""
-			% (env['PUBLIC_IP'], env['PUBLIC_IP']))
+			% (env['PUBLIC_IP'], zen, env['PUBLIC_IP']))
 
 def run_domain_checks(env):
 	# Get the list of domains we handle mail for.
@@ -197,7 +210,9 @@ def check_dns_zone(domain, env, dns_zonefiles):
 		env['out'].print_line("Key Tag: " + ds_keytag + ("" if not ds_looks_valid or ds[0] == ds_keytag else " (Got '%s')" % ds[0]))
 		env['out'].print_line("Key Flags: KSK")
 		env['out'].print_line("Algorithm: 7 / RSASHA1-NSEC3-SHA1" + ("" if not ds_looks_valid or ds[1] == '7' else " (Got '%s')" % ds[1]))
+			# see http://www.iana.org/assignments/dns-sec-alg-numbers/dns-sec-alg-numbers.xhtml
 		env['out'].print_line("Digest Type: 2 / SHA-256")
+			# http://www.ietf.org/assignments/ds-rr-types/ds-rr-types.xml
 		env['out'].print_line("Digest: " + digests['2'])
 		if ds_looks_valid and ds[3] != digests.get(ds[2]):
 			env['out'].print_line("(Got digest type %s and digest %s which do not match.)" % (ds[2], ds[3]))
@@ -248,11 +263,13 @@ def check_mail_domain(domain, env):
 	# Stop if the domain is listed in the Spamhaus Domain Block List.
 	# The user might have chosen a domain that was previously in use by a spammer
 	# and will not be able to reliably send mail.
-	if not query_dns(domain+'.dbl.spamhaus.org', "A", nxdomain=None):
+	dbl = query_dns(domain+'.dbl.spamhaus.org', "A", nxdomain=None)
+	if dbl is None:
 		env['out'].print_ok("Domain is not blacklisted by dbl.spamhaus.org.")
 	else:
-		env['out'].print_error("""This domain is listed in the Spamhaus Domain Block List, which may prevent recipients from receiving your mail.
-			See http://www.spamhaus.org/dbl/ and http://www.spamhaus.org/query/domain/%s.""" % domain)
+		env['out'].print_error("""This domain is listed in the Spamhaus Domain Block List (code %s),
+			which may prevent recipients from receiving your mail.
+			See http://www.spamhaus.org/dbl/ and http://www.spamhaus.org/query/domain/%s.""" % (dbl, domain))
 
 def check_web_domain(domain, env):
 	# See if the domain's A record resolves to our PUBLIC_IP. This is already checked
@@ -349,7 +366,8 @@ def check_certificate(domain, ssl_certificate, ssl_private_key):
 
 	# First check that the certificate is for the right domain. The domain
 	# must be found in the Subject Common Name (CN) or be one of the
-	# Subject Alternative Names.
+	# Subject Alternative Names. A wildcard might also appear as the CN
+	# or in the SAN list, so check for that tool.
 	cert_dump = shell('check_output', [
 		"openssl", "x509",
 		"-in", ssl_certificate,
@@ -357,6 +375,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key):
 		])
 	cert_dump = cert_dump.split("\n")
 	certificate_names = set()
+	cert_expiration_date = None
 	while len(cert_dump) > 0:
 		line = cert_dump.pop(0)
 
@@ -378,7 +397,12 @@ def check_certificate(domain, ssl_certificate, ssl_private_key):
 				if m:
 					certificate_names.add(m.group(1))
 
-	if domain is not None and domain not in certificate_names:
+		m = re.match("            Not After : (.*)", line)
+		if m:
+			cert_expiration_date = dateutil.parser.parse(m.group(1))
+
+	wildcard_domain = re.sub("^[^\.]+", "*", domain)
+	if domain is not None and domain not in certificate_names and wildcard_domain not in certificate_names:
 		return "This certificate is for the wrong domain names. It is for %s." % \
 			", ".join(sorted(certificate_names))
 
@@ -427,11 +451,56 @@ def check_certificate(domain, ssl_certificate, ssl_private_key):
 	if "self signed" in verifyoutput:
 		# Certificate is self-signed.
 		return "SELF-SIGNED"
-	elif retcode == 0:
-		# Certificate is OK.
-		return "OK"
-	else:
+	elif retcode != 0:
+		# There is some unknown problem. Return the `openssl verify` raw output.
 		return verifyoutput.strip()
+	else:
+		# `openssl verify` returned a zero exit status so the cert is currently
+		# good.
+
+		# But is it expiring soon?
+		now = datetime.datetime.now(dateutil.tz.tzlocal())
+		ndays = (cert_expiration_date-now).days
+		if ndays <= 31:
+			return "This certificate expires in %d days on %s." % (ndays, cert_expiration_date.strftime("%x"))
+
+		# Return the special OK code.
+		return "OK"
+
+_apt_updates = None
+def list_apt_updates(apt_update=True):
+	# See if we have this information cached recently.
+	# Keep the information for 8 hours.
+	global _apt_updates
+	if _apt_updates is not None and _apt_updates[0] > datetime.datetime.now() - datetime.timedelta(hours=8):
+		return _apt_updates[1]
+
+	# Run apt-get update to refresh package list. This should be running daily
+	# anyway, so on the status checks page don't do this because it is slow.
+	if apt_update:
+		shell("check_call", ["/usr/bin/apt-get", "-qq", "update"])
+
+	# Run apt-get upgrade in simulate mode to get a list of what
+	# it would do.
+	simulated_install = shell("check_output", ["/usr/bin/apt-get", "-qq", "-s", "upgrade"])
+	pkgs = []
+	for line in simulated_install.split('\n'):
+		if line.strip() == "":
+			continue
+		if re.match(r'^Conf .*', line):
+			 # remove these lines, not informative
+			continue
+		m = re.match(r'^Inst (.*) \[(.*)\] \((\S*)', line)
+		if m:
+			pkgs.append({ "package": m.group(1), "version": m.group(3), "current_version": m.group(2) })
+		else:
+			pkgs.append({ "package": "[" + line + "]", "version": "", "current_version": "" })
+
+	# Cache for future requests.
+	_apt_updates = (datetime.datetime.now(), pkgs)
+
+	return pkgs
+
 
 try:
 	terminal_columns = int(shell('check_output', ['stty', 'size']).split()[1])

management/templates/index.html

@@ -30,7 +30,7 @@
               margin-bottom: 1.25em;
             }
 
-            h1, h2, h3 {
+            h1, h2, h3, h4 {
               font-family: Raleway, sans-serif;
               font-weight: bold;
             }
@@ -44,7 +44,13 @@
               border-bottom: 1px solid black;
               padding-bottom: 3px;
               margin-bottom: 13px;
-              margin-top: 26px;
+              margin-top: 30px;
+            }
+
+            h4 {
+              font-size: 110%;
+              margin-bottom: 13px;
+              margin-top: 18px;
             }
 
             .panel {
@@ -79,21 +85,20 @@
               <a href="#" class="dropdown-toggle" data-toggle="dropdown">System <b class="caret"></b></a>
               <ul class="dropdown-menu">
                 <li><a href="#system_status" onclick="return show_panel(this);">Status Checks</a></li>
-                <li><a href="#system_external_dns" onclick="return show_panel(this);">External DNS (Advanced)</a></li>
+                <li><a href="#system_backup" onclick="return show_panel(this);">Backup Status</a></li>
+                <li class="divider"></li>
+                <li class="dropdown-header">Super Advanced Options</li>
+                <li><a href="#system_dns" onclick="return show_panel(this);">DNS (Custom/External)</a></li>
               </ul>
             </li>
             <li class="dropdown active">
-              <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mail <b class="caret"></b></a>
+              <a href="#" class="dropdown-toggle" data-toggle="dropdown">Mail/Sync <b class="caret"></b></a>
               <ul class="dropdown-menu">
-                <li><a href="#mail-guide" onclick="return show_panel(this);">Instructions</a></li>
+                <li><a href="#mail-guide" onclick="return show_panel(this);">Mail Instructions</a></li>
+                <li><a href="#sync_guide" onclick="return show_panel(this);">Contacts/Calendar</a></li>
+                <li class="divider"></li>
                 <li><a href="#users" onclick="return show_panel(this);">Users</a></li>
                 <li><a href="#aliases" onclick="return show_panel(this);">Aliases</a></li>
-                <!--<li><a href="#">Another action</a></li>
-                <li><a href="#">Something else here</a></li>
-                <li class="divider"></li>
-                <li class="dropdown-header">Nav header</li>
-                <li><a href="#">Separated link</a></li>
-                <li><a href="#">One more separated link</a></li>-->
               </ul>
             </li>
           </ul>
@@ -109,8 +114,12 @@
       {% include "system-status.html" %}
       </div>
 
-      <div id="panel_system_external_dns" class="container panel">
-      {% include "system-external-dns.html" %}
+      <div id="panel_system_backup" class="container panel">
+      {% include "system-backup.html" %}
+      </div>
+
+      <div id="panel_system_dns" class="container panel">
+      {% include "system-dns.html" %}
       </div>
 
       <div id="panel_login" class="panel">
@@ -129,6 +138,10 @@
       {% include "aliases.html" %}
       </div>
 
+      <div id="panel_sync_guide" class="container panel">
+      {% include "sync-guide.html" %}
+      </div>
+
       <hr>
 
       <footer>
@@ -136,7 +149,7 @@
       </footer>
     </div> <!-- /container -->
 
-        <div id="ajax_loading_indicator" style="display: none; position: absolute; left: 0; top: 0; width: 100%; height: 100%; text-align: center; background-color: rgba(255,255,255,.75)">
+        <div id="ajax_loading_indicator" style="display: none; position: fixed; left: 0; top: 0; width: 100%; height: 100%; text-align: center; background-color: rgba(255,255,255,.75)">
           <div style="margin: 20% auto">
             <div><span class="glyphicon glyphicon-time"></span></div>
             <div>Loading...</div>
@@ -169,6 +182,11 @@ var global_modal_state = null;
 var global_modal_funcs = null;
 
 $(function() {
+  $('#global_modal').on('shown.bs.modal', function (e) {
+    // set focus to first input in the global modal's body
+    var input = $('#global_modal .modal-body input');
+    if (input.length > 0) $(input[0]).focus();  
+  })  
   $('#global_modal .btn-danger').click(function() {
     // Don't take action now. Wait for the modal to be totally hidden
     // so that we don't attempt to show another modal while this one

management/templates/login.html

@@ -1,7 +1,20 @@
+<h1 style="margin: 1em; text-align: center">{{hostname}}</h1>
+
+{% if no_admins_exist %}
+<div class="container">
+<div class="col-md-offset-2 col-md-8">
+  <p class="text-danger">There are no administrative users on this system! To make an administrative user,
+  log into this machine using SSH (like when you first set it up) and run:</p>
+  <pre>cd mailinabox
+sudo tools/mail.py user make-admin your@emailaddress.com</pre>
+  <hr>
+</div>
+</div>  
+{% endif %}
+
 <div class="row">
   <div class="col-sm-offset-2 col-sm-8 col-md-offset-3 col-md-6 col-lg-offset-4 col-lg-4">
     <center>
-      <h1 style="margin: 1em">{{hostname}}</h1>
       <p style="margin: 2em">Log in here for your Mail-in-a-Box control panel.</p>
     </center>
 
@@ -36,6 +49,7 @@
   </div>
 </div>
 
+
 <script>
 function do_login() {
   if ($('#loginEmail').val() == "") {

management/templates/mail-guide.html

@@ -1,41 +1,62 @@
+<style>#panel_mail-guide table.table { width: auto; margin-left: 1.5em; }</style>
+
 <div class="container">
 	<h2>Checking and Sending Mail</h2>
 
-	<h4>App Configuration</h4>
+	<p>You can use either webmail, IMAP/SMTP, or Exchange/ActiveSync to access your mail.</p>
 
-	<p>You can access your email using webmail, desktop mail clients, or mobile apps.</p>
+	<hr>
 
-	<p>Here is what you need to know for webmail:</p>
+	<div class="row">
+		<div class="col-md-6">
+			<h4>How to log in</h4>
 
-	<style>#panel_mail-guide table.table { width: auto; margin-left: 1.5em; }</style>
+			<p>Your username and password are the same no matter how you check your mail:</p>
 
-	<table class="table">
-	<tr><th>Webmail Address:</th> <td><a href="https://{{hostname}}/mail"><b>https://{{hostname}}/mail</b></a></td></tr>
-	<tr><th>Username:</th> <td>Your whole email address.</td></tr>
-	<tr><th>Password:</th> <td>Your mail password.</td></tr>
-	</table>
+			<table class="table" style="max-width: 30em">
+			<tr><th>Username:</th> <td>Your whole email address.</td></tr>
+			<tr><th>Password:</th> <td>Your mail password.</td></tr>
+			</table>
+		</div>
+	</div>
 
-	<p>On mobile devices you might need to install a &ldquo;mail client&rdquo; app. We recommend <a href="https://play.google.com/store/apps/details?id=com.fsck.k9">K-9 Mail</a>. On a desktop you could try <a href="https://www.mozilla.org/en-US/thunderbird/">Mozilla Thunderbird</a>.</p>
+	<div class="row">
+		<div class="col-sm-6">
+			<h4>Webmail</h4>
 
-	<p>Configure your device or desktop mail client as follows:</p>
+			<p>Your webmail site is <a href="https://{{hostname}}/mail">https://{{hostname}}/mail</a>.</p>
+		</div>
+	</div>
 
-	<table class="table" style="max-width: 30em">
-	<tr><th>Server Name:</th> <td>{{hostname}}</td></tr>
-	<tr><th>Username:</th> <td>Your whole email address.</td></tr>
-	<tr><th>Password:</th> <td>Your mail password.</td></tr>
-	</table>
+	<div class="row">
+		<div class="col-sm-6">
+			<h4>IMAP/SMTP settings</h4>
 
+			<p>This method is preferred on Android devices and is not available on iOS devices.</p>
 
-	<table class="table">
-	<thead><tr><th>Protocol</th> <th>Port</th> <th>Options</th></tr></thead>
-	<tr><th>IMAP</th> <td>993</td> <td>SSL</td></tr>
-	<tr><th>SMTP</th> <td>587</td> <td>STARTTLS</td></tr>
-	<tr><th>Exchange ActiveSync</th> <td>n/a</td> <td>Secure Connection</td></tr>
-	</table>
+			<table class="table">
+			<thead>
+				<tr><th style="padding-bottom: 1em; border-bottom: none">Server:</th> <td colspan="2">{{hostname}}</td></tr>
+				<tr><th>Protocol</th> <th>Port</th> <th>Options</th></tr>
+			</thead>
+			<tr><th>IMAP</th> <td>993</td> <td>SSL</td></tr>
+			<tr><th>SMTP</th> <td>587</td> <td>STARTTLS</td></tr>
+			</table>
 
-	<p>Depending on your mail program, you will use either IMAP &amp; SMTP or Exchange ActiveSync. See this <a href="http://z-push.org/compatibility/">list of compatible devices</a> for Exchange ActiveSync.</p>
+			<p>In addition to setting up your email, you&rsquo;ll also need to set up <a href="#sync_guide" onclick="return show_panel(this);">contacts and calendar synchronization</a>.</p>
+		</div>
 
-	<h4>Notes</h4>
+		<div class="col-sm-6">
+			<h4>Exchange/ActiveSync settings</h4>
 
-	<p>Mail-in-a-Box uses <a href="http://en.wikipedia.org/wiki/Greylisting">greylisting</a> to cut down on spam. The first time you receive an email from a recipient, it may be delayed for ten minutes.</p>
+			<p>On iOS devices and devices on this <a href="http://z-push.org/compatibility/">compatibility list</a>, set up your mail as an Exchange or ActiveSync server:</p>
+
+			<table class="table">
+			<tr><th>Server</th> <td>{{hostname}}</td></tr>
+			<tr><th>Options</th> <td>Secure Connection</td></tr>
+			</table>
+
+			<p>Your device should also provide a contacts list and calendar that syncs to this box when you use this method.</p>
+		</div>
+	</div>
 </div>

management/templates/sync-guide.html

@@ -0,0 +1,48 @@
+<div class="container">
+	<h2>Contacts &amp; Calendar Synchronization</h2>
+
+	<p>This box can hold your contacts and calendar, just like it holds your email.</p>
+
+	<hr>
+
+	<div class="row">
+		<div class="col-sm-6">
+			<h4>In your browser</h4>
+
+			<p>You can edit your contacts and calendar from your web browser.</p>
+
+			<table class="table">
+			<thead><tr><th>For...</th> <th>Visit this URL</th></tr></thead>
+			<tr><td>Contacts</td> <td><a href="https://{{hostname}}/cloud/contacts">https://{{hostname}}/cloud/contacts</a></td></tr>
+			<tr><td>Calendar</td> <td><a href="https://{{hostname}}/cloud/calendar">https://{{hostname}}/cloud/calendar</a></td></tr>
+			</table>
+
+			<p>Log in settings are the same as with <a href="#mail-guide" onclick="return show_panel(this);">mail</a>: your
+			complete email address and your mail password.</p>
+		</div>
+
+		<div class="col-sm-6">
+			<h4>On your mobile device</h4>
+
+			<p>If you set up your <a href="#mail-guide" onclick="return show_panel(this);">mail</a> using Exchange/ActiveSync,
+			your contacts and calendar may already appear on your device.</p>
+			<p>Otherwise, here are some apps that can synchronize your contacts and calendar to your Android phone.</p>
+
+			<table class="table">
+			<thead><tr><th>For...</th> <th>Use...</th></tr></thead>
+			<tr><td>Contacts</td> <td><a href="https://play.google.com/store/apps/details?id=org.dmfs.carddav.sync">CardDAV-Sync free beta</a> (free)</td></tr>
+			<tr><td>Calendar</td> <td><a href="https://play.google.com/store/apps/details?id=org.dmfs.caldav.lib">CalDAV-Sync</a> ($2.89)</td></tr>
+			</table>
+
+			<p>Use the following settings:</p>
+
+			<table class="table">
+			<tr><td>Account Type</td> <td>CardDAV or CalDAV</td></tr>
+			<tr><td>Server Name</td> <td>{{hostname}}</td></tr>
+			<tr><td>Use SSL</td> <td>Yes</td></tr>
+			<tr><td>Username</td> <td>Your complete email address.</td></tr>
+			<tr><td>Password</td> <td>Your mail password.</td></tr>
+			</table>
+		</div>
+	</div>
+</div>

management/templates/system-backup.html

@@ -0,0 +1,78 @@
+<style>
+#backup-status th { text-align: center; }
+#backup-status tr.full-backup td { font-weight: bold; }
+</style>
+
+<h2>Backup Status</h2>
+
+<h3>Copying Backup Files</h3>
+
+<p>The box makes an incremental backup each night. The backup is stored on the machine itself. You are responsible for copying the backup files off of the machine.</p>
+
+<p>Many cloud providers make this easy by allowing you to take snapshots of the machine's disk.</p>
+
+<p>You can also use SFTP (FTP over SSH) to copy files from <tt id="backup-location"></tt>. These files are encrpyted, so they are safe to store anywhere. Copy the encryption password from <tt id="backup-encpassword-file"></tt> also but keep it in a safe location.</p>
+
+<h3>Current Backups</h3>
+
+<p>The backup directory currently contains the backups listed below. The total size on disk of the backups is <span id="backup-total-size"></span>.</p>
+
+<table id="backup-status" class="table" style="width: auto">
+  <thead>
+    <th colspan="2">When</th>
+    <th>Type</th>
+    <th>Size</th>
+    <th>Deleted in...</th>
+  </thead>
+  <tbody>
+  </tbody>
+</table>
+
+<p><small>The size column in the table indicates the size of the encrpyted backup, but the total size on disk shown above includes storage for unencrpyted intermediate files.</small></p>
+
+<script>
+function nice_size(bytes) {
+  var powers = ['bytes', 'KB', 'MB', 'GB', 'TB'];
+  while (true) {
+    if (powers.length == 1) break;
+    if (bytes < 1000) break;
+    bytes /= 1024;
+    powers.shift();
+  }
+  return (Math.round(bytes*10)/10) + " " + powers[0];
+}
+
+function show_system_backup() {
+  $('#backup-status tbody').html("<tr><td colspan='2' class='text-muted'>Loading...</td></tr>")
+  api(
+    "/system/backup/status",
+    "GET",
+    { },
+    function(r) {
+      $('#backup-location').text(r.encdirectory);
+      $('#backup-encpassword-file').text(r.encpwfile);
+
+      $('#backup-status tbody').html("");
+      var total_disk_size = 0;
+      for (var i = 0; i < r.backups.length; i++) {
+        var b = r.backups[i];
+        var tr = $('<tr/>');
+        if (b.full) tr.addClass("full-backup");
+        tr.append( $('<td/>').text(b.date_str + " " + r.tz) );
+        tr.append( $('<td/>').text(b.date_delta + " ago") );
+        tr.append( $('<td/>').text(b.full ? "full" : "increment") );
+        tr.append( $('<td style="text-align: right"/>').text( nice_size(b.encsize)) );
+        if (b.deleted_in)
+          tr.append( $('<td/>').text(b.deleted_in) );
+        else
+          tr.append( $('<td class="text-muted">n/a</td>') );
+        $('#backup-status tbody').append(tr);
+
+        total_disk_size += b.size;
+        total_disk_size += b.encsize;
+      }
+
+      $('#backup-total-size').text(nice_size(total_disk_size));
+    })
+}
+</script>

management/templates/system-dns.html

@@ -0,0 +1,118 @@
+<style>
+#external_dns_settings .heading td {
+  font-weight: bold;
+  font-size: 120%;
+  padding-top: 1.5em;
+}
+#external_dns_settings .heading.first td {
+  border-top: none;
+  padding-top: 0;
+}
+#external_dns_settings .values td {
+  border: 0;
+	padding-top: .75em;
+	padding-bottom: 0;
+	max-width: 50vw;
+	word-wrap: break-word;
+}
+#external_dns_settings .explanation td {
+	border: 0;
+	padding-top: .5em;
+	padding-bottom: .75em;
+	font-style: italic;
+  font-size: 95%;
+	color: #777;
+}
+</style>
+
+<h2>DNS (Advanced)</h2>
+
+<p class="text-danger">Custom and external DNS are for advanced configurations.</p>
+
+<h3>Custom DNS</h3>
+
+<p>It is possible to set custom DNS records on domains hosted here. For instance, you can create your own dynamic DNS service. To do so, you will need to call your box&rsquo;s DNS API.</p>
+
+<h4>The HTTP POST request</h4>
+
+<p>Send a POST request like this:</p>
+
+<pre>curl -d "" --user {email}:{password} https://{{hostname}}/admin/dns/set/<b>qname</b>[/<b>rtype</b>[/<b>value</b>]]</pre>
+
+<table class="table">
+<thead><th>Parameter</th> <th>Value</th></thead>
+<tr><td>email</td> <td>The email address of any administrative user here.</td></tr>
+<tr><td>password</td> <td>That user&rsquo;s password.</td></tr>
+<tr><td>qname</td> <td>The fully qualified domain name for the record you are trying to set.</td></tr>
+<tr><td>rtype</td> <td>The resource type. <code>A</code> if omitted. Possible values: <code>A</code> (an IPv4 address), <code>AAAA</code> (an IPv6 address), <code>TXT</code> (a text string), or <code>CNAME</code> (an alias, which is a fully qualified domain name).</td></tr>
+<tr><td>value</td> <td>The new record&rsquo;s value. If omitted, the IPv4 address of the remote host is used. This is handy for dynamic DNS! To delete a record, use &ldquo;__delete__&rdquo;.</td></tr>
+</table>
+
+<p style="margin-top: 1em">Note that <code>-d ""</code> is merely to ensure curl sends a POST request. You do not need to put anything inside the quotes. You can also pass the value using typical form encoding in the POST body.</p>
+
+<h4>Examples:</h4>
+
+<pre># sets laptop.mydomain.com to point to the IP address of the machine you are executing curl on
+curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/laptop.mydomain.com
+
+# sets an alias
+curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/foo.mydomain.com/cname/bar.mydomain.com
+
+# clears the alias
+curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/bar.mydomain.com/cname/__delete__
+
+# sets a TXT record using the alternate value syntax
+curl -d "value=something%20here" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/foo.mydomain.com/txt
+</pre>
+
+
+<h3>External DNS</h3>
+
+<p>Although your box is configured to serve its own DNS, it is possible to host your DNS elsewhere.</p>
+
+<p>If you do so, you are responsible for keeping your DNS entries up to date! If you previously enabled DNSSEC on your domain name by setting a DS record at your registrar, you will likely have to turn it off before changing nameservers.</p>
+
+<p>Enter the following DNS entries at your DNS provider:</p>
+
+<table id="external_dns_settings" class="table">
+	<thead>
+		<tr>
+			<th>QName</th>
+			<th>Type</th>
+			<th>Value</th>
+		</tr>
+	</thead>
+	<tbody>
+	</tbody>
+</table>
+
+<script>
+function show_system_dns() {
+  $('#external_dns_settings tbody').html("<tr><td colspan='2' class='text-muted'>Loading...</td></tr>")
+  api(
+    "/dns/dump",
+    "GET",
+    { },
+    function(zones) {
+      $('#external_dns_settings tbody').html("");
+      for (var j = 0; j < zones.length; j++) {
+        var h = $("<tr class='heading'><td colspan='3'></td></tr>");
+        h.find("td").text(zones[j][0]);
+        $('#external_dns_settings tbody').append(h);
+
+        var r = zones[j][1];
+        for (var i = 0; i < r.length; i++) {
+          var n = $("<tr class='values'><td class='qname'/><td class='rtype'/><td class='value'/></tr>");
+          n.find('.qname').text(r[i].qname);
+          n.find('.rtype').text(r[i].rtype);
+          n.find('.value').text(r[i].value);
+          $('#external_dns_settings tbody').append(n);
+
+          var n = $("<tr class='explanation'><td colspan='3'/></tr>");
+          n.find('td').text(r[i].explanation);
+          $('#external_dns_settings tbody').append(n);
+        }
+      }
+    })
+}
+</script>

management/templates/system-external-dns.html

@@ -1,81 +0,0 @@
-<style>
-#external_dns_settings .heading td {
-  font-weight: bold;
-  font-size: 120%;
-  padding-top: 1.5em;
-}
-#external_dns_settings .heading.first td {
-  border-top: none;
-  padding-top: 0;
-}
-#external_dns_settings .values td {
-	padding-top: .75em;
-	padding-bottom: 0;
-	max-width: 50vw;
-	word-wrap: break-word;
-}
-#external_dns_settings .explanation td {
-	border: 0;
-	padding-top: .5em;
-	padding-bottom: .75em;
-	font-style: italic;
-	color: #777;
-}
-</style>
-
-<h2>External DNS</h2>
-
-<p class="text-danger">This is for advanced configurations.</p>
-
-<h3>Overview</h3>
-
-<p>Although your box is configured to serve its own DNS, it is possible to host your DNS elsewhere. We do not recommend this.</p>
-
-<p>If you do so, you are responsible for keeping your DNS entries up to date. In particular DNSSEC entries must be re-signed periodically. Do not set a DS record at your registrar or publish DNSSEC entries in your DNS zones if you do not intend to keep them up to date.</p>
-
-<h3>DNS Settings</h3>
-
-<p>Enter the following DNS entries at your DNS provider:</p>
-
-<table id="external_dns_settings" class="table">
-	<thead>
-		<tr>
-			<th>QName</th>
-			<th>Type</th>
-			<th>Value</th>
-		</tr>
-	</thead>
-	<tbody>
-	</tbody>
-</table>
-
-<script>
-function show_system_external_dns() {
-  $('#external_dns_settings tbody').html("<tr><td colspan='2' class='text-muted'>Loading...</td></tr>")
-  api(
-    "/dns/dump",
-    "GET",
-    { },
-    function(zones) {
-      $('#external_dns_settings tbody').html("");
-      for (var j = 0; j < zones.length; j++) {
-        var h = $("<tr class='heading'><td colspan='3'></td></tr>");
-        h.find("td").text(zones[j][0]);
-        $('#external_dns_settings tbody').append(h);
-
-        var r = zones[j][1];
-        for (var i = 0; i < r.length; i++) {
-          var n = $("<tr class='values'><td class='qname'/><td class='rtype'/><td class='value'/></tr>");
-          n.find('.qname').text(r[i].qname);
-          n.find('.rtype').text(r[i].rtype);
-          n.find('.value').text(r[i].value);
-          $('#external_dns_settings tbody').append(n);
-
-          var n = $("<tr class='explanation'><td colspan='3'/></tr>");
-          n.find('td').text(r[i].explanation);
-          $('#external_dns_settings tbody').append(n);
-        }
-      }
-    })
-}
-</script>

management/templates/users.html

@@ -2,13 +2,12 @@
 
 <style>
 #user_table tr.account_inactive td .address { color: #888; text-decoration: line-through; }
-#user_table .aliases { margin: .25em 0 0 1em; font-size: 95%; }
+#user_table .aliases { margin-top: .33em; font-size: 95%; }
 #user_table .aliases div:before { content: "⇖ "; }
 #user_table .aliases div {  }
-#user_table .actions { margin: .25em 0 0 1em; font-size: 95%; }
-#user_table .actions > * { display: none; }
-#user_table .account_active .actions a.archive { display: inline; }
-#user_table .account_inactive .actions .restore { display: inline; }
+#user_table .actions { margin-top: .33em; font-size: 95%; }
+#user_table .account_inactive .if_active { display: none; }
+#user_table .account_active .if_inactive { display: none; }
 </style>
 
 <h3>Add a mail user</h3>
@@ -39,13 +38,6 @@
 
 <h3>Existing mail users</h3>
 <table id="user_table" class="table" style="width: auto">
-  <thead>
-    <tr>
-      <th></th>
-      <th>Email Address<br><small style="font-weight: normal">(Also the user&rsquo;s login username.)</small></th>
-      <th>Privileges</th>
-    </tr>
-  </thead>
   <tbody>
   </tbody>
 </table>
@@ -53,19 +45,32 @@
 <div style="display: none">
   <table>
   <tr id="user-template">
-    <td class='actions'>
-        <a href="#" onclick="users_remove(this); return false;" class='archive' title="Archive Account">
-          <span class="glyphicon glyphicon-trash"></span>
-        </a>
-    </td>
     <td class='email'>
       <div class='address'> </div>
-      <div class='aliases' style='display: none'> </div>
+
       <div class='actions'>
-        <span class='restore'>To restore account, create a new account with this email address.</span>
+        <span class='privs'>
+        </span>
+
+        <span class="if_active">
+          <a href="#" onclick="users_set_password(this); return false;" class='setpw' title="Set Password">
+            set password
+          </a>
+          |
+        </span>
+
+        <span class='add-privs'>
+        </span>
+
+        <a href="#" onclick="users_remove(this); return false;" class='if_active' title="Archive Account">
+          archive account
+        </a>
+
+        <div class='if_inactive' style='color: #888; font-size: 90%'>To restore account, create a new account with this email address.</div>
       </div>
+
+      <div class='aliases' style='display: none'> </div>
     </td>
-    <td class='privs'> </td>
   </tr>
   </table>
 </div>
@@ -87,21 +92,24 @@ function show_users() {
         n.addClass("account_" + r[i].status);
         n.attr('data-email', r[i].email);
         n.find('td.email .address').text(r[i].email)
+        $('#user_table tbody').append(n);
+
+        if (r[i].status == 'inactive') continue;
 
         var add_privs = ["admin"];
 
         for (var j = 0; j < r[i].privileges.length; j++) {
-          var p = $("<div><span class='name'></span> <a href='#' onclick='mod_priv(this, \"remove\"); return false;'><span class=\"glyphicon glyphicon-trash\" style='font-size: 90%'></span></a></div>");
+          var p = $("<span><b><span class='name'></span></b> (<a href='#' onclick='mod_priv(this, \"remove\"); return false;' title='Remove Privilege'>remove privilege</a>) |</span>");
           p.find('span.name').text(r[i].privileges[j]);
-          n.find('td.privs').append(p);
+          n.find('.privs').append(p);
           if (add_privs.indexOf(r[i].privileges[j]) >= 0)
             add_privs.splice(add_privs.indexOf(r[i].privileges[j]), 1);
         }
 
         for (var j = 0; j < add_privs.length; j++) {
-          var p = $("<div><small><a href='#' onclick='mod_priv(this, \"add\"); return false;'><span class=\"glyphicon glyphicon-plus\" style='font-size: 90%'></span> <span class='name'></span>?</a></small></div>");
+          var p = $("<span><a href='#' onclick='mod_priv(this, \"add\"); return false;' title='Add Privilege'>make <span class='name'></span></a> | </span>");
           p.find('span.name').text(add_privs[j]);
-          n.find('td.privs').append(p);
+          n.find('.add-privs').append(p);
         }
 
         if (r[i].aliases && r[i].aliases.length > 0) {
@@ -113,7 +121,6 @@ function show_users() {
               ))
           }
         }
-        $('#user_table tbody').append(n);
       }
     })
 }
@@ -141,11 +148,35 @@ function do_add_user() {
   return false;
 }
 
+function users_set_password(elem) {
+  var email = $(elem).parents('tr').attr('data-email');
+  show_modal_confirm(
+    "Archive User",
+    $("<p>Set a new password for <b>" + email + "</b>?</p> <p><label for='users_set_password_pw' style='display: block; font-weight: normal'>New Password:</label><input type='password' id='users_set_password_pw'></p><p><small>Passwords must be at least four characters and may not contain spaces.</small></p>"),
+    "Set Password",
+    function() {
+      api(
+        "/mail/users/password",
+        "POST",
+        {
+          email: email,
+          password: $('#users_set_password_pw').val()
+        },
+        function(r) {
+          // Responses are multiple lines of pre-formatted text.
+          show_modal_error("Set Password", $("<pre/>").text(r));
+        },
+        function(r) {
+          show_modal_error("Set Password", r);
+        });
+    });
+}
+
 function users_remove(elem) {
   var email = $(elem).parents('tr').attr('data-email');
   show_modal_confirm(
     "Archive User",
-    $("<p>Are you sure you want to archive " + email + "?</p> <p>The user's mailboxes will not be deleted (you can do that later), but the user will no longer be able to log into any services on this machine.</p>"),
+    $("<p>Are you sure you want to archive <b>" + email + "</b>?</p> <p>The user's mailboxes will not be deleted (you can do that later), but the user will no longer be able to log into any services on this machine.</p>"),
     "Archive",
     function() {
       api(
@@ -178,7 +209,7 @@ function mod_priv(elem, add_remove) {
   var add_remove1 = add_remove.charAt(0).toUpperCase() + add_remove.substring(1);
   show_modal_confirm(
     "Modify Privileges",
-    "Are you sure you want to " + add_remove + " the " + priv + " privilege for " + email + "?",
+    "Are you sure you want to " + add_remove + " the " + priv + " privilege for <b>" + email + "</b>?",
     add_remove1,
     function() {
       api(
@@ -193,4 +224,4 @@ function mod_priv(elem, add_remove) {
         });
     });
 }
-</script>
+</script>

management/web_update.py

@@ -94,7 +94,7 @@ def make_domain_config(domain, template, template_for_primaryhost, env):
 	nginx_conf = nginx_conf.replace("$SSL_KEY", ssl_key)
 	nginx_conf = nginx_conf.replace("$SSL_CERTIFICATE", ssl_certificate)
 
-	# Add in any user customizations.
+	# Add in any user customizations in YAML format.
 	nginx_conf_custom_fn = os.path.join(env["STORAGE_ROOT"], "www/custom.yaml")
 	if os.path.exists(nginx_conf_custom_fn):
 		yaml = rtyaml.load(open(nginx_conf_custom_fn))
@@ -103,6 +103,11 @@ def make_domain_config(domain, template, template_for_primaryhost, env):
 			for path, url in yaml.get("proxies", {}).items():
 				nginx_conf += "\tlocation %s {\n\t\tproxy_pass %s;\n\t}\n" % (path, url)
 
+	# Add in any user customizations in the includes/ folder.
+	nginx_conf_custom_include = os.path.join(env["STORAGE_ROOT"], "www", safe_domain_name(domain) + ".conf")
+	if os.path.exists(nginx_conf_custom_include):
+		nginx_conf += "\tinclude %s;\n" % (nginx_conf_custom_include)
+
 	# Ending.
 	nginx_conf += nginx_conf_parts[1]
 
@@ -140,7 +145,7 @@ def get_domain_ssl_files(domain, env):
 		# a Subject Alternative Name matching this domain. Don't do this if
 		# the user has uploaded a different private key for this domain.
 		if not ssl_key_is_alt:
-			from whats_next import check_certificate
+			from status_checks import check_certificate
 			if check_certificate(domain, ssl_certificate_primary, None) == "OK":
 				ssl_certificate = ssl_certificate_primary
 
@@ -177,6 +182,7 @@ def ensure_ssl_certificate_exists(domain, ssl_key, ssl_certificate, csr_path, en
 		"openssl", "req", "-new",
 		"-key", ssl_key,
 		"-out",  csr_path,
+		"-sha256",
 		"-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain)])
 
 	# And then make the certificate.

setup/bootstrap.sh

@@ -2,14 +2,14 @@
 #########################################################
 # This script is intended to be run like this:
 #
-#   wget https://raw.githubusercontent.com/mail-in-a-box/mailinabox/master/setup/bootstrap.sh
-#   sudo bash bootstrap.sh
-#
-# We can't pipe directly to bash because setup/start.sh
-# asks for user input on stdin.
+#   curl https://.../bootstrap.sh | sudo bash
 #
 #########################################################
 
+if [ -z "$TAG" ]; then
+	TAG=v0.03
+fi
+
 # Are we running as root?
 if [[ $EUID -ne 0 ]]; then
 	echo "This script must be run as root. Did you leave out sudo?"
@@ -21,16 +21,19 @@ cd
 
 # Clone the Mail-in-a-Box repository if it doesn't exist.
 if [ ! -d mailinabox ]; then
-	echo Downloading Mail-in-a-Box . . .
+	echo Installing git . . .
 	apt-get -q -q install -y git
-	git clone -q --depth 1 -b master https://github.com/mail-in-a-box/mailinabox
+
+	echo Downloading Mail-in-a-Box . . .
+	git clone -b $TAG --depth 1 https://github.com/mail-in-a-box/mailinabox 2> /dev/null
 	cd mailinabox
 
 # If it does exist, update it.
 else
-	echo Updating Mail-in-a-Box . . .
+	echo Updating Mail-in-a-Box to $TAG . . .
 	cd mailinabox
-	if ! git pull -q --ff-only; then
+	git fetch
+	if ! git checkout -q $TAG; then
 		echo "Update failed. Did you modify something in `pwd`?"
 		exit
 	fi
@@ -38,3 +41,4 @@ fi
 
 # Start setup script.
 setup/start.sh
+

setup/csr_country_codes.tsv

@@ -0,0 +1,253 @@
+# This list is derived from https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2.
+# The columns are ISO_3166-1_alpha-2 code, display name, Wikipedia page name.
+# The top 20 countries by number of Internet users are grouped first, see
+# https://en.wikipedia.org/wiki/List_of_countries_by_number_of_Internet_users.
+BR	Brazil	
+CA	Canada	
+CN	China	
+EG	Egypt	
+FR	France	
+DE	Germany	
+IN	India	
+ID	Indonesia	
+IT	Italy	
+JP	Japan	
+MX	Mexico	
+NG	Nigeria	
+PH	Philippines	
+RU	Russian Federation	Russia
+ES	Spain	
+KR	South Korea
+TR	Turkey	
+GB	United Kingdom	
+US	United States	
+VN	Vietnam
+AD	Andorra	
+AE	United Arab Emirates	
+AF	Afghanistan	
+AG	Antigua and Barbuda	
+AI	Anguilla	
+AL	Albania	
+AM	Armenia	
+AO	Angola	
+AQ	Antarctica	
+AR	Argentina	
+AS	American Samoa	
+AT	Austria	
+AU	Australia	
+AW	Aruba	
+AX	Åland Islands	
+AZ	Azerbaijan	
+BA	Bosnia and Herzegovina	
+BB	Barbados	
+BD	Bangladesh	
+BE	Belgium	
+BF	Burkina Faso	
+BG	Bulgaria	
+BH	Bahrain	
+BI	Burundi	
+BJ	Benin	
+BL	Saint Barthélemy	
+BM	Bermuda	
+BN	Brunei
+BO	Bolivia
+BQ	Bonaire, Sint Eustatius and Saba	Caribbean Netherlands
+BS	Bahamas	The Bahamas
+BT	Bhutan	
+BV	Bouvet Island	
+BW	Botswana	
+BY	Belarus	
+BZ	Belize	
+CC	Cocos (Keeling) Islands	
+CD	Congo, the Democratic Republic of the	Democratic Republic of the Congo
+CF	Central African Republic	
+CG	Congo	Republic of the Congo
+CH	Switzerland	
+CI	Côte d'Ivoire	
+CK	Cook Islands	
+CL	Chile	
+CM	Cameroon	
+CO	Colombia	
+CR	Costa Rica	
+CU	Cuba	
+CV	Cabo Verde	
+CW	Curaçao	
+CX	Christmas Island	
+CY	Cyprus	
+CZ	Czech Republic	
+DJ	Djibouti	
+DK	Denmark	
+DM	Dominica	
+DO	Dominican Republic	
+DZ	Algeria	
+EC	Ecuador	
+EE	Estonia	
+EH	Western Sahara	
+ER	Eritrea	
+ET	Ethiopia	
+FI	Finland	
+FJ	Fiji	
+FK	Falkland Islands (Malvinas)	Falkland Islands
+FM	Federated States of Micronesia
+FO	Faroe Islands	
+GA	Gabon	
+GD	Grenada	
+GE	Georgia	Georgia (country)
+GF	French Guiana	
+GG	Guernsey	
+GH	Ghana	
+GI	Gibraltar	
+GL	Greenland	
+GM	Gambia	The Gambia
+GN	Guinea	
+GP	Guadeloupe	
+GQ	Equatorial Guinea	
+GR	Greece	
+GS	South Georgia and the South Sandwich Islands	
+GT	Guatemala	
+GU	Guam	
+GW	Guinea-Bissau	
+GY	Guyana	
+HK	Hong Kong	
+HM	Heard Island and McDonald Islands	
+HN	Honduras	
+HR	Croatia	
+HT	Haiti	
+HU	Hungary	
+IE	Ireland	Republic of Ireland
+IL	Israel	
+IM	Isle of Man	
+IO	British Indian Ocean Territory	
+IQ	Iraq	
+IR	Iran
+IS	Iceland	
+JE	Jersey	
+JM	Jamaica	
+JO	Jordan	
+KE	Kenya	
+KG	Kyrgyzstan	
+KH	Cambodia	
+KI	Kiribati	
+KM	Comoros	
+KN	Saint Kitts and Nevis	
+KP	North Korea
+KW	Kuwait	
+KY	Cayman Islands	
+KZ	Kazakhstan	
+LA	Laos
+LB	Lebanon	
+LC	Saint Lucia	
+LI	Liechtenstein	
+LK	Sri Lanka	
+LR	Liberia	
+LS	Lesotho	
+LT	Lithuania	
+LU	Luxembourg	
+LV	Latvia	
+LY	Libya	
+MA	Morocco	
+MC	Monaco	
+MD	Moldova
+ME	Montenegro	
+MF	Saint Martin (French part)	Collectivity of Saint Martin
+MG	Madagascar	
+MH	Marshall Islands	
+MK	Macedonia	Republic of Macedonia
+ML	Mali	
+MM	Myanmar	
+MN	Mongolia	
+MO	Macao	Macau
+MP	Northern Mariana Islands	
+MQ	Martinique	
+MR	Mauritania	
+MS	Montserrat	
+MT	Malta	
+MU	Mauritius	
+MV	Maldives	
+MW	Malawi	
+MY	Malaysia	
+MZ	Mozambique	
+NA	Namibia	
+NC	New Caledonia	
+NE	Niger	
+NF	Norfolk Island	
+NI	Nicaragua	
+NL	Netherlands	
+NO	Norway	
+NP	Nepal	
+NR	Nauru	
+NU	Niue	
+NZ	New Zealand	
+OM	Oman	
+PA	Panama	
+PE	Peru	
+PF	French Polynesia	
+PG	Papua New Guinea	
+PK	Pakistan	
+PL	Poland	
+PM	Saint Pierre and Miquelon	
+PN	Pitcairn	Pitcairn Islands
+PR	Puerto Rico	
+PS	Palestine	State of Palestine
+PT	Portugal	
+PW	Palau	
+PY	Paraguay	
+QA	Qatar	
+RE	Réunion	
+RO	Romania	
+RS	Serbia	
+RW	Rwanda	
+SA	Saudi Arabia	
+SB	Solomon Islands	
+SC	Seychelles	
+SD	Sudan	
+SE	Sweden	
+SG	Singapore	
+SH	Saint Helena, Ascension and Tristan da Cunha	
+SI	Slovenia	
+SJ	Svalbard and Jan Mayen	
+SK	Slovakia	
+SL	Sierra Leone	
+SM	San Marino	
+SN	Senegal	
+SO	Somalia	
+SR	Suriname	
+SS	South Sudan	
+ST	Sao Tome and Principe
+SV	El Salvador	
+SX	Sint Maarten (Dutch part)	Sint Maarten
+SY	Syria
+SZ	Swaziland	
+TC	Turks and Caicos Islands	
+TD	Chad	
+TF	French Southern Territories	French Southern and Antarctic Lands
+TG	Togo	
+TH	Thailand	
+TJ	Tajikistan	
+TK	Tokelau	
+TL	Timor-Leste	East Timor
+TM	Turkmenistan	
+TN	Tunisia	
+TO	Tonga	
+TT	Trinidad and Tobago	
+TV	Tuvalu	
+TW	Taiwan	
+TZ	Tanzania
+UA	Ukraine	
+UG	Uganda	
+UM	United States Minor Outlying Islands	
+UY	Uruguay	
+UZ	Uzbekistan	
+VA	Vatican City
+VC	Saint Vincent and the Grenadines	
+VE	Venezuela
+VG	Virgin Islands, British	British Virgin Islands
+VI	Virgin Islands, U.S.	United States Virgin Islands
+VU	Vanuatu	
+WF	Wallis and Futuna	
+WS	Samoa	
+YE	Yemen	
+YT	Mayotte	
+ZA	South Africa	
+ZM	Zambia	
+ZW	Zimbabwe	

setup/dns.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
-# DNS: Configure a DNS server using nsd
-#######################################
+# DNS: Configure a DNS server to host our own DNS
+# -----------------------------------------------
 
 # This script installs packages, but the DNS zone files are only
 # created by the /dns/update API in the management server because
@@ -9,21 +9,25 @@
 
 source setup/functions.sh # load our functions
 
-# Install nsd, our DNS server software, and ldnsutils which helps
+# Install `nsd`, our DNS server software, and `ldnsutils` which helps
 # us sign zones for DNSSEC.
 
 # ...but first, we have to create the user because the 
 # current Ubuntu forgets to do so in the .deb
 # see issue #25 and https://bugs.launchpad.net/ubuntu/+source/nsd/+bug/1311886
 if id nsd > /dev/null 2>&1; then
-	true; #echo "nsd user exists... good";
+	true; #echo "nsd user exists... good"; #NODOC
 else
 	useradd nsd;
 fi
 
 # Okay now install the packages.
+#
+# * nsd: The non-recursive nameserver that publishes our DNS records.
+# * ldnsutils: Helper utilities for signing DNSSEC zones.
+# * openssh-client: Provides ssh-keyscan which we use to create SSHFP records.
 
-apt_install nsd ldnsutils
+apt_install nsd ldnsutils openssh-client
 
 # Prepare nsd's configuration.
 
@@ -49,9 +53,10 @@ if [ ! -f "$STORAGE_ROOT/dns/dnssec/keys.conf" ]; then
 	ZSK=$(umask 077; cd $STORAGE_ROOT/dns/dnssec; ldns-keygen -a RSASHA1-NSEC3-SHA1 -b 1024 _domain_);
 
 	# These generate two sets of files like:
-	# K_domain_.+007+08882.ds <- DS record for adding to NSD configuration files
-	# K_domain_.+007+08882.key <- public key (goes into DS record & upstream DNS provider like your registrar)
-	# K_domain_.+007+08882.private  <- private key (secret!)
+	#
+	# * `K_domain_.+007+08882.ds`: DS record to provide to domain name registrar
+	# * `K_domain_.+007+08882.key`: public key (goes into DS record & upstream DNS provider like your registrar)
+	# * `K_domain_.+007+08882.private`: private key (secret!)
 
 	# The filenames are unpredictable and encode the key generation
 	# options. So we'll store the names of the files we just generated.

setup/firstuser.sh

@@ -0,0 +1,57 @@
+# If there aren't any mail users yet, create one.
+if [ -z "`tools/mail.py user`" ]; then
+	# The outut of "tools/mail.py user" is a list of mail users. If there
+	# aren't any yet, it'll be empty.
+
+	# If we didn't ask for an email address at the start, do so now.
+	if [ -z "$EMAIL_ADDR" ]; then
+		# In an interactive shell, ask the user for an email address.
+		if [ -z "$NONINTERACTIVE" ]; then
+			input_box "Mail Account" \
+				"Let's create your first mail account.
+				\n\nWhat email address do you want?" \
+				me@`get_default_hostname` \
+				EMAIL_ADDR
+
+			if [ -z "$EMAIL_ADDR" ]; then
+				# user hit ESC/cancel
+				exit
+			fi
+			while ! management/mailconfig.py validate-email "$EMAIL_ADDR"
+			do
+				input_box "Mail Account" \
+					"That's not a valid email address.
+					\n\nWhat email address do you want?" \
+					$EMAIL_ADDR \
+					EMAIL_ADDR
+				if [ -z "$EMAIL_ADDR" ]; then
+					# user hit ESC/cancel
+					exit
+				fi
+			done
+
+		# But in a non-interactive shell, just make something up.
+		# This is normally for testing.
+		else
+			# Use me@PRIMARY_HOSTNAME
+			EMAIL_ADDR=me@$PRIMARY_HOSTNAME
+			EMAIL_PW=1234
+			echo
+			echo "Creating a new administrative mail account for $EMAIL_ADDR with password $EMAIL_PW."
+			echo
+		fi
+	else
+		echo
+		echo "Okay. I'm about to set up $EMAIL_ADDR for you. This account will also"
+		echo "have access to the box's control panel."
+	fi
+
+	# Create the user's mail account. This will ask for a password if none was given above.
+	tools/mail.py user add $EMAIL_ADDR $EMAIL_PW
+
+	# Make it an admin.
+	hide_output tools/mail.py user make-admin $EMAIL_ADDR
+
+	# Create an alias to which we'll direct all automatically-created administrative aliases.
+	tools/mail.py alias add administrator@$PRIMARY_HOSTNAME $EMAIL_ADDR
+fi

setup/functions.sh

@@ -85,6 +85,9 @@ function get_default_privateip {
 	# Return the IP address of the network interface connected
 	# to the Internet.
 	#
+	# Pass '4' or '6' as an argument to this function to specify
+	# what type of address to get (IPv4, IPv6).
+	#
 	# We used to use `hostname -I` and then filter for either
 	# IPv4 or IPv6 addresses. However if there are multiple
 	# network interfaces on the machine, not all may be for
@@ -99,11 +102,16 @@ function get_default_privateip {
 	# assigned to an interface. `ip route get` reports the
 	# preferred. That's good enough for us. See issue #121.
 	#
+	# With IPv6, the best route may be via an interface that
+	# only has a link-local address (fe80::*). These addresses
+	# are only unique to an interface and so need an explicit
+	# interface specification in order to use them with bind().
+	# In these cases, we append "%interface" to the address.
+	# See the Notes section in the man page for getaddrinfo and
+	# https://discourse.mailinabox.email/t/update-broke-mailinabox/34/9.
+	#
 	# Also see ae67409603c49b7fa73c227449264ddd10aae6a9 and
 	# issue #3 for why/how we originally added IPv6.
-	#
-	# Pass '4' or '6' as an argument to this function to specify
-	# what type of address to get (IPv4, IPv6).
 
 	target=8.8.8.8
 
@@ -112,9 +120,21 @@ function get_default_privateip {
 	# as it's an address on the public Internet.
 	if [ "$1" == "6" ]; then target=2001:4860:4860::8888; fi
 
-	ip -$1 -o route get $target \
-		| grep -v unreachable \
-		| sed "s/.* src \([^ ]*\).*/\1/"
+	# Get the route information.
+	route=$(ip -$1 -o route get $target | grep -v unreachable)
+
+	# Parse the address out of the route information.
+	address=$(echo $route | sed "s/.* src \([^ ]*\).*/\1/")
+
+	if [[ "$1" == "6" && $address == fe80:* ]]; then
+		# For IPv6 link-local addresses, parse the interface out
+		# of the route information and append it with a '%'.
+		interface=$(echo $route | sed "s/.* dev \([^ ]*\).*/\1/")
+		address=$address%$interface
+	fi
+
+	echo $address
+		
 }
 
 function ufw_allow {
@@ -127,3 +147,29 @@ function ufw_allow {
 function restart_service {
 	hide_output service $1 restart
 }
+
+## Dialog Functions ##
+function message_box {
+	dialog --title "$1" --msgbox "$2" 0 0
+}
+
+function input_box {
+	# input_box "title" "prompt" "defaultvalue" VARIABLE
+	# The user's input will be stored in the variable VARIABLE.
+	# The exit code from dialog will be stored in VARIABLE_EXITCODE.
+	declare -n result=$4
+	declare -n result_code=$4_EXITCODE
+	result=$(dialog --stdout --title "$1" --inputbox "$2" 0 0 "$3")
+	result_code=$?
+}
+
+function input_menu {
+	# input_menu "title" "prompt" "tag item tag item" VARIABLE
+	# The user's input will be stored in the variable VARIABLE.
+	# The exit code from dialog will be stored in VARIABLE_EXITCODE.
+	declare -n result=$4
+	declare -n result_code=$4_EXITCODE
+	local IFS=^$'\n'
+	result=$(dialog --stdout --title "$1" --menu "$2" 0 0 0 $3)
+	result_code=$?
+}

setup/mail-dovecot.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 #
 # Dovecot (IMAP and LDA)
+# ----------------------
 #
 # Dovecot is *both* the IMAP server (the protocol that email applications
 # use to query a mailbox) as well as the local delivery agent (LDA),
@@ -17,13 +18,13 @@
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
-# Install packages.
+# ### Install packages and basic setup
 
 apt_install \
 	dovecot-core dovecot-imapd dovecot-lmtpd dovecot-sqlite sqlite3 \
 	dovecot-sieve dovecot-managesieved
 
-# The dovecot-imapd dovecot-lmtpd packages automatically enable IMAP and LMTP protocols.
+# The dovecot-imapd and dovecot-lmtpd packages automatically enable IMAP and LMTP protocols.
 
 # Set the location where we'll store user mailboxes.
 tools/editconf.py /etc/dovecot/conf.d/10-mail.conf \
@@ -31,7 +32,7 @@ tools/editconf.py /etc/dovecot/conf.d/10-mail.conf \
 	mail_privileged_group=mail \
 	first_valid_uid=0
 
-# IMAP
+# ### IMAP
 
 # Require that passwords are sent over SSL only, and allow the usual IMAP authentication mechanisms.
 # The LOGIN mechanism is supposedly for Microsoft products like Outlook to do SMTP login (I guess
@@ -62,7 +63,7 @@ sed -i "s/#port = 110/port = 0/" /etc/dovecot/conf.d/10-master.conf
 tools/editconf.py /etc/dovecot/conf.d/20-imap.conf \
 	imap_idle_notify_interval="4 mins"
 
-# LDA (LMTP)
+# ### LDA (LMTP)
 
 # Enable Dovecot's LDA service with the LMTP protocol. It will listen
 # in port 10026, and Spamassassin will be configured to pass mail there.
@@ -94,12 +95,12 @@ EOF
 tools/editconf.py /etc/dovecot/conf.d/15-lda.conf \
 	postmaster_address=postmaster@$PRIMARY_HOSTNAME
 
-# SIEVE
+# ### Sieve
 
 # Enable the Dovecot sieve plugin which let's users run scripts that process
 # mail as it comes in. We'll also set a global script that moves mail marked
 # as spam by Spamassassin into the user's Spam folder.
-sudo sed -i "s/#mail_plugins = .*/mail_plugins = \$mail_plugins sieve/" /etc/dovecot/conf.d/20-lmtp.conf
+sed -i "s/#mail_plugins = .*/mail_plugins = \$mail_plugins sieve/" /etc/dovecot/conf.d/20-lmtp.conf
 
 cat > /etc/dovecot/conf.d/99-local-sieve.conf << EOF;
 plugin {

setup/mail-postfix.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 #
 # Postfix (SMTP)
+# --------------
 #
 # Postfix handles the transmission of email between servers
 # using the SMTP protocol. It is a Mail Transfer Agent (MTA).
@@ -29,11 +30,11 @@
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
-# Install packages.
+# ### Install packages.
 
 apt_install postfix postgrey postfix-pcre ca-certificates
 
-# Basic Settings
+# ### Basic Settings
 
 # Have postfix listen on all network interfaces, set our name (the Debian default seems to be localhost),
 # and set the name of the local machine to localhost for xxx@localhost mail (but I don't think this will have any effect because
@@ -44,15 +45,16 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtpd_banner="\$myhostname ESMTP Hi, I'm a Mail-in-a-Box (Ubuntu/Postfix; see https://mailinabox.email/)" \
 	mydestination=localhost
 
-# Outgoing Mail
+# ### Outgoing Mail
 
 # Enable the 'submission' port 587 smtpd server and tweak its settings.
-# a) Require the best ciphers for incoming connections per http://baldric.net/2013/12/07/tls-ciphers-in-postfix-and-dovecot/.
-#    but without affecting opportunistic TLS on incoming mail, which will allow any cipher (it's better than none).
-# b) Give it a different name in syslog to distinguish it from the port 25 smtpd server.
-# c) Add a new cleanup service specific to the submission service ('authclean')
-#    that filters out privacy-sensitive headers on mail being sent out by
-#    authenticated users.
+#
+# * Require the best ciphers for incoming connections per http://baldric.net/2013/12/07/tls-ciphers-in-postfix-and-dovecot/.
+#   but without affecting opportunistic TLS on incoming mail, which will allow any cipher (it's better than none).
+# * Give it a different name in syslog to distinguish it from the port 25 smtpd server.
+# * Add a new cleanup service specific to the submission service ('authclean')
+#   that filters out privacy-sensitive headers on mail being sent out by
+#   authenticated users.
 tools/editconf.py /etc/postfix/master.cf -s -w \
 	"submission=inet n       -       -       -       -       smtpd
 	  -o syslog_name=postfix/submission
@@ -64,7 +66,7 @@ tools/editconf.py /etc/postfix/master.cf -s -w \
 # Install the `outgoing_mail_header_filters` file required by the new 'authclean' service.
 cp conf/postfix_outgoing_mail_header_filters /etc/postfix/outgoing_mail_header_filters
 
-# Enable TLS on incoming connections (i.e. ports 25 *and* 587) and
+# Enable TLS on these and all other connections (i.e. ports 25 *and* 587) and
 # require TLS before a user is allowed to authenticate. This also makes
 # opportunistic TLS available on *incoming* mail.
 tools/editconf.py /etc/postfix/main.cf \
@@ -74,6 +76,19 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtpd_tls_key_file=$STORAGE_ROOT/ssl/ssl_private_key.pem \
 	smtpd_tls_received_header=yes
 
+# Prevent non-authenticated users from sending mail that requires being
+# relayed elsewhere. We don't want to be an "open relay". On outbound
+# mail, require one of:
+#
+# * permit_sasl_authenticated: Authenticated users (i.e. on port 587).
+# * permit_mynetworks: Mail that originates locally.
+# * reject_unauth_destination: No one else. (Permits mail whose destination is local and rejects other mail.)
+tools/editconf.py /etc/postfix/main.cf \
+	smtpd_relay_restrictions=permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination
+
+
+# ### DANE
+#
 # When connecting to remote SMTP servers, prefer TLS and use DANE if available.
 #
 # Prefering ("opportunistic") TLS means Postfix will accept whatever SSL certificate the remote
@@ -98,38 +113,27 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtp_tls_CAfile=/etc/ssl/certs/ca-certificates.crt \
 	smtp_tls_loglevel=2
 
-# Incoming Mail
+# ### Incoming Mail
 
 # Pass any incoming mail over to a local delivery agent. Spamassassin
 # will act as the LDA agent at first. It is listening on port 10025
 # with LMTP. Spamassassin will pass the mail over to Dovecot after.
 #
-# In a basic setup we would pass mail directly to Dovecot like so:
-# tools/editconf.py /etc/postfix/main.cf virtual_transport=lmtp:unix:private/dovecot-lmtp
+# In a basic setup we would pass mail directly to Dovecot by setting
+# virtual_transport to `lmtp:unix:private/dovecot-lmtp`.
 #
 tools/editconf.py /etc/postfix/main.cf virtual_transport=lmtp:[127.0.0.1]:10025
 
-# Who can send outbound mail? The purpose of this is to prevent
-# non-authenticated users from sending mail that requires being
-# relayed elsewhere. We don't want to be an "open relay".
-#
-# permit_sasl_authenticated: Authenticated users (i.e. on port 587).
-# permit_mynetworks: Mail that originates locally.
-# reject_unauth_destination: No one else. (Permits mail whose destination is local and rejects other mail.)
-tools/editconf.py /etc/postfix/main.cf \
-	smtpd_relay_restrictions=permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination
-
 # Who can send mail to us? Some basic filters.
 #
-# reject_non_fqdn_sender: Reject not-nice-looking return paths.
-# reject_unknown_sender_domain: Reject return paths with invalid domains.
-# reject_rhsbl_sender: Reject return paths that use blacklisted domains.
-#
-# permit_sasl_authenticated: Authenticated users (i.e. on port 587) can skip further checks.
-# permit_mynetworks: Mail that originates locally can skip further checks.
-# reject_rbl_client: Reject connections from IP addresses blacklisted in zen.spamhaus.org
-# reject_unlisted_recipient: Although Postfix will reject mail to unknown recipients, it's nicer to reject such mail ahead of greylisting rather than after.
-# check_policy_service: Apply greylisting using postgrey.
+# * reject_non_fqdn_sender: Reject not-nice-looking return paths.
+# * reject_unknown_sender_domain: Reject return paths with invalid domains.
+# * reject_rhsbl_sender: Reject return paths that use blacklisted domains.
+# * permit_sasl_authenticated: Authenticated users (i.e. on port 587) can skip further checks.
+# * permit_mynetworks: Mail that originates locally can skip further checks.
+# * reject_rbl_client: Reject connections from IP addresses blacklisted in zen.spamhaus.org
+# * reject_unlisted_recipient: Although Postfix will reject mail to unknown recipients, it's nicer to reject such mail ahead of greylisting rather than after.
+# * check_policy_service: Apply greylisting using postgrey.
 #
 # Notes:
 # permit_dnswl_client can pass through mail from whitelisted IP addresses, which would be good to put before greylisting

setup/mail-users.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 #
 # User Authentication and Destination Validation
+# ----------------------------------------------
 #
 # This script configures user authentication for Dovecot
 # and Postfix (which relies on Dovecot) and destination
@@ -9,6 +10,8 @@
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
+# ### User and Alias Database
+
 # The database of mail users (i.e. authenticated users, who have mailboxes)
 # and aliases (forwarders).
 
@@ -21,8 +24,7 @@ if [ ! -f $db_path ]; then
 	echo "CREATE TABLE aliases (id INTEGER PRIMARY KEY AUTOINCREMENT, source TEXT NOT NULL UNIQUE, destination TEXT NOT NULL);" | sqlite3 $db_path;
 fi
 
-# User Authentication
-#####################
+# ### User Authentication
 
 # Have Dovecot query our database, and not system users, for authentication.
 sed -i "s/#*\(\!include auth-system.conf.ext\)/#\1/"  /etc/dovecot/conf.d/10-auth.conf
@@ -68,8 +70,7 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtpd_sasl_path=private/auth \
 	smtpd_sasl_auth_enable=yes
 
-# Destination Validation
-########################
+# ### Destination Validation
 
 # Use a Sqlite3 database to check whether a destination email address exists,
 # and to perform any email alias rewrites in Postfix.

setup/management.sh

@@ -2,13 +2,13 @@
 
 source setup/functions.sh
 
-apt_install python3-flask links duplicity libyaml-dev python3-dnspython
+apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil
 hide_output pip3 install rtyaml
 
 # Create a backup directory and a random key for encrypting backups.
 mkdir -p $STORAGE_ROOT/backup
 if [ ! -f $STORAGE_ROOT/backup/secret_key.txt ]; then
-	openssl rand -base64 2048 > $STORAGE_ROOT/backup/secret_key.txt
+	$(umask 077; openssl rand -base64 2048 > $STORAGE_ROOT/backup/secret_key.txt)
 fi
 
 # Link the management server daemon into a well known location.

setup/migrate.py

@@ -56,6 +56,10 @@ def migration_4(env):
 	db = os.path.join(env["STORAGE_ROOT"], 'mail/users.sqlite')
 	shell("check_call", ["sqlite3", db, "ALTER TABLE users ADD privileges TEXT NOT NULL DEFAULT ''"])
 
+def migration_5(env):
+	# The secret key for encrypting backups was world readable. Fix here.
+	os.chmod(os.path.join(env["STORAGE_ROOT"], 'backup/secret_key.txt'), 0o600)
+
 def get_current_migration():
 	ver = 0
 	while True:

setup/network-checks.sh

@@ -1,3 +1,7 @@
+# Install the 'host', 'sed', and and 'nc' tools. This script is run before
+# the rest of the system setup so we may not yet have things installed.
+hide_output apt-get -y install bind9-host sed netcat-openbsd
+
 # Stop if the PRIMARY_HOSTNAME is listed in the Spamhaus Domain Block List.
 # The user might have chosen a name that was previously in use by a spammer
 # and will not be able to reliably send mail. Do this after any automatic

setup/owncloud.sh

@@ -5,6 +5,8 @@
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
+# ### Installing ownCloud
+
 apt_install \
 	dbconfig-common \
 	php5-cli php5-sqlite php5-gd php5-imap php5-curl php-pear php-apc curl libapr1 libtool libcurl4-openssl-dev php-xml-parser \
@@ -12,23 +14,30 @@ apt_install \
 
 apt-get purge -qq -y owncloud*
 
-# Install ownCloud from source if it is not already present
-# TODO: Check version?
-if [ ! -d /usr/local/lib/owncloud ]; then
+# Install ownCloud from source of this version:
+owncloud_ver=7.0.2
+
+# Check if ownCloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
+if [ ! -d /usr/local/lib/owncloud/ ] \
+	|| ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then
+
 	echo installing ownCloud...
 	rm -f /tmp/owncloud.zip
-	wget -qO /tmp/owncloud.zip https://download.owncloud.org/community/owncloud-7.0.1.zip
-	unzip -q /tmp/owncloud.zip -d /usr/local/lib
+	wget -qO /tmp/owncloud.zip https://download.owncloud.org/community/owncloud-$owncloud_ver.zip
+	unzip -u -o -q /tmp/owncloud.zip -d /usr/local/lib #either extracts new or replaces current files
+	hide_output php /usr/local/lib/owncloud/occ upgrade #if OC is up-to-date it wont matter
 	rm -f /tmp/owncloud.zip
 fi
 
+# ### Configuring ownCloud
+
 # Setup ownCloud if the ownCloud database does not yet exist. Running setup when
 # the database does exist wipes the database and user data.
 if [ ! -f $STORAGE_ROOT/owncloud/owncloud.db ]; then
 	# Create a configuration file.
-	TIMEZONE=`cat /etc/timezone`
+	TIMEZONE=$(cat /etc/timezone)
 	instanceid=oc$(echo $PRIMARY_HOSTNAME | sha1sum | fold -w 10 | head -n 1)
-	cat - > /usr/local/lib/owncloud/config/config.php <<EOF;
+	cat > /usr/local/lib/owncloud/config/config.php <<EOF;
 <?php
 \$CONFIG = array (
   'datadirectory' => '$STORAGE_ROOT/owncloud',
@@ -69,8 +78,8 @@ EOF
 	# Create an auto-configuration file to fill in database settings
 	# when the install script is run. Make an administrator account
 	# here or else the install can't finish.
-	adminpassword=$(dd if=/dev/random bs=40 count=1 2>/dev/null | sha1sum | fold -w 30 | head -n 1)
-	cat - > /usr/local/lib/owncloud/config/autoconfig.php <<EOF;
+	adminpassword=$(dd if=/dev/random bs=1 count=40 2>/dev/null | sha1sum | fold -w 30 | head -n 1)
+	cat > /usr/local/lib/owncloud/config/autoconfig.php <<EOF;
 <?php
 \$AUTOCONFIG = array (
   # storage/database
@@ -118,13 +127,15 @@ sudo -u www-data php -f /usr/local/lib/owncloud/cron.php
 EOF
 chmod +x /etc/cron.hourly/mailinabox-owncloud
 
-## Ensure all system admins are ownCloud admins.
-## Actually we don't do this. There's nothing much of interest that the user could
-## change from the ownCloud admin, and there's a lot they could mess up.
-#for user in $(tools/mail.py user admins); do
-#	sqlite3 $STORAGE_ROOT/owncloud/owncloud.db "INSERT OR IGNORE INTO oc_group_user VALUES ('admin', '$user')"
-#done
+# There's nothing much of interest that a user could do as an admin for ownCloud,
+# and there's a lot they could mess up, so we don't make any users admins of ownCloud.
+# But if we wanted to, we would do this:
+# ```
+# for user in $(tools/mail.py user admins); do
+#	 sqlite3 $STORAGE_ROOT/owncloud/owncloud.db "INSERT OR IGNORE INTO oc_group_user VALUES ('admin', '$user')"
+# done
+# ```
 
-# Finished.
+# Enable PHP modules and restart PHP.
 php5enmod imap
 restart_service php5-fpm

setup/preflight.sh

@@ -0,0 +1,29 @@
+# Are we running as root?
+if [[ $EUID -ne 0 ]]; then
+	echo "This script must be run as root. Please re-run like this:"
+	echo
+	echo "sudo $0"
+	echo
+	exit
+fi
+
+# Check that we are running on Ubuntu 14.04 LTS (or 14.04.xx).
+if [ "`lsb_release -d | sed 's/.*:\s*//' | sed 's/14\.04\.[0-9]/14.04/' `" != "Ubuntu 14.04 LTS" ]; then
+	echo "Mail-in-a-Box only supports being installed on Ubuntu 14.04, sorry. You are running:"
+	echo
+	lsb_release -d | sed 's/.*:\s*//'
+	echo
+	echo "We can't write scripts that run on every possible setup, sorry."
+	exit
+fi
+
+# Check that we have enough memory. Skip the check if we appear to be
+# running inside of Vagrant, because that's really just for testing.
+TOTAL_PHYSICAL_MEM=$(head -n 1 /proc/meminfo | awk '{print $2}')
+if [ $TOTAL_PHYSICAL_MEM -lt 786432 ]; then
+if [ ! -d /vagrant ]; then
+	echo "Your Mail-in-a-Box needs more than $TOTAL_PHYSICAL_MEM MB RAM."
+	echo "Please provision a machine with at least 768 MB, 1 GB recommended."
+	exit
+fi
+fi

setup/questions.sh

@@ -0,0 +1,188 @@
+if [ -z "$NONINTERACTIVE" ]; then
+	# Install 'dialog' so we can ask the user questions. The original motivation for
+	# this was being able to ask the user for input even if stdin has been redirected,
+	# e.g. if we piped a bootstrapping install script to bash to get started. In that
+	# case, the nifty '[ -t 0 ]' test won't work. But with Vagrant we must suppress so we
+	# use a shell flag instead. Really supress any output from installing dialog.
+	hide_output apt-get -y install dialog
+	message_box "Mail-in-a-Box Installation" \
+		"Hello and thanks for deploying a Mail-in-a-Box!
+		\n\nI'm going to ask you a few questions.
+		\n\nTo change your answers later, just run 'sudo mailinabox' from the command line."
+fi
+
+# The box needs a name.
+if [ -z "$PRIMARY_HOSTNAME" ]; then
+	if [ -z "$DEFAULT_PRIMARY_HOSTNAME" ]; then
+		# We recommend to use box.example.com as this hosts name. The
+		# domain the user possibly wants to use is example.com then.
+		# We strip the string "box." from the hostname to get the mail
+		# domain. If the hostname differs, nothing happens here.
+		DEFAULT_DOMAIN_GUESS=$(echo $(get_default_hostname) | sed -e 's/^box\.//')
+
+		# This is the first run. Ask the user for his email address so we can
+		# provide the best default for the box's hostname.
+		input_box "Your Email Address" \
+"What email address are you setting this box up to manage?
+\n\nThe part after the @-sign must be a domain name or subdomain
+that you control. You can add other email addresses to this
+box later (including email addresses on other domain names
+or subdomains you control).
+\n\nWe've guessed an email address. Backspace it and type in what
+you really want.
+\n\nEmail Address:" \
+			"me@$DEFAULT_DOMAIN_GUESS" \
+			EMAIL_ADDR
+
+		if [ -z "$EMAIL_ADDR" ]; then
+			# user hit ESC/cancel
+			exit
+		fi
+		while ! management/mailconfig.py validate-email "$EMAIL_ADDR"
+		do
+			input_box "Your Email Address" \
+				"That's not a valid email address.\n\nWhat email address are you setting this box up to manage?" \
+				$EMAIL_ADDR \
+				EMAIL_ADDR
+			if [ -z "$EMAIL_ADDR" ]; then
+				# user hit ESC/cancel
+				exit
+			fi
+		done
+
+		# Take the part after the @-sign as the user's domain name, and add
+		# 'box.' to the beginning to create a default hostname for this machine.
+		DEFAULT_PRIMARY_HOSTNAME=box.$(echo $EMAIL_ADDR | sed 's/.*@//')
+	fi
+
+	input_box "Hostname" \
+"This box needs a name, called a 'hostname'. The name will form a part of the box's web address.
+\n\nWe recommend that the name be a subdomain of the domain in your email
+address, so we're suggesting $DEFAULT_PRIMARY_HOSTNAME.
+\n\nYou can change it, but we recommend you don't.
+\n\nHostname:" \
+		$DEFAULT_PRIMARY_HOSTNAME \
+		PRIMARY_HOSTNAME
+
+	if [ -z "$PRIMARY_HOSTNAME" ]; then
+		# user hit ESC/cancel
+		exit
+	fi
+fi
+
+# If the machine is behind a NAT, inside a VM, etc., it may not know
+# its IP address on the public network / the Internet. Ask the Internet
+# and possibly confirm with user.
+if [ -z "$PUBLIC_IP" ]; then
+	# Ask the Internet.
+	GUESSED_IP=$(get_publicip_from_web_service 4)
+
+	# On the first run, if we got an answer from the Internet then don't
+	# ask the user.
+	if [[ -z "$DEFAULT_PUBLIC_IP" && ! -z "$GUESSED_IP" ]]; then
+		PUBLIC_IP=$GUESSED_IP
+
+	# Otherwise on the first run at least provide a default.
+	elif [[ -z "$DEFAULT_PUBLIC_IP" ]]; then
+		DEFAULT_PUBLIC_IP=$(get_default_privateip 4)
+
+	# On later runs, if the previous value matches the guessed value then
+	# don't ask the user either.
+	elif [ "$DEFAULT_PUBLIC_IP" == "$GUESSED_IP" ]; then
+		PUBLIC_IP=$GUESSED_IP
+	fi
+
+	if [ -z "$PUBLIC_IP" ]; then
+		input_box "Public IP Address" \
+			"Enter the public IP address of this machine, as given to you by your ISP.
+			\n\nPublic IP address:" \
+			$DEFAULT_PUBLIC_IP \
+			PUBLIC_IP
+
+		if [ -z "$PUBLIC_IP" ]; then
+			# user hit ESC/cancel
+			exit
+		fi
+	fi
+fi
+
+# Same for IPv6. But it's optional. Also, if it looks like the system
+# doesn't have an IPv6, don't ask for one.
+if [ -z "$PUBLIC_IPV6" ]; then
+	# Ask the Internet.
+	GUESSED_IP=$(get_publicip_from_web_service 6)
+	MATCHED=0
+	if [[ -z "$DEFAULT_PUBLIC_IPV6" && ! -z "$GUESSED_IP" ]]; then
+		PUBLIC_IPV6=$GUESSED_IP
+	elif [[ "$DEFAULT_PUBLIC_IPV6" == "$GUESSED_IP" ]]; then
+		# No IPv6 entered and machine seems to have none, or what
+		# the user entered matches what the Internet tells us.
+		PUBLIC_IPV6=$GUESSED_IP
+		MATCHED=1
+	elif [[ -z "$DEFAULT_PUBLIC_IPV6" ]]; then
+		DEFAULT_PUBLIC_IP=$(get_default_privateip 6)
+	fi
+
+	if [[ -z "$PUBLIC_IPV6" && $MATCHED == 0 ]]; then
+		input_box "IPv6 Address (Optional)" \
+			"Enter the public IPv6 address of this machine, as given to you by your ISP.
+			\n\nLeave blank if the machine does not have an IPv6 address.
+			\n\nPublic IPv6 address:" \
+			$DEFAULT_PUBLIC_IPV6 \
+			PUBLIC_IPV6
+
+		if [ ! $PUBLIC_IPV6_EXITCODE ]; then
+			# user hit ESC/cancel
+			exit
+		fi
+	fi
+fi
+
+# Get the IP addresses of the local network interface(s) that are connected
+# to the Internet. We need these when we want to have services bind only to
+# the public network interfaces (not loopback, not tunnel interfaces).
+if [ -z "$PRIVATE_IP" ]; then
+	PRIVATE_IP=$(get_default_privateip 4)
+fi
+if [ -z "$PRIVATE_IPV6" ]; then
+	PRIVATE_IPV6=$(get_default_privateip 6)
+fi
+if [[ -z "$PRIVATE_IP" && -z "$PRIVATE_IPV6" ]]; then
+	echo
+	echo "I could not determine the IP or IPv6 address of the network inteface"
+	echo "for connecting to the Internet. Setup must stop."
+	echo
+	hostname -I
+	route
+	echo
+	exit
+fi
+
+# We need a country code to generate a certificate signing request. However
+# if a CSR already exists then we won't be generating a new one and there's
+# no reason to ask for the country code now. $STORAGE_ROOT has not yet been
+# set so we'll check if $DEFAULT_STORAGE_ROOT and $DEFAULT_CSR_COUNTRY are
+# set (the values from the current mailinabox.conf) and if the CSR exists
+# in the expected location.
+if [ ! -z "$DEFAULT_STORAGE_ROOT" ] && [ ! -z "$DEFAULT_CSR_COUNTRY" ] && [ -f $DEFAULT_STORAGE_ROOT/ssl/ssl_cert_sign_req.csr ]; then
+	CSR_COUNTRY=$DEFAULT_CSR_COUNTRY
+fi
+
+if [ -z "$CSR_COUNTRY" ]; then
+	# Get a list of country codes. Separate codes from country names with a ^.
+	# The input_menu function modifies shell word expansion to ignore spaces
+	# (since country names can have spaces) and use ^ instead.
+	country_code_list=$(grep -v "^#" setup/csr_country_codes.tsv | sed "s/\(..\)\t\([^\t]*\).*/\1^\2/")
+
+	input_menu "Country Code" \
+		"Choose the country where you live or where your organization is based.
+		\n\n(This is used to create an SSL certificate.)
+		\n\nCountry Code:" \
+		"$country_code_list" \
+		CSR_COUNTRY
+
+	if [ -z "$CSR_COUNTRY" ]; then
+		# user hit ESC/cancel
+		exit
+	fi
+fi

setup/ssl.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 #
 # SSL Certificate
+# ---------------
 #
 # Create a self-signed SSL certificate if one has not yet been created.
 #
@@ -21,20 +22,22 @@ source /etc/mailinabox.conf # load global vars
 apt_install openssl
 
 mkdir -p $STORAGE_ROOT/ssl
+# Generate a new private key if one doesn't already exist.
+# Set the umask so the key file is not world-readable.
 if [ ! -f $STORAGE_ROOT/ssl/ssl_private_key.pem ]; then
-	# Generate a new private key if one doesn't already exist.
-	# Set the umask so the key file is not world-readable.
 	(umask 077; hide_output \
 		openssl genrsa -out $STORAGE_ROOT/ssl/ssl_private_key.pem 2048)
 fi
+
+# Generate a certificate signing request if one doesn't already exist.
 if [ ! -f $STORAGE_ROOT/ssl/ssl_cert_sign_req.csr ]; then
-	# Generate a certificate signing request if one doesn't already exist.
 	hide_output \
 	openssl req -new -key $STORAGE_ROOT/ssl/ssl_private_key.pem -out $STORAGE_ROOT/ssl/ssl_cert_sign_req.csr \
-	  -subj "/C=$CSR_COUNTRY/ST=/L=/O=/CN=$PRIMARY_HOSTNAME"
+	  -sha256 -subj "/C=$CSR_COUNTRY/ST=/L=/O=/CN=$PRIMARY_HOSTNAME"
 fi
+
+# Generate a SSL certificate by self-signing if a SSL certificate doesn't yet exist.
 if [ ! -f $STORAGE_ROOT/ssl/ssl_certificate.pem ]; then
-	# Generate a SSL certificate by self-signing if a SSL certificate doesn't yet exist.
 	hide_output \
 	openssl x509 -req -days 365 \
 	  -in $STORAGE_ROOT/ssl/ssl_cert_sign_req.csr -signkey $STORAGE_ROOT/ssl/ssl_private_key.pem -out $STORAGE_ROOT/ssl/ssl_certificate.pem

setup/start.sh

@@ -4,47 +4,9 @@
 
 source setup/functions.sh # load our functions
 
-# Check system setup.
-
-# Are we running as root?
-if [[ $EUID -ne 0 ]]; then
-	echo "This script must be run as root. Please re-run like this:"
-	echo
-	echo "sudo setup/start.sh"
-	echo
-	exit
-fi
-
-# Check that we are running on Ubuntu 14.04 LTS (or 14.04.xx).
-if [ "`lsb_release -d | sed 's/.*:\s*//' | sed 's/14\.04\.[0-9]/14.04/' `" != "Ubuntu 14.04 LTS" ]; then
-	echo "Mail-in-a-Box only supports being installed on Ubuntu 14.04, sorry. You are running:"
-	echo
-	lsb_release -d | sed 's/.*:\s*//'
-	echo
-	echo "We can't write scripts that run on every possible setup, sorry."
-	exit
-fi
-
-# Check that we have enough memory. Skip the check if we appear to be
-# running inside of Vagrant, because that's really just for testing.
-TOTAL_PHYSICAL_MEM=$(free -m | grep ^Mem: | sed "s/^Mem: *\([0-9]*\).*/\1/")
-if [ $TOTAL_PHYSICAL_MEM -lt 768 ]; then
-if [ ! -d /vagrant ]; then
-	echo "Your Mail-in-a-Box needs more than $TOTAL_PHYSICAL_MEM MB RAM."
-	echo "Please provision a machine with at least 768 MB, 1 GB recommended."
-	exit
-fi
-fi
-
-if [ -t 0 ]; then
-	# In an interactive shell...
-	echo
-	echo "Hello and thanks for deploying a Mail-in-a-Box!"
-	echo "-----------------------------------------------"
-	echo
-	echo "I'm going to ask you a few questions. To change your answers later,"
-	echo "later, just re-run this script."
-fi
+# Check system setup: Are we running as root on Ubuntu 14.04 on a
+# machine with enough memory? If not, this shows an error and exits.
+. setup/preflight.sh
 
 # Recall the last settings used if we're running this a second time.
 if [ -f /etc/mailinabox.conf ]; then
@@ -59,151 +21,19 @@ if [ -f /etc/mailinabox.conf ]; then
 	rm -f /tmp/mailinabox.prev.conf
 fi
 
-# The box needs a name.
-if [ -z "$PRIMARY_HOSTNAME" ]; then
-	if [ -z "$DEFAULT_PRIMARY_HOSTNAME" ]; then
-		# This is the first run. Ask the user for his email address so we can
-		# provide the best default for the box's hostname.
-		echo
-		echo "What email address are you setting this box up to manage?"
-		echo ""
-		echo "The part after the @-sign must be a domain name or subdomain"
-		echo "that you control. You can add other email addresses to this"
-		echo "box later (including email addresses on other domain names"
-		echo "or subdomains you control)."
-		echo
-		echo "We've guessed an email address. Backspace it and type in what"
-		echo "you really want."
-		echo
-		read -e -i "me@`get_default_hostname`" -p "Email Address: " EMAIL_ADDR
+# Put a start script in a global location. We tell the user to run 'mailinabox'
+# in the first dialog prompt, so we should do this before that starts.
+cat > /usr/local/bin/mailinabox << EOF;
+#!/bin/bash
+cd `pwd`
+source setup/start.sh
+EOF
+chmod +x /usr/local/bin/mailinabox
 
-		while ! management/mailconfig.py validate-email "$EMAIL_ADDR"
-		do
-		   echo "That's not a valid email address."
-		   echo
-		read -e -i "$EMAIL_ADDR" -p "Email Address: " EMAIL_ADDR
-		done
-
-		# Take the part after the @-sign as the user's domain name, and add
-		# 'box.' to the beginning to create a default hostname for this machine.
-		DEFAULT_PRIMARY_HOSTNAME=box.$(echo $EMAIL_ADDR | sed 's/.*@//')
-	fi
-
-	echo
-	echo "This box needs a name, called a 'hostname'. The name will form a part"
-	echo "of the box's web address."
-	echo
-	echo "We recommend that the name be a subdomain of the domain in your email"
-	echo "address, so we're suggesting $DEFAULT_PRIMARY_HOSTNAME."
-	echo
-	echo "You can change it, but we recommend you don't."
-	echo
-
-	read -e -i "$DEFAULT_PRIMARY_HOSTNAME" -p "Hostname: " PRIMARY_HOSTNAME
-fi
-
-# If the machine is behind a NAT, inside a VM, etc., it may not know
-# its IP address on the public network / the Internet. Ask the Internet
-# and possibly confirm with user.
-if [ -z "$PUBLIC_IP" ]; then
-	# Ask the Internet.
-	GUESSED_IP=$(get_publicip_from_web_service 4)
-
-	# On the first run, if we got an answer from the Internet then don't
-	# ask the user.
-	if [[ -z "$DEFAULT_PUBLIC_IP" && ! -z "$GUESSED_IP" ]]; then
-		PUBLIC_IP=$GUESSED_IP
-
-	# Otherwise on the first run at least provide a default.
-	elif [[ -z "$DEFAULT_PUBLIC_IP" ]]; then
-		DEFAULT_PUBLIC_IP=$(get_default_privateip 4)
-
-	# On later runs, if the previous value matches the guessed value then
-	# don't ask the user either.
-	elif [ "$DEFAULT_PUBLIC_IP" == "$GUESSED_IP" ]; then
-		PUBLIC_IP=$GUESSED_IP
-	fi
-
-	if [ -z "$PUBLIC_IP" ]; then
-		echo
-		echo "Enter the public IP address of this machine, as given to you by your ISP."
-		echo
-
-		read -e -i "$DEFAULT_PUBLIC_IP" -p "Public IP: " PUBLIC_IP
-	fi
-fi
-
-# Same for IPv6. But it's optional. Also, if it looks like the system
-# doesn't have an IPv6, don't ask for one.
-if [ -z "$PUBLIC_IPV6" ]; then
-	# Ask the Internet.
-	GUESSED_IP=$(get_publicip_from_web_service 6)
-	MATCHED=0
-	if [[ -z "$DEFAULT_PUBLIC_IPV6" && ! -z "$GUESSED_IP" ]]; then
-		PUBLIC_IPV6=$GUESSED_IP
-	elif [[ "$DEFAULT_PUBLIC_IPV6" == "$GUESSED_IP" ]]; then
-		# No IPv6 entered and machine seems to have none, or what
-		# the user entered matches what the Internet tells us.
-		PUBLIC_IPV6=$GUESSED_IP
-		MATCHED=1
-	elif [[ -z "$DEFAULT_PUBLIC_IPV6" ]]; then
-		DEFAULT_PUBLIC_IP=$(get_default_privateip 6)
-	fi
-
-	if [[ -z "$PUBLIC_IPV6" && $MATCHED == 0 ]]; then
-		echo
-		echo "Optional:"
-		echo "Enter the public IPv6 address of this machine, as given to you by your ISP."
-		echo "Leave blank if the machine does not have an IPv6 address."
-		echo
-
-		read -e -i "$DEFAULT_PUBLIC_IPV6" -p "Public IPv6: " PUBLIC_IPV6
-	fi
-fi
-
-# Get the IP addresses of the local network interface(s) that are connected
-# to the Internet. We need these when we want to have services bind only to
-# the public network interfaces (not loopback, not tunnel interfaces).
-if [ -z "$PRIVATE_IP" ]; then
-	PRIVATE_IP=$(get_default_privateip 4)
-fi
-if [ -z "$PRIVATE_IPV6" ]; then
-	PRIVATE_IPV6=$(get_default_privateip 6)
-fi
-if [[ -z "$PRIVATE_IP" && -z "$PRIVATE_IPV6" ]]; then
-	echo
-	echo "I could not determine the IP or IPv6 address of the network inteface"
-	echo "for connecting to the Internet. Setup must stop."
-	echo
-	hostname -I
-	route
-	echo
-	exit
-fi
-
-# We need a country code to generate a certificate signing request. However
-# if a CSR already exists then we won't be generating a new one and there's
-# no reason to ask for the country code now. $STORAGE_ROOT has not yet been
-# set so we'll check if $DEFAULT_STORAGE_ROOT and $DEFAULT_CSR_COUNTRY are
-# set (the values from the current mailinabox.conf) and if the CSR exists
-# in the expected location.
-if [ ! -z "$DEFAULT_STORAGE_ROOT" ] && [ ! -z "$DEFAULT_CSR_COUNTRY" ] && [ -f $DEFAULT_STORAGE_ROOT/ssl/ssl_cert_sign_req.csr ]; then
-	CSR_COUNTRY=$DEFAULT_CSR_COUNTRY
-fi
-if [ -z "$CSR_COUNTRY" ]; then
-	echo
-	echo "Enter the two-letter, uppercase country code for where you"
-	echo "live or where your organization is based. (This is used to"
-	echo "create an SSL certificate.)"
-	echo
-
-	#if [ -z "$DEFAULT_CSR_COUNTRY" ]; then
-	#	# set a default on first run
-	#	DEFAULT_CSR_COUNTRY=...?
-	#fi
-
-	read -e -i "$DEFAULT_CSR_COUNTRY" -p "Country Code: " CSR_COUNTRY
-fi
+# Ask the user for the PRIMARY_HOSTNAME, PUBLIC_IP, PUBLIC_IPV6, and CSR_COUNTRY
+# if values have not already been set in environment variables. When running
+# non-interactively, be sure to set values for all!
+. setup/questions.sh
 
 # Automatic configuration, e.g. as used in our Vagrant configuration.
 if [ "$PUBLIC_IP" = "auto" ]; then
@@ -232,6 +62,9 @@ fi
 if [ "$PRIVATE_IPV6" != "$PUBLIC_IPV6" ]; then
 	echo "Private IPv6 Address: $PRIVATE_IPV6"
 fi
+if [ -f /usr/bin/git ]; then
+	echo "Mail-in-a-Box Version: " $(git describe)
+fi
 echo
 
 # Run some network checks to make sure setup on this machine makes sense.
@@ -284,44 +117,9 @@ curl -s -d POSTDATA --user $(</var/lib/mailinabox/api.key): http://127.0.0.1:102
 curl -s -d POSTDATA --user $(</var/lib/mailinabox/api.key): http://127.0.0.1:10222/web/update
 
 # If there aren't any mail users yet, create one.
-if [ -z "`tools/mail.py user`" ]; then
-	# The outut of "tools/mail.py user" is a list of mail users. If there
-	# aren't any yet, it'll be empty.
-
-	# If we didn't ask for an email address at the start, do so now.
-	if [ -z "$EMAIL_ADDR" ]; then
-		# In an interactive shell, ask the user for an email address.
-		if [ -t 0 ]; then
-			echo
-			echo "Let's create your first mail user."
-			read -e -i "user@$PRIMARY_HOSTNAME" -p "Email Address: " EMAIL_ADDR
-
-		# But in a non-interactive shell, just make something up. This
-		# is normally for testing.
-		else
-			# Use me@PRIMARY_HOSTNAME
-			EMAIL_ADDR=me@$PRIMARY_HOSTNAME
-			EMAIL_PW=1234
-			echo
-			echo "Creating a new administrative mail account for $EMAIL_ADDR with password $EMAIL_PW."
-			echo
-		fi
-	else
-		echo
-		echo "Okay. I'm about to set up $EMAIL_ADDR for you. This account will also"
-		echo "have access to the box's control panel."
-	fi
-
-	# Create the user's mail account. This will ask for a password if none was given above.
-	tools/mail.py user add $EMAIL_ADDR $EMAIL_PW
-
-	# Make it an admin.
-	hide_output tools/mail.py user make-admin $EMAIL_ADDR
-
-	# Create an alias to which we'll direct all automatically-created administrative aliases.
-	tools/mail.py alias add administrator@$PRIMARY_HOSTNAME $EMAIL_ADDR
-fi
+. setup/firstuser.sh
 
+# Done.
 echo
 echo "-----------------------------------------------"
 echo
@@ -329,18 +127,19 @@ echo Your Mail-in-a-Box is running.
 echo
 echo Please log in to the control panel for further instructions at:
 echo
-if management/whats_next.py --check-primary-hostname; then
+if management/status_checks.py --check-primary-hostname; then
 	# Show the nice URL if it appears to be resolving and has a valid certificate.
 	echo https://$PRIMARY_HOSTNAME/admin
 	echo
-	echo If there are problems with this URL, instead use:
+	echo If you have a DNS problem use the box\'s IP address and check the SSL fingerprint:
+	echo https://$PUBLIC_IP/admin
+else
+	echo https://$PUBLIC_IP/admin
+	echo
+	echo You will be alerted that the website has an invalid certificate. Check that
+	echo the certificate fingerprint matches:
 	echo
 fi
-echo https://$PUBLIC_IP/admin
-echo
-echo You will be alerted that the website has an invalid certificate. Check that
-echo the certificate fingerprint matches:
-echo
 openssl x509 -in $STORAGE_ROOT/ssl/ssl_certificate.pem -noout -fingerprint \
         | sed "s/SHA1 Fingerprint=//"
 echo

setup/system.sh

@@ -1,6 +1,11 @@
 source setup/functions.sh # load our functions
 
-# Base system configuration.
+# Base system configuration
+# -------------------------
+
+# ### Base packages
+
+# Update system packages:
 
 echo Updating system packages...
 hide_output apt-get update
@@ -8,53 +13,79 @@ hide_output apt-get -y upgrade
 
 # Install basic utilities.
 #
-#	haveged: Provides extra entropy to /dev/random so it doesn't stall
+# * haveged: Provides extra entropy to /dev/random so it doesn't stall
 #	         when generating random numbers for private keys (e.g. during
 #	         ldns-keygen).
+# * unattended-upgrades: Apt tool to install security updates automatically.
+# * ntp: keeps the system time correct
+# * fail2ban: scans log files for repeated failed login attempts and blocks the remote IP at the firewall
 
-apt_install python3 python3-pip wget curl bind9-host haveged
+apt_install python3 python3-dev python3-pip \
+	wget curl \
+	haveged unattended-upgrades ntp fail2ban
 
-# Turn on basic services:
+# Allow apt to install system updates automatically every day.
+
+cat > /etc/apt/apt.conf.d/02periodic <<EOF;
+APT::Periodic::MaxAge "7";
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Unattended-Upgrade "1";
+APT::Periodic::Verbose "1";
+EOF
+
+# ### Firewall
+
+# Turn on the firewall.
 #
-#   ntp: keeps the system time correct
-#
-#   fail2ban: scans log files for repeated failed login attempts and blocks the remote IP at the firewall
-#
-# These services don't need further configuration and are started immediately after installation.
-
-apt_install ntp fail2ban
-
+# Various virtualized environments like Docker and some VPSs don't provide #NODOC
+# a kernel that supports iptables. To avoid error-like output in these cases, #NODOC
+# we skip this if the user sets DISABLE_FIREWALL=1. #NODOC
 if [ -z "$DISABLE_FIREWALL" ]; then
-	# Turn on the firewall. First allow incoming SSH, then turn on the firewall.
-	# Other ports will be opened at the point where we set up those services.
-	#
-	# Various virtualized environments like Docker and some VPSs don't provide
-	# a kernel that supports iptables. To avoid error-like output in these cases,
-	# let us disable the firewall.
+	# Install `ufw` which provides a simple firewall configuration.
 	apt_install ufw
-	ufw_allow ssh;
-	ufw --force enable;
-fi
 
-# Resolve DNS using bind9 locally, rather than whatever DNS server is supplied
-# by the machine's network configuration. We do this to ensure that DNS queries
+	# Allow incoming connections to SSH.
+	ufw_allow ssh;
+
+	# ssh might be running on an alternate port. Use sshd -T to dump sshd's
+	# settings, find the port it is supposedly running on, and open that port
+	# too.
+	SSH_PORT=$(sshd -T 2>/dev/null | grep "^port " | sed "s/port //")
+	if [ ! -z "$SSH_PORT" ]; then
+	if [ "$SSH_PORT" != "22" ]; then
+
+	echo Opening alternate SSH port $SSH_PORT.
+	ufw_allow $SSH_PORT;
+
+	fi
+	fi
+
+	ufw --force enable;
+fi #NODOC
+
+# ### Local DNS Service
+
+# Install a local DNS server, rather than using the DNS server provided by the
+# ISP's network configuration.
+#
+# We do this to ensure that DNS queries
 # that *we* make (i.e. looking up other external domains) perform DNSSEC checks.
 # We could use Google's Public DNS, but we don't want to create a dependency on
-# Google per our goals of decentralization. bind9, as packaged for Ubuntu, has
+# Google per our goals of decentralization. `bind9`, as packaged for Ubuntu, has
 # DNSSEC enabled by default via "dnssec-validation auto".
 #
-# So we'll be running bind9 bound to 127.0.0.1 for locally-issued DNS queries
-# and nsd bound to the public ethernet interface for remote DNS queries asking
-# about our domain names. nsd is configured in dns.sh.
+# So we'll be running `bind9` bound to 127.0.0.1 for locally-issued DNS queries
+# and `nsd` bound to the public ethernet interface for remote DNS queries asking
+# about our domain names. `nsd` is configured later.
 #
 # About the settings:
 #
-# * RESOLVCONF=yes will have bind9 take over /etc/resolv.conf to tell
+# * RESOLVCONF=yes will have `bind9` take over /etc/resolv.conf to tell
 #   local services that DNS queries are handled on localhost.
-# * Adding -4 to OPTIONS will have bind9 not listen on IPv6 addresses
+# * Adding -4 to OPTIONS will have `bind9` not listen on IPv6 addresses
 #   so that we're sure there's no conflict with nsd, our public domain
 #   name server, on IPV6.
-# * The listen-on directive in named.conf.options restricts bind9 to
+# * The listen-on directive in named.conf.options restricts `bind9` to
 #   binding to the loopback interface instead of all interfaces.
 apt_install bind9 resolvconf
 tools/editconf.py /etc/default/bind9 \
@@ -65,9 +96,11 @@ if ! grep -q "listen-on " /etc/bind/named.conf.options; then
 	sed -i "s/^}/\n\tlisten-on { 127.0.0.1; };\n}/" /etc/bind/named.conf.options
 fi
 if [ -f /etc/resolvconf/resolv.conf.d/original ]; then
-	echo "Archiving old resolv.conf (was /etc/resolvconf/resolv.conf.d/original, now /etc/resolvconf/resolv.conf.original)."
-	mv /etc/resolvconf/resolv.conf.d/original /etc/resolvconf/resolv.conf.original
+	echo "Archiving old resolv.conf (was /etc/resolvconf/resolv.conf.d/original, now /etc/resolvconf/resolv.conf.original)." #NODOC
+	mv /etc/resolvconf/resolv.conf.d/original /etc/resolvconf/resolv.conf.original #NODOC
 fi
 
+# Restart the DNS services.
+
 restart_service bind9
 restart_service resolvconf

setup/web.sh

@@ -18,6 +18,10 @@ cp conf/nginx-ssl.conf /etc/nginx/nginx-ssl.conf
 tools/editconf.py /etc/nginx/nginx.conf -s \
 	server_names_hash_bucket_size="64;"
 
+# Bump up max_children to support more concurrent connections
+tools/editconf.py /etc/php5/fpm/pool.d/www.conf -c ';' \
+	pm.max_children=8
+
 # Other nginx settings will be configured by the management service
 # since it depends on what domains we're serving, which we don't know
 # until mail accounts have been created.
@@ -48,7 +52,7 @@ done
 # Remove obsoleted scripts.
 # exchange-autodiscover is now handled by Z-Push.
 for f in exchange-autodiscover; do
-	rm /usr/local/bin/mailinabox-$f.php
+	rm -f /usr/local/bin/mailinabox-$f.php
 done
 
 # Make some space for users to customize their webfinger responses.

setup/webmail.sh

@@ -1,20 +1,23 @@
-# Webmail: Using roundcube
-##########################
+# Webmail with Roundcube
+# ----------------------
 
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
-# Ubuntu's roundcube-core has dependencies on Apache & MySQL, which we don't want, so we can't
-# install roundcube directly via apt-get install.
+# ### Installing Roundcube
+
+# We install Roundcube from sources, rather than from Ubuntu, because:
 #
-# Additionally, the Roundcube shipped with Ubuntu is consistently out of date.
+# 1. Ubuntu's `roundcube-core` package has dependencies on Apache & MySQL, which we don't want.
 #
-# And it's packaged incorrectly --- it seems to be missing a directory of files.
+# 2. The Roundcube shipped with Ubuntu is consistently out of date.
+#
+# 3. It's packaged incorrectly --- it seems to be missing a directory of files.
 #
 # So we'll use apt-get to manually install the dependencies of roundcube that we know we need,
 # and then we'll manually install roundcube from source.
 
-# These dependencies are from 'apt-cache showpkg roundcube-core'.
+# These dependencies are from `apt-cache showpkg roundcube-core`.
 apt_install \
 	dbconfig-common \
 	php5 php5-sqlite php5-mcrypt php5-intl php5-json php5-common php-auth php-net-smtp php-net-socket php-net-sieve php-mail-mime php-crypt-gpg php5-gd php5-pspell \
@@ -25,25 +28,37 @@ apt_install \
 # Now that we're beyond that, get rid of those debs before installing from source.
 apt-get purge -qq -y roundcube*
 
-# Install Roundcube from source if it is not already present.
-# TODO: Check version?
-if [ ! -d /usr/local/lib/roundcubemail ]; then
+# Install Roundcube from source if it is not already present or if it is out of date.
+VERSION=1.0.2
+needs_update=0 #NODOC
+if [ ! -f /usr/local/lib/roundcubemail/version ]; then
+	# not installed yet
+	needs_update=1 #NODOC
+elif [[ $VERSION != `cat /usr/local/lib/roundcubemail/version` ]]; then
+	# checks if the version is what we want
+	needs_update=1 #NODOC
+fi
+if [ $needs_update == 1 ]; then
+	echo installing roudcube webmail $VERSION...
 	rm -f /tmp/roundcube.tgz
-	wget -qO /tmp/roundcube.tgz http://downloads.sourceforge.net/project/roundcubemail/roundcubemail/1.0.1/roundcubemail-1.0.1.tar.gz
+	wget -qO /tmp/roundcube.tgz http://downloads.sourceforge.net/project/roundcubemail/roundcubemail/1.0.2/roundcubemail-$VERSION.tar.gz
 	tar -C /usr/local/lib -zxf /tmp/roundcube.tgz
-	mv /usr/local/lib/roundcubemail-1.0.1/ /usr/local/lib/roundcubemail
+	mv /usr/local/lib/roundcubemail-$VERSION/ /usr/local/lib/roundcubemail
 	rm -f /tmp/roundcube.tgz
+	echo $VERSION > /usr/local/lib/roundcubemail/version
 fi
 
+# ### Configuring Roundcube
+
 # Generate a safe 24-character secret key of safe characters.
-SECRET_KEY=$(dd if=/dev/random bs=20 count=1 2>/dev/null | base64 | fold -w 24 | head -n 1)
+SECRET_KEY=$(dd if=/dev/random bs=1 count=18 2>/dev/null | base64 | fold -w 24 | head -n 1)
 
 # Create a configuration file.
 #
 # For security, temp and log files are not stored in the default locations
 # which are inside the roundcube sources directory. We put them instead
 # in normal places.
-cat - > /usr/local/lib/roundcubemail/config/config.inc.php <<EOF;
+cat > /usr/local/lib/roundcubemail/config/config.inc.php <<EOF;
 <?php
 /*
  * Do not edit. Written by Mail-in-a-Box. Regenerated on updates.

setup/zpush.sh

@@ -1,6 +1,8 @@
 #!/bin/bash
 #
-# Z-Push: The Microsoft Exchange protocol server.
+# Z-Push: The Microsoft Exchange protocol server
+# ----------------------------------------------
+#
 # Mostly for use on iOS which doesn't support IMAP.
 #
 # Although Ubuntu ships Z-Push (as d-push) it has a dependency on Apache
@@ -19,30 +21,33 @@ apt_install \
 php5enmod imap
 
 # Copy Z-Push into place.
-needs_update=0
+TARGETHASH=d0cd5a47c53afac5c3b287006dc8a48a1c4ffcd5
+needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/z-push/version ]; then
-	needs_update=1
-elif [[ `curl -s https://api.github.com/repos/fmbiete/Z-Push-contrib/git/refs/heads/master` != `cat /usr/local/lib/z-push/version` ]]; then
+	needs_update=1 #NODOC
+elif [[ $TARGETHASH != `cat /usr/local/lib/z-push/version` ]]; then
 	# checks if the version 
-	needs_update=1
+	needs_update=1 #NODOC
 fi
 if [ $needs_update == 1 ]; then
 	rm -rf /usr/local/lib/z-push
-	rm -f /tmp/zpush.zip
+	rm -f /tmp/zpush-repo
 	echo installing z-push \(fmbiete fork\)...
-	wget -qO /tmp/zpush.zip https://github.com/fmbiete/Z-Push-contrib/archive/master.zip
-	unzip -q /tmp/zpush.zip -d /usr/local/lib/
-	mv /usr/local/lib/Z-Push-contrib-master /usr/local/lib/z-push
+	git clone -q https://github.com/fmbiete/Z-Push-contrib /tmp/zpush-repo
+	(cd /tmp/zpush-repo/; git checkout -q $TARGETHASH;)
+	rm -rf /tmp/zpush-repo/.git
+	mv /tmp/zpush-repo /usr/local/lib/z-push
 	rm -f /usr/sbin/z-push-{admin,top}
 	ln -s /usr/local/lib/z-push/z-push-admin.php /usr/sbin/z-push-admin
 	ln -s /usr/local/lib/z-push/z-push-top.php /usr/sbin/z-push-top
-	rm /tmp/zpush.zip;
-	curl -s https://api.github.com/repos/fmbiete/Z-Push-contrib/git/refs/heads/master > /usr/local/lib/z-push/version
+	rm -f /tmp/zpush-repo
+	echo $TARGETHASH > /usr/local/lib/z-push/version
 fi
 
 # Configure default config.
-sed -i "s/define('TIMEZONE', .*/define('TIMEZONE', 'Etc\/UTC');/" /usr/local/lib/z-push/config.php
+sed -i "s^define('TIMEZONE', .*^define('TIMEZONE', '$(cat /etc/timezone)');^" /usr/local/lib/z-push/config.php
 sed -i "s/define('BACKEND_PROVIDER', .*/define('BACKEND_PROVIDER', 'BackendCombined');/" /usr/local/lib/z-push/config.php
+sed -i "s/define('USE_FULLEMAIL_FOR_LOGIN', .*/define('USE_FULLEMAIL_FOR_LOGIN', true);/" /usr/local/lib/z-push/config.php
 
 # Configure BACKEND
 rm -f /usr/local/lib/z-push/backend/combined/config.php

tools/mail.py

@@ -58,6 +58,7 @@ if len(sys.argv) < 2:
 	print("  tools/mail.py user admins (lists admins)")
 	print("  tools/mail.py alias  (lists aliases)")
 	print("  tools/mail.py alias add incoming.name@domain.com sent.to@other.domain.com")
+	print("  tools/mail.py alias add incoming.name@domain.com 'sent.to@other.domain.com, multiple.people@other.domain.com'")
 	print("  tools/mail.py alias remove incoming.name@domain.com")
 	print()
 	print("Removing a mail user does not delete their mail folders on disk. It only prevents IMAP/SMTP login.")
@@ -67,6 +68,7 @@ elif sys.argv[1] == "user" and len(sys.argv) == 2:
 	# Dump a list of users, one per line. Mark admins with an asterisk.
 	users = mgmt("/mail/users?format=json", is_json=True)
 	for user in users:
+		if user['status'] == 'inactive': continue
 		print(user['email'], end='')
 		if "admin" in user['privileges']:
 			print("*", end='')

tools/owncloud-unlockadmin.sh

@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# This script will give you administrative access to the ownCloud
+# instance running here.
+#
+# Run this at your own risk. This is for testing & experimentation
+# purpopses only. After this point you are on your own.
+
+source /etc/mailinabox.conf # load global vars
+
+ADMIN=$(./mail.py user admins | head -n 1)
+test -z "$1" || ADMIN=$1 
+
+echo I am going to unlock admin features for $ADMIN.
+echo You can provide another user to unlock as the first argument of this script.
+echo 
+echo WARNING: you could break mail-in-a-box when fiddling around with owncloud\'s admin interface
+echo If in doubt, press CTRL-C to cancel.
+echo 
+echo Press enter to continue.
+read
+
+sqlite3 $STORAGE_ROOT/owncloud/owncloud.db "INSERT OR IGNORE INTO oc_group_user VALUES ('admin', '$ADMIN')" && echo Done.

tools/readable_bash.py

@@ -0,0 +1,266 @@
+#!/usr/bin/python3
+#
+# Generate documentation for how this machine works by
+# parsing our bash scripts!
+
+import cgi, re
+import markdown
+from modgrammar import *
+
+def generate_documentation():
+	print("""<!DOCTYPE html>
+<html>
+    <head>
+        <meta charset="utf-8">
+        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
+        <meta name="viewport" content="width=device-width">
+
+        <title>Build Your Own Mail Server From Scratch</title>
+
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css">
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap-theme.min.css">
+
+        <style>
+		    @import url(https://fonts.googleapis.com/css?family=Iceland);
+		    @import url(https://fonts.googleapis.com/css?family=Raleway:400,700);
+			@import url(https://fonts.googleapis.com/css?family=Ubuntu:300,500);
+		   	body {
+		    		font-family: Raleway, sans-serif;
+		    		font-size: 16px;
+					color: #555;
+	    	}
+	    	h2, h3 {
+	    		margin-bottom: 1em;
+	    	}
+	    	p {
+	    		margin-bottom: 1em;
+	    	}
+
+	    	pre {
+	    		margin: 1em 1em 1.5em 1em;
+	    		color: black;
+	    	}
+
+	    	div.write-to {
+	    		margin: 1em;
+	    		border: 1px solid #999;
+	    	}
+	    	div.write-to p {
+	    		padding: .5em;
+	    		margin: 0;
+	    	}
+	    	div.write-to .filename {
+	    		background-color: #EEE;
+	    		padding: .5em;
+	    		font-weight: bold;
+	    	}
+	    	div.write-to pre {
+	    		padding: .5em;
+	    		margin: 0;
+	    	}
+
+	    	pre.shell > div:before {
+	    		content: "$ ";
+	    		color: #666;
+	    	}
+        </style>
+    </head>
+    <body>
+    <div class="container">
+      <div class="row">
+        <div class="col-xs-12">
+        <h1>Build Your Own Mail Server From Scratch</h1>
+        <p>Here&rsquo;s how you can build your own mail server from scratch. This document is generated automatically from our setup script.</p>
+        <hr>
+ """)
+
+	parser = Source.parser()
+	for line in open("setup/start.sh"):
+		try:
+			fn = parser.parse_string(line).filename()
+		except:
+			continue
+		if fn in ("setup/preflight.sh", "setup/questions.sh", "setup/firstuser.sh", "setup/management.sh"):
+			continue
+
+		import sys
+		print(fn, file=sys.stderr)
+
+		print(BashScript.parse(fn))
+
+	print("""
+        <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.1/jquery.min.js"></script>
+        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>
+    </body>
+</html>
+""")
+
+class HashBang(Grammar):
+	grammar = (L('#!'), REST_OF_LINE, EOL)
+	def value(self):
+		return ""
+
+def strip_indent(s):
+	lines = s.split("\n")
+	min_indent = min(len(re.match(r"\s*", line).group(0)) for line in lines if len(line) > 0)
+	lines = [line[min_indent:] for line in lines]
+	return "\n".join(lines)
+
+class Comment(Grammar):
+	grammar = ONE_OR_MORE(ZERO_OR_MORE(SPACE), L('#'), REST_OF_LINE, EOL)
+	def value(self):
+		if self.string.replace("#", "").strip() == "":
+			return "\n"
+		lines = [x[2].string for x in self[0]]
+		content = "\n".join(lines)
+		content = strip_indent(content)
+		return markdown.markdown(content, output_format="html4") + "\n\n"
+
+FILENAME = WORD('a-z0-9-/.')
+
+class Source(Grammar):
+	grammar = ((L('.') | L('source')), L(' '), FILENAME, Comment | EOL)
+	def filename(self):
+		return self[2].string.strip()
+	def value(self):
+		return BashScript.parse(self.filename())
+
+class CatEOF(Grammar):
+	grammar = (ZERO_OR_MORE(SPACE), L('cat > '), ANY_EXCEPT(WHITESPACE), L(" <<"), OPTIONAL(SPACE), L("EOF;"), EOL, REPEAT(ANY, greedy=False), EOL, L("EOF"), EOL)
+	def value(self):
+		content = self[7].string
+		content = re.sub(r"\\([$])", r"\1", content) # un-escape bash-escaped characters
+		return "<div class='write-to'><div class='filename'>overwrite<br>" + self[2].string + "</div><pre>" + cgi.escape(content) + "</pre></div>\n"
+
+class HideOutput(Grammar):
+	grammar = (L("hide_output "), REF("BashElement"))
+	def value(self):
+		return self[1].value()
+
+class SuppressedLine(Grammar):
+	grammar = (OPTIONAL(SPACE), L("echo "), REST_OF_LINE, EOL)
+	def value(self):
+		if "|" in self.string  or ">" in self.string:
+			return "<pre class='shell'><div>" + cgi.escape(self.string.strip()) + "</div></pre>\n"
+		return ""
+
+class EditConf(Grammar):
+	grammar = (
+		L('tools/editconf.py '),
+		FILENAME,
+		SPACE,
+		OPTIONAL((LIST_OF(
+			L("-w") | L("-s") | L("-c ';'"),
+			sep=SPACE,
+		), SPACE)),
+		REST_OF_LINE,
+		OPTIONAL(SPACE),
+		EOL
+		)
+	def value(self):
+		conffile = self[1]
+		options = [""]
+		mode = 1
+		for c in self[4].string:
+			if mode == 1 and c in (" ", "\t") and options[-1] != "":
+				# new word
+				options.append("")
+			elif mode < 0:
+				# escaped character
+				options[-1] += c
+				mode = -mode
+			elif c == "\\":
+				# escape next character
+				mode = -mode
+			elif mode == 1 and c == '"':
+				mode = 2
+			elif mode == 2 and c == '"':
+				mode = 1
+			else:
+				options[-1] += c
+		if options[-1] == "": options.pop(-1)
+		return "<div class='write-to'><div class='filename'>additional settings for<br>" + self[1].string + "</div><pre>" + "\n".join(cgi.escape(s) for s in options) + "</pre></div>\n"
+
+class CaptureOutput(Grammar):
+	grammar = OPTIONAL(SPACE), WORD("A-Za-z_"), L('=$('), REST_OF_LINE, L(")"), OPTIONAL(L(';')), EOL
+	def value(self):
+		cmd = self[3].string
+		cmd = cmd.replace("; ", "\n")
+		return "<div class='write-to'><div class='filename'>$" + self[1].string + "=</div><pre>" + cgi.escape(cmd) + "</pre></div>\n"
+
+class SedReplace(Grammar):
+	grammar = OPTIONAL(SPACE), L('sed -i "s/'), OPTIONAL(L('^')), ONE_OR_MORE(WORD("-A-Za-z0-9 #=\\{};.*$_!()")), L('/'), ONE_OR_MORE(WORD("-A-Za-z0-9 #=\\{};.*$_!()")), L('/"'), SPACE, FILENAME, EOL
+	def value(self):
+		return "<div class='write-to'><div class='filename'>edit<br>" + self[8].string + "</div><p>replace</p><pre>" + cgi.escape(self[3].string.replace(".*", ". . .")) + "</pre><p>with</p><pre>" + cgi.escape(self[5].string.replace("\\n", "\n").replace("\\t", "\t")) + "</pre></div>\n"
+
+def shell_line(bash):
+	return "<pre class='shell'><div>" + cgi.escape(wrap_lines(bash.strip())) + "</div></pre>\n"
+
+class AptGet(Grammar):
+	grammar = (ZERO_OR_MORE(SPACE), L("apt_install "), REST_OF_LINE, EOL)
+	def value(self):
+		return shell_line("apt-get install -y " + re.sub(r"\s+", " ", self[2].string))
+class UfwAllow(Grammar):
+	grammar = (ZERO_OR_MORE(SPACE), L("ufw_allow "), REST_OF_LINE, EOL)
+	def value(self):
+		return shell_line("ufw allow " + self[2].string)
+class RestartService(Grammar):
+	grammar = (ZERO_OR_MORE(SPACE), L("restart_service "), REST_OF_LINE, EOL)
+	def value(self):
+		return shell_line("service " + self[2].string + " restart")
+
+class OtherLine(Grammar):
+	grammar = (REST_OF_LINE, EOL)
+	def value(self):
+		if self.string.strip() == "": return ""
+		return "<pre class='shell'><div>" + cgi.escape(self.string.rstrip()) + "</div></pre>\n"
+
+class BashElement(Grammar):
+	grammar = Comment | Source | CatEOF | SuppressedLine | HideOutput | EditConf | CaptureOutput | SedReplace | AptGet | UfwAllow | RestartService | OtherLine
+	def value(self):
+		return self[0].value()
+
+class BashScript(Grammar):
+	grammar = (OPTIONAL(HashBang), REPEAT(BashElement))
+	def value(self):
+		return [line.value() for line in self[1]]
+
+	@staticmethod
+	def parse(fn):
+		if fn in ("setup/functions.sh", "/etc/mailinabox.conf"): return ""
+		parser = BashScript.parser()
+		string = open(fn).read()
+		string = re.sub(r"\s*\\\n\s*", " ", string)
+		string = re.sub(".* #NODOC\n", "", string)
+		string = re.sub("\n\s*if .*|\n\s*fi|\n\s*else|\n\s*elif .*", "", string)
+		string = re.sub("hide_output ", "", string)
+		result = parser.parse_string(string)
+	
+		v = "<div class='sourcefile'><a href=\"%s\">%s</a></div>\n" % ("https://github.com/mail-in-a-box/mailinabox/tree/master/" + fn, fn)
+		v += "".join(result.value())
+
+		v = v.replace("</pre>\n<pre class='shell'>", "")
+		v = re.sub("<pre>([\w\W]*?)</pre>", lambda m : "<pre>" + strip_indent(m.group(1)) + "</pre>", v)
+
+		v = re.sub(r"\$?PRIMARY_HOSTNAME", "<b>box.yourdomain.com</b>", v)
+		v = re.sub(r"\$?STORAGE_ROOT", "<code><b>/path/to/user-data</b></code>", v)
+		v = v.replace("`pwd`",  "<code><b>/path/to/mailinabox</b></code>")
+
+		return v
+
+def wrap_lines(text, cols=60):
+	ret = ""
+	words = re.split("(\s+)", text)
+	linelen = 0
+	for w in words:
+		if linelen + len(w) > cols-1:
+			ret += " \\\n"
+			ret += "   "
+			linelen = 0
+		if linelen == 0 and w.strip() == "": continue
+		ret += w
+		linelen += len(w)
+	return ret
+
+if __name__ == '__main__':
+	generate_documentation()