v0.09

=====

May 8, 2015

Mail:

* Spam checking is now performed on messages larger than the previous limit of 64KB.
* POP3S is now enabled (port 995).
* Roundcube is updated to version 1.1.1.
* Minor security improvements (more mail headers with user agent info are anonymized; crypto settings were tightened).

ownCloud:

* Downloading files you uploaded to ownCloud broke because of a change in ownCloud 8.

DNS:

* Internationalized Domain Names (IDNs) should now work in email. If you had custom DNS or custom web settings for internationalized domains, check that they are still working.
* It is now possible to set multiple TXT and other types of records on the same domain in the control panel.
* The custom DNS API was completely rewritten to support setting multiple records of the same type on a domain. Any existing client code using the DNS API will have to be rewritten. (Existing code will just get 404s back.)
* On some systems the `nsd` service failed to start if network inferfaces were not ready.

System / Control Panel:

* In order to guard against misconfiguration that can lead to domain control validation hijacking, email addresses that begin with admin, administrator, postmaster, hostmaster, and webmaster can no longer be used for (new) mail user accounts, and aliases for these addresses may direct mail only to the box's administrator(s).
* Backups now use duplicity's built-in gpg symmetric AES256 encryption rather than my home-brewed encryption. Old backups will be incorporated inside the first backup after this update but then deleted from disk (i.e. your backups from the previous few days will be backed up).
* There was a race condition between backups and the new nightly status checks.
* The control panel would sometimes lock up with an unnecessary loading indicator.
* You can no longer delete your own account from the control panel.

Setup:

* All Mail-in-a-Box release tags are now signed on github, instructions for verifying the signature are added to the README, and the integrity of some packages downloaded during setup is now verified against a SHA1 hash stored in the tag itself.
* Bugs in first user account creation were fixed.

CHANGELOG.md

@@ -1,6 +1,40 @@
 CHANGELOG
 =========
 
+v0.09 (May 8, 2015)
+-------------------
+
+Mail:
+
+* Spam checking is now performed on messages larger than the previous limit of 64KB.
+* POP3S is now enabled (port 995).
+* Roundcube is updated to version 1.1.1.
+* Minor security improvements (more mail headers with user agent info are anonymized; crypto settings were tightened).
+
+ownCloud:
+
+* Downloading files you uploaded to ownCloud broke because of a change in ownCloud 8.
+
+DNS:
+
+* Internationalized Domain Names (IDNs) should now work in email. If you had custom DNS or custom web settings for internationalized domains, check that they are still working.
+* It is now possible to set multiple TXT and other types of records on the same domain in the control panel.
+* The custom DNS API was completely rewritten to support setting multiple records of the same type on a domain. Any existing client code using the DNS API will have to be rewritten. (Existing code will just get 404s back.)
+* On some systems the `nsd` service failed to start if network inferfaces were not ready.
+
+System / Control Panel:
+
+* In order to guard against misconfiguration that can lead to domain control validation hijacking, email addresses that begin with admin, administrator, postmaster, hostmaster, and webmaster can no longer be used for (new) mail user accounts, and aliases for these addresses may direct mail only to the box's administrator(s).
+* Backups now use duplicity's built-in gpg symmetric AES256 encryption rather than my home-brewed encryption. Old backups will be incorporated inside the first backup after this update but then deleted from disk (i.e. your backups from the previous few days will be backed up).
+* There was a race condition between backups and the new nightly status checks.
+* The control panel would sometimes lock up with an unnecessary loading indicator.
+* You can no longer delete your own account from the control panel.
+
+Setup:
+
+* All Mail-in-a-Box release tags are now signed on github, instructions for verifying the signature are added to the README, and the integrity of some packages downloaded during setup is now verified against a SHA1 hash stored in the tag itself.
+* Bugs in first user account creation were fixed.
+
 v0.08 (April 1, 2015)
 ---------------------
 

README.md

@@ -28,6 +28,29 @@ The Box
 
 Mail-in-a-Box turns a fresh Ubuntu 14.04 LTS 64-bit machine into a working mail server, including SMTP ([postfix](http://www.postfix.org/)), IMAP ([dovecot](http://dovecot.org/)), Exchange ActiveSync ([z-push](https://github.com/fmbiete/Z-Push-contrib)), webmail ([Roundcube](http://roundcube.net/)), spam filtering ([spamassassin](https://spamassassin.apache.org/)), greylisting ([postgrey](http://postgrey.schweikert.ch/)), CardDAV/CalDAV ([ownCloud](http://owncloud.org/)), DNS, [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework), DKIM ([OpenDKIM](http://www.opendkim.org/)), [DMARC](https://en.wikipedia.org/wiki/DMARC), [DNSSEC](https://en.wikipedia.org/wiki/DNSSEC), [DANE TLSA](https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities), [SSHFP](https://tools.ietf.org/html/rfc4255), and basic system services like a firewall, intrusion protection, and setting the system clock.
 
+Authenticity
+------------
+
+I sign the release tags. To verify that a tag is signed by me, you can perform the following steps:
+
+	# Download my PGP key.
+	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
+	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported
+
+	# Clone this repository.
+	$ git clone https://github.com/mail-in-a-box/mailinabox
+	$ cd mailinabox
+
+	# Verify the tag.
+	$ git verify-tag v0.08
+	gpg: Signature made ..... using RSA key ID C10BDD81
+	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
+	gpg: WARNING: This key is not certified with a trusted signature!
+	gpg:          There is no indication that the signature belongs to the owner.
+	Primary key fingerprint: 5F4C 0E73 13CC D744 693B  2AEA B920 41F4 C10B DD81
+
+The key ID and fingerprint above should match my [Keybase.io key](https://keybase.io/joshdata) and the fingerprint I publish on [my homepage](https://razor.occams.info/).
+
 The Acknowledgements
 --------------------
 

conf/nginx-primaryonly.conf

@@ -28,6 +28,7 @@
 		fastcgi_param SCRIPT_FILENAME /usr/local/lib/owncloud/$2;
 		fastcgi_param SCRIPT_NAME $1$2;
 		fastcgi_param PATH_INFO $3;
+		# TODO: see the dispreferred "method 2" for xaccel at https://doc.owncloud.org/server/8.1/admin_manual/configuration_files/serving_static_files_configuration.html
 		fastcgi_param MOD_X_ACCEL_REDIRECT_ENABLED on;
 		fastcgi_read_timeout 630;
 		fastcgi_pass php-fpm;
@@ -36,10 +37,13 @@
 		client_max_body_size 1G;
 		fastcgi_buffers 64 4K;
 	}
-	location ^~ /cloud/data {
-		# In order to support MOD_X_ACCEL_REDIRECT_ENABLED, we need to expose
-		# the data directory but only allow 'internal' redirects within nginx
-		# so that this is not exposed to the world.
+	location ^~ $STORAGE_ROOT/owncloud {
+		# This directory is for MOD_X_ACCEL_REDIRECT_ENABLED. It works a little weird.
+		# The full path on disk of the file is passed as the URL path. ownCloud 8 totally
+		# busted the sane way this worked in ownCloud 7. There's a pending change using
+		# a new parameter to make this make more sense.
+		# We need to only allow 'internal' redirects within nginx so that the filesystem
+		# is not exposed to the world.
 		internal;
 		alias $STORAGE_ROOT/owncloud;
 	}

conf/postfix_outgoing_mail_header_filters

@@ -8,3 +8,7 @@
 /^\s*X-Enigmail:/        IGNORE
 /^\s*X-Mailer:/          IGNORE
 /^\s*X-Originating-IP:/  IGNORE
+/^\s*X-Pgp-Agent:/       IGNORE
+
+# The Mime-Version header can leak the user agent too, e.g. in Mime-Version: 1.0 (Mac OS X Mail 8.1 \(2010.6\)).
+/^\s*(Mime-Version:\s*[0-9\.]+)\s.+/  REPLACE $1

management/auth.py

@@ -111,12 +111,12 @@ class KeyAuthService:
 				# Login failed.
 				raise ValueError("Invalid password.")
 
-		# Get privileges for authorization.
-
-		# (This call should never fail on a valid user. But if it did fail, it would
-		# return a tuple of an error message and an HTTP status code.)
+		# Get privileges for authorization. This call should never fail on a valid user,
+		# but if the caller passed a user-specific API key then the user may no longer
+		# exist --- in that case, get_mail_user_privileges will return a tuple of an
+		# error message and an HTTP status code.
 		privs = get_mail_user_privileges(email, env)
-		if isinstance(privs, tuple): raise Exception("Error getting privileges.")
+		if isinstance(privs, tuple): raise ValueError(privs[0])
 
 		# Return a list of privileges.
 		return privs

management/backup.py

@@ -2,25 +2,24 @@
 
 # This script performs a backup of all user data:
 # 1) System services are stopped while a copy of user data is made.
-# 2) An incremental backup is made using duplicity into the
-#    directory STORAGE_ROOT/backup/duplicity.
+# 2) An incremental encrypted backup is made using duplicity into the
+#    directory STORAGE_ROOT/backup/encrypted. The password used for
+#    encryption is stored in backup/secret_key.txt.
 # 3) The stopped services are restarted.
-# 4) The backup files are encrypted with a long password (stored in
-#    backup/secret_key.txt) to STORAGE_ROOT/backup/encrypted.
 # 5) STORAGE_ROOT/backup/after-backup is executd if it exists.
 
 import os, os.path, shutil, glob, re, datetime
 import dateutil.parser, dateutil.relativedelta, dateutil.tz
 
-from utils import exclusive_process, load_environment, shell
+from utils import exclusive_process, load_environment, shell, wait_for_service
 
-# destroy backups when the most recent increment in the chain
+# Destroy backups when the most recent increment in the chain
 # that depends on it is this many days old.
 keep_backups_for_days = 3
 
 def backup_status(env):
 	# What is the current status of backups?
-	# Loop through all of the files in STORAGE_ROOT/backup/duplicity to
+	# Loop through all of the files in STORAGE_ROOT/backup/encrypted to
 	# get a list of all of the backups taken and sum up file sizes to
 	# see how large the storage is.
 
@@ -36,10 +35,10 @@ def backup_status(env):
 		return "%d hours, %d minutes" % (rd.hours, rd.minutes)
 
 	backups = { }
-	basedir = os.path.join(env['STORAGE_ROOT'], 'backup/duplicity/')
-	encdir = os.path.join(env['STORAGE_ROOT'], 'backup/encrypted/')
-	os.makedirs(basedir, exist_ok=True) # os.listdir fails if directory does not exist
-	for fn in os.listdir(basedir):
+	backup_root = os.path.join(env["STORAGE_ROOT"], 'backup')
+	backup_dir = os.path.join(backup_root, 'encrypted')
+	os.makedirs(backup_dir, exist_ok=True) # os.listdir fails if directory does not exist
+	for fn in os.listdir(backup_dir):
 		m = re.match(r"duplicity-(full|full-signatures|(inc|new-signatures)\.(?P<incbase>\d+T\d+Z)\.to)\.(?P<date>\d+T\d+Z)\.", fn)
 		if not m: raise ValueError(fn)
 
@@ -53,15 +52,9 @@ def backup_status(env):
 				"full": m.group("incbase") is None,
 				"previous": m.group("incbase"),
 				"size": 0,
-				"encsize": 0,
 			}
 
-		backups[key]["size"] += os.path.getsize(os.path.join(basedir, fn))
-
-		# Also check encrypted size.
-		encfn = os.path.join(encdir, fn + ".enc")
-		if os.path.exists(encfn):
-			backups[key]["encsize"] += os.path.getsize(encfn)
+		backups[key]["size"] += os.path.getsize(os.path.join(backup_dir, fn))
 
 	# Ensure the rows are sorted reverse chronologically.
 	# This is relied on by should_force_full() and the next step.
@@ -78,7 +71,7 @@ def backup_status(env):
 			break
 		incremental_count += 1
 		incremental_size += bak["size"]
-	
+
 	# Predict how many more increments until the next full backup,
 	# and add to that the time we hold onto backups, to predict
 	# how long the most recent full backup+increments will be held
@@ -106,9 +99,8 @@ def backup_status(env):
 			bak["deleted_in"] = deleted_in
 
 	return {
-		"directory": basedir,
-		"encpwfile": os.path.join(env['STORAGE_ROOT'], 'backup/secret_key.txt'),
-		"encdirectory": encdir,
+		"directory": backup_dir,
+		"encpwfile": os.path.join(backup_root, 'secret_key.txt'),
 		"tz": now.tzname(),
 		"backups": backups,
 	}
@@ -137,10 +129,35 @@ def perform_backup(full_backup):
 
 	exclusive_process("backup")
 
-	# Ensure the backup directory exists.
-	backup_dir = os.path.join(env["STORAGE_ROOT"], 'backup')
-	backup_duplicity_dir = os.path.join(backup_dir, 'duplicity')
-	os.makedirs(backup_duplicity_dir, exist_ok=True)
+	backup_root = os.path.join(env["STORAGE_ROOT"], 'backup')
+	backup_cache_dir = os.path.join(backup_root, 'cache')
+	backup_dir = os.path.join(backup_root, 'encrypted')
+
+	# In an older version of this script, duplicity was called
+	# such that it did not encrypt the backups it created (in
+	# backup/duplicity), and instead openssl was called separately
+	# after each backup run, creating AES256 encrypted copies of
+	# each file created by duplicity in backup/encrypted.
+	#
+	# We detect the transition by the presence of backup/duplicity
+	# and handle it by 'dupliception': we move all the old *un*encrypted
+	# duplicity files up out of the backup/duplicity directory (as
+	# backup/ is excluded from duplicity runs) in order that it is
+	# included in the next run, and we delete backup/encrypted (which
+	# duplicity will output files directly to, post-transition).
+	old_backup_dir = os.path.join(backup_root, 'duplicity')
+	migrated_unencrypted_backup_dir = os.path.join(env["STORAGE_ROOT"], "migrated_unencrypted_backup")
+	if os.path.isdir(old_backup_dir):
+		# Move the old unencrpyted files to a new location outside of
+		# the backup root so they get included in the next (new) backup.
+		# Then we'll delete them. Also so that they do not get in the
+		# way of duplicity doing a full backup on the first run after
+		# we take care of this.
+		shutil.move(old_backup_dir, migrated_unencrypted_backup_dir)
+
+		# The backup_dir (backup/encrypted) now has a new purpose.
+		# Clear it out.
+		shutil.rmtree(backup_dir)
 
 	# On the first run, always do a full backup. Incremental
 	# will fail. Otherwise do a full backup when the size of
@@ -152,81 +169,111 @@ def perform_backup(full_backup):
 	shell('check_call', ["/usr/sbin/service", "dovecot", "stop"])
 	shell('check_call', ["/usr/sbin/service", "postfix", "stop"])
 
+	# Get the encryption passphrase. secret_key.txt is 2048 random
+	# bits base64-encoded and with line breaks every 65 characters.
+	# gpg will only take the first line of text, so sanity check that
+	# that line is long enough to be a reasonable passphrase. It
+	# only needs to be 43 base64-characters to match AES256's key
+	# length of 32 bytes.
+	with open(os.path.join(backup_root, 'secret_key.txt')) as f:
+		passphrase = f.readline().strip()
+	if len(passphrase) < 43: raise Exception("secret_key.txt's first line is too short!")
+	env_with_passphrase = { "PASSPHRASE" : passphrase }
+
 	# Update the backup mirror directory which mirrors the current
 	# STORAGE_ROOT (but excluding the backups themselves!).
 	try:
 		shell('check_call', [
 			"/usr/bin/duplicity",
 			"full" if full_backup else "incr",
-			"--no-encryption",
-			"--archive-dir", "/tmp/duplicity-archive-dir",
-			"--name", "mailinabox",
-			"--exclude", backup_dir,
-			"--volsize", "100",
-			"--verbosity", "warning",
+			"--archive-dir", backup_cache_dir,
+			"--exclude", backup_root,
+			"--volsize", "250",
+			"--gpg-options", "--cipher-algo=AES256",
 			env["STORAGE_ROOT"],
-			"file://" + backup_duplicity_dir
-			])
+			"file://" + backup_dir
+			],
+			env_with_passphrase)
 	finally:
 		# Start services again.
 		shell('check_call', ["/usr/sbin/service", "dovecot", "start"])
 		shell('check_call', ["/usr/sbin/service", "postfix", "start"])
 
+	# Once the migrated backup is included in a new backup, it can be deleted.
+	if os.path.isdir(migrated_unencrypted_backup_dir):
+		shutil.rmtree(migrated_unencrypted_backup_dir)
+
 	# Remove old backups. This deletes all backup data no longer needed
-	# from more than 31 days ago. Must do this before destroying the
-	# cache directory or else this command will re-create it.
+	# from more than 3 days ago.
 	shell('check_call', [
 		"/usr/bin/duplicity",
 		"remove-older-than",
 		"%dD" % keep_backups_for_days,
-		"--archive-dir", "/tmp/duplicity-archive-dir",
-		"--name", "mailinabox",
+		"--archive-dir", backup_cache_dir,
 		"--force",
-		"--verbosity", "warning",
-		"file://" + backup_duplicity_dir
-		])
+		"file://" + backup_dir
+		],
+		env_with_passphrase)
 
-	# Remove duplicity's cache directory because it's redundant with our backup directory.
-	shutil.rmtree("/tmp/duplicity-archive-dir")
+	# From duplicity's manual:
+	# "This should only be necessary after a duplicity session fails or is
+	# aborted prematurely."
+	# That may be unlikely here but we may as well ensure we tidy up if
+	# that does happen - it might just have been a poorly timed reboot.
+	shell('check_call', [
+		"/usr/bin/duplicity",
+		"cleanup",
+		"--archive-dir", backup_cache_dir,
+		"--force",
+		"file://" + backup_dir
+		],
+		env_with_passphrase)
 
-	# Encrypt all of the new files.
-	backup_encrypted_dir = os.path.join(backup_dir, 'encrypted')
-	os.makedirs(backup_encrypted_dir, exist_ok=True)
-	for fn in os.listdir(backup_duplicity_dir):
-		fn2 = os.path.join(backup_encrypted_dir, fn) + ".enc"
-		if os.path.exists(fn2): continue
-
-		# Encrypt the backup using the backup private key.
-		shell('check_call', [
-			"/usr/bin/openssl",
-			"enc",
-			"-aes-256-cbc",
-			"-a",
-			"-salt",
-			"-in", os.path.join(backup_duplicity_dir, fn),
-			"-out", fn2,
-			"-pass", "file:%s" % os.path.join(backup_dir, "secret_key.txt"),
-			])
-
-		# The backup can be decrypted with:
-		# openssl enc -d -aes-256-cbc -a -in latest.tgz.enc -out /dev/stdout -pass file:secret_key.txt | tar -z
-
-	# Remove encrypted backups that are no longer needed.
-	for fn in os.listdir(backup_encrypted_dir):
-		fn2 = os.path.join(backup_duplicity_dir, fn.replace(".enc", ""))
-		if os.path.exists(fn2): continue
-		os.unlink(os.path.join(backup_encrypted_dir, fn))
+	# Change ownership of backups to the user-data user, so that the after-bcakup
+	# script can access them.
+	shell('check_call', ["/bin/chown", "-R", env["STORAGE_USER"], backup_dir])
 
 	# Execute a post-backup script that does the copying to a remote server.
 	# Run as the STORAGE_USER user, not as root. Pass our settings in
 	# environment variables so the script has access to STORAGE_ROOT.
-	post_script = os.path.join(backup_dir, 'after-backup')
+	post_script = os.path.join(backup_root, 'after-backup')
 	if os.path.exists(post_script):
 		shell('check_call',
 			['su', env['STORAGE_USER'], '-c', post_script],
 			env=env)
 
+	# Our nightly cron job executes system status checks immediately after this
+	# backup. Since it checks that dovecot and postfix are running, block for a
+	# bit (maximum of 10 seconds each) to give each a chance to finish restarting
+	# before the status checks might catch them down. See #381.
+	wait_for_service(25, True, env, 10)
+	wait_for_service(993, True, env, 10)
+
+def run_duplicity_verification():
+	env = load_environment()
+	backup_root = os.path.join(env["STORAGE_ROOT"], 'backup')
+	backup_cache_dir = os.path.join(backup_root, 'cache')
+	backup_dir = os.path.join(backup_root, 'encrypted')
+	env_with_passphrase = { "PASSPHRASE" : open(os.path.join(backup_root, 'secret_key.txt')).read() }
+	shell('check_call', [
+		"/usr/bin/duplicity",
+		"--verbosity", "info",
+		"verify",
+		"--compare-data",
+		"--archive-dir", backup_cache_dir,
+		"--exclude", backup_root,
+		"file://" + backup_dir,
+		env["STORAGE_ROOT"],
+	], env_with_passphrase)
+
 if __name__ == "__main__":
 	import sys
-	full_backup = "--full" in sys.argv
-	perform_backup(full_backup)
+	if sys.argv[-1] == "--verify":
+		# Run duplicity's verification command to check a) the backup files
+		# are readable, and b) report if they are up to date.
+		run_duplicity_verification()
+	else:
+		# Perform a backup. Add --full to force a full backup rather than
+		# possibly performing an incremental backup.
+		full_backup = "--full" in sys.argv
+		perform_backup(full_backup)

management/daemon.py

@@ -80,7 +80,7 @@ def unauthorized(error):
 	return auth_service.make_unauthorized_response()
 
 def json_response(data):
-	return Response(json.dumps(data), status=200, mimetype='application/json')
+	return Response(json.dumps(data, indent=2, sort_keys=True)+'\n', status=200, mimetype='application/json')
 
 ###################################
 
@@ -90,10 +90,12 @@ def json_response(data):
 def index():
 	# Render the control panel. This route does not require user authentication
 	# so it must be safe!
+	no_users_exist = (len(get_mail_users(env)) == 0)
 	no_admins_exist = (len(get_admins(env)) == 0)
 	return render_template('index.html',
 		hostname=env['PRIMARY_HOSTNAME'],
 		storage_root=env['STORAGE_ROOT'],
+		no_users_exist=no_users_exist,
 		no_admins_exist=no_admins_exist,
 	)
 
@@ -219,8 +221,8 @@ def dns_update():
 @app.route('/dns/secondary-nameserver')
 @authorized_personnel_only
 def dns_get_secondary_nameserver():
-	from dns_update import get_custom_dns_config
-	return json_response({ "hostname": get_custom_dns_config(env).get("_secondary_nameserver") })
+	from dns_update import get_custom_dns_config, get_secondary_dns
+	return json_response({ "hostname": get_secondary_dns(get_custom_dns_config(env)) })
 
 @app.route('/dns/secondary-nameserver', methods=['POST'])
 @authorized_personnel_only
@@ -231,38 +233,70 @@ def dns_set_secondary_nameserver():
 	except ValueError as e:
 		return (str(e), 400)
 
-@app.route('/dns/set')
+@app.route('/dns/custom')
 @authorized_personnel_only
-def dns_get_records():
-	from dns_update import get_custom_dns_config, get_custom_records
-	additional_records = get_custom_dns_config(env)
-	records = get_custom_records(None, additional_records, env)
-	return json_response([{
+def dns_get_records(qname=None, rtype=None):
+	from dns_update import get_custom_dns_config
+	return json_response([
+	{
 		"qname": r[0],
 		"rtype": r[1],
 		"value": r[2],
-		} for r in records])
+	}
+	for r in get_custom_dns_config(env)
+	if r[0] != "_secondary_nameserver"
+		and (not qname or r[0] == qname)
+		and (not rtype or r[1] == rtype) ])
 
-@app.route('/dns/set/<qname>', methods=['POST'])
-@app.route('/dns/set/<qname>/<rtype>', methods=['POST'])
-@app.route('/dns/set/<qname>/<rtype>/<value>', methods=['POST'])
+@app.route('/dns/custom/<qname>', methods=['GET', 'POST', 'PUT', 'DELETE'])
+@app.route('/dns/custom/<qname>/<rtype>', methods=['GET', 'POST', 'PUT', 'DELETE'])
 @authorized_personnel_only
-def dns_set_record(qname, rtype="A", value=None):
+def dns_set_record(qname, rtype="A"):
 	from dns_update import do_dns_update, set_custom_dns_record
 	try:
-		# Get the value from the URL, then the POST parameters, or if it is not set then
-		# use the remote IP address of the request --- makes dynamic DNS easy. To clear a
-		# value, '' must be explicitly passed.
-		if value is None:
-			value = request.form.get("value")
-		if value is None:
-			value = request.environ.get("HTTP_X_FORWARDED_FOR") # normally REMOTE_ADDR but we're behind nginx as a reverse proxy
-		if value == '' or value == '__delete__':
-			# request deletion
-			value = None
-		if set_custom_dns_record(qname, rtype, value, env):
-			return do_dns_update(env)
+		# Normalize.
+		rtype = rtype.upper()
+
+		# Read the record value from the request BODY, which must be
+		# ASCII-only. Not used with GET.
+		value = request.stream.read().decode("ascii", "ignore").strip()
+
+		if request.method == "GET":
+			# Get the existing records matching the qname and rtype.
+			return dns_get_records(qname, rtype)
+
+		elif request.method in ("POST", "PUT"):
+			# There is a default value for A/AAAA records.
+			if rtype in ("A", "AAAA") and value == "":
+				value = request.environ.get("HTTP_X_FORWARDED_FOR") # normally REMOTE_ADDR but we're behind nginx as a reverse proxy
+
+			# Cannot add empty records.
+			if value == '':
+				return ("No value for the record provided.", 400)
+
+			if request.method == "POST":
+				# Add a new record (in addition to any existing records
+				# for this qname-rtype pair).
+				action = "add"
+			elif request.method == "PUT":
+				# In REST, PUT is supposed to be idempotent, so we'll
+				# make this action set (replace all records for this
+				# qname-rtype pair) rather than add (add a new record).
+				action = "set"
+		
+		elif request.method == "DELETE":
+			if value == '':
+				# Delete all records for this qname-type pair.
+				value = None
+			else:
+				# Delete just the qname-rtype-value record exactly.
+				pass
+			action = "remove"
+
+		if set_custom_dns_record(qname, rtype, value, action, env):
+			return do_dns_update(env) or "Something isn't right."
 		return "OK"
+
 	except ValueError as e:
 		return (str(e), 400)
 

management/dns_update.py

@@ -4,7 +4,7 @@
 # and mail aliases and restarts nsd.
 ########################################################################
 
-import os, os.path, urllib.parse, datetime, re, hashlib, base64
+import sys, os, os.path, urllib.parse, datetime, re, hashlib, base64
 import ipaddress
 import rtyaml
 import dns.resolver
@@ -50,24 +50,13 @@ def get_dns_zones(env):
 
 	return zonefiles
 	
-def get_custom_dns_config(env):
-	try:
-		return rtyaml.load(open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml')))
-	except:
-		return { }
-
-def write_custom_dns_config(config, env):
-	config_yaml = rtyaml.dump(config)
-	with open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml'), "w") as f:
-		f.write(config_yaml)
-
 def do_dns_update(env, force=False):
 	# What domains (and their zone filenames) should we build?
 	domains = get_dns_domains(env)
 	zonefiles = get_dns_zones(env)
 
 	# Custom records to add to zones.
-	additional_records = get_custom_dns_config(env)
+	additional_records = list(get_custom_dns_config(env))
 
 	# Write zone files.
 	os.makedirs('/etc/nsd/zones', exist_ok=True)
@@ -153,7 +142,7 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 		records.append((None,  "NS",  "ns1.%s." % env["PRIMARY_HOSTNAME"], False))
 
 		# Define ns2.PRIMARY_HOSTNAME or whatever the user overrides.
-		secondary_ns = additional_records.get("_secondary_nameserver", "ns2." + env["PRIMARY_HOSTNAME"])
+		secondary_ns = get_secondary_dns(additional_records) or ("ns2." + env["PRIMARY_HOSTNAME"])
 		records.append((None,  "NS", secondary_ns+'.', False))
 
 
@@ -196,20 +185,34 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 				child_qname += "." + subdomain_qname
 			records.append((child_qname, child_rtype, child_value, child_explanation))
 
+	has_rec_base = list(records) # clone current state
 	def has_rec(qname, rtype, prefix=None):
-		for rec in records:
+		for rec in has_rec_base:
 			if rec[0] == qname and rec[1] == rtype and (prefix is None or rec[2].startswith(prefix)):
 				return True
 		return False
 
 	# The user may set other records that don't conflict with our settings.
 	# Don't put any TXT records above this line, or it'll prevent any custom TXT records.
-	for qname, rtype, value in get_custom_records(domain, additional_records, env):
+	for qname, rtype, value in filter_custom_records(domain, additional_records):
+		# Don't allow custom records for record types that override anything above.
+		# But allow multiple custom records for the same rtype --- see how has_rec_base is used.
 		if has_rec(qname, rtype): continue
+
+		# The "local" keyword on A/AAAA records are short-hand for our own IP.
+		# This also flags for web configuration that the user wants a website here.
+		if rtype == "A" and value == "local":
+			value = env["PUBLIC_IP"]
+		if rtype == "AAAA" and value == "local":
+			if "PUBLIC_IPV6" in env:
+				value = env["PUBLIC_IPV6"]
+			else:
+				continue
 		records.append((qname, rtype, value, "(Set by user.)"))
 
 	# Add defaults if not overridden by the user's custom settings (and not otherwise configured).
 	# Any "CNAME" record on the qname overrides A and AAAA.
+	has_rec_base = records
 	defaults = [
 		(None,  "A",    env["PUBLIC_IP"],       "Required. May have a different value. Sets the IP address that %s resolves to for web hosting and other services besides mail. The A record must be present but its value does not affect mail delivery." % domain),
 		("www", "A",    env["PUBLIC_IP"],       "Optional. Sets the IP address that www.%s resolves to, e.g. for web hosting." % domain),
@@ -263,52 +266,6 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 
 ########################################################################
 
-def get_custom_records(domain, additional_records, env):
-	for qname, value in additional_records.items():
-		# We don't count the secondary nameserver config (if present) as a record - that would just be
-		# confusing to users. Instead it is accessed/manipulated directly via (get/set)_custom_dns_config.
-		if qname == "_secondary_nameserver": continue
-
-		# Is this record for the domain or one of its subdomains?
-		# If `domain` is None, return records for all domains.
-		if domain is not None and qname != domain and not qname.endswith("." + domain): continue
-
-		# Turn the fully qualified domain name in the YAML file into
-		# our short form (None => domain, or a relative QNAME) if
-		# domain is not None.
-		if domain is not None:
-			if qname == domain:
-				qname = None
-			else:
-				qname = qname[0:len(qname)-len("." + domain)]
-
-		# Short form. Mapping a domain name to a string is short-hand
-		# for creating A records.
-		if isinstance(value, str):
-			values = [("A", value)]
-			if value == "local" and env.get("PUBLIC_IPV6"):
-				values.append( ("AAAA", value) )
-
-		# A mapping creates multiple records.
-		elif isinstance(value, dict):
-			values = value.items()
-
-		# No other type of data is allowed.
-		else:
-			raise ValueError()
-
-		for rtype, value2 in values:
-			# The "local" keyword on A/AAAA records are short-hand for our own IP.
-			# This also flags for web configuration that the user wants a website here.
-			if rtype == "A" and value2 == "local":
-				value2 = env["PUBLIC_IP"]
-			if rtype == "AAAA" and value2 == "local":
-				if "PUBLIC_IPV6" not in env: continue # no IPv6 address is available so don't set anything
-				value2 = env["PUBLIC_IPV6"]
-			yield (qname, rtype, value2)
-
-########################################################################
-
 def build_tlsa_record(env):
 	# A DANE TLSA record in DNS specifies that connections on a port
 	# must use TLS and the certificate must match a particular certificate.
@@ -397,26 +354,17 @@ $TTL 1800           ; default time to live
 """
 
 	# Replace replacement strings.
-	zone = zone.format(domain=domain.encode("idna").decode("ascii"), primary_domain=env["PRIMARY_HOSTNAME"].encode("idna").decode("ascii"))
+	zone = zone.format(domain=domain, primary_domain=env["PRIMARY_HOSTNAME"])
 
 	# Add records.
 	for subdomain, querytype, value, explanation in records:
 		if subdomain:
-			zone += subdomain.encode("idna").decode("ascii")
+			zone += subdomain
 		zone += "\tIN\t" + querytype + "\t"
 		if querytype == "TXT":
-			# Quote and escape.
 			value = value.replace('\\', '\\\\') # escape backslashes
 			value = value.replace('"', '\\"') # escape quotes
 			value = '"' + value + '"' # wrap in quotes
-		elif querytype in ("NS", "CNAME"):
-			# These records must be IDNA-encoded.
-			value = value.encode("idna").decode("ascii")
-		elif querytype == "MX":
-			# Also IDNA-encoded, but must parse first.
-			priority, host = value.split(" ", 1)
-			host = host.encode("idna").decode("ascii")
-			value = priority + " " + host
 		zone += value + "\n"
 
 	# DNSSEC requires re-signing a zone periodically. That requires
@@ -484,39 +432,23 @@ $TTL 1800           ; default time to live
 ########################################################################
 
 def write_nsd_conf(zonefiles, additional_records, env):
-	# Basic header.
-	nsdconf = """
-server:
-  hide-version: yes
-
-  # identify the server (CH TXT ID.SERVER entry).
-  identity: ""
-
-  # The directory for zonefile: files.
-  zonesdir: "/etc/nsd/zones"
-"""
+	# Write the list of zones to a configuration file.
+	nsd_conf_file = "/etc/nsd/zones.conf"
+	nsdconf = ""
 	
-	# Since we have bind9 listening on localhost for locally-generated
-	# DNS queries that require a recursive nameserver, and the system
-	# might have other network interfaces for e.g. tunnelling, we have
-	# to be specific about the network interfaces that nsd binds to.
-	for ipaddr in (env.get("PRIVATE_IP", "") + " " + env.get("PRIVATE_IPV6", "")).split(" "):
-		if ipaddr == "": continue
-		nsdconf += "  ip-address: %s\n" % ipaddr
-
 	# Append the zones.
 	for domain, zonefile in zonefiles:
 		nsdconf += """
 zone:
 	name: %s
 	zonefile: %s
-""" % (domain.encode("idna").decode("ascii"), zonefile)
+""" % (domain, zonefile)
 
 		# If a custom secondary nameserver has been set, allow zone transfers
 		# and notifies to that nameserver.
-		if additional_records.get("_secondary_nameserver"):
+		if get_secondary_dns(additional_records):
 			# Get the IP address of the nameserver by resolving it.
-			hostname = additional_records.get("_secondary_nameserver")
+			hostname = get_secondary_dns(additional_records)
 			resolver = dns.resolver.get_default_resolver()
 			response = dns.resolver.query(hostname+'.', "A")
 			ipaddr = str(response[0])
@@ -524,16 +456,17 @@ zone:
 	provide-xfr: %s NOKEY
 """ % (ipaddr, ipaddr)
 
-
-	# Check if the nsd.conf is changing. If it isn't changing,
+	# Check if the file is changing. If it isn't changing,
 	# return False to flag that no change was made.
-	with open("/etc/nsd/nsd.conf") as f:
-		if f.read() == nsdconf:
-			return False
+	if os.path.exists(nsd_conf_file):
+		with open(nsd_conf_file) as f:
+			if f.read() == nsdconf:
+				return False
 
-	with open("/etc/nsd/nsd.conf", "w") as f:
+	# Write out new contents and return True to signal that
+	# configuration changed.
+	with open(nsd_conf_file, "w") as f:
 		f.write(nsdconf)
-
 	return True
 
 ########################################################################
@@ -555,9 +488,6 @@ def sign_zone(domain, zonefile, env):
 	algo = dnssec_choose_algo(domain, env)
 	dnssec_keys = load_env_vars_from_file(os.path.join(env['STORAGE_ROOT'], 'dns/dnssec/%s.conf' % algo))
 
-	# From here, use the IDNA encoding of the domain name.
-	domain = domain.encode("idna").decode("ascii")
-
 	# In order to use the same keys for all domains, we have to generate
 	# a new .key file with a DNSSEC record for the specific domain. We
 	# can reuse the same key, but it won't validate without a DNSSEC
@@ -680,7 +610,94 @@ def write_opendkim_tables(domains, env):
 
 ########################################################################
 
-def set_custom_dns_record(qname, rtype, value, env):
+def get_custom_dns_config(env):
+	try:
+		custom_dns = rtyaml.load(open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml')))
+		if not isinstance(custom_dns, dict): raise ValueError() # caught below
+	except:
+		return [ ]
+
+	for qname, value in custom_dns.items():
+		# Short form. Mapping a domain name to a string is short-hand
+		# for creating A records.
+		if isinstance(value, str):
+			values = [("A", value)]
+
+		# A mapping creates multiple records.
+		elif isinstance(value, dict):
+			values = value.items()
+
+		# No other type of data is allowed.
+		else:
+			raise ValueError()
+
+		for rtype, value2 in values:
+			if isinstance(value2, str):
+				yield (qname, rtype, value2)
+			elif isinstance(value2, list):
+				for value3 in value2:
+					yield (qname, rtype, value3)
+			# No other type of data is allowed.
+			else:
+				raise ValueError()
+
+def filter_custom_records(domain, custom_dns_iter):
+	for qname, rtype, value in custom_dns_iter:
+		# We don't count the secondary nameserver config (if present) as a record - that would just be
+		# confusing to users. Instead it is accessed/manipulated directly via (get/set)_custom_dns_config.
+		if qname == "_secondary_nameserver": continue
+
+		# Is this record for the domain or one of its subdomains?
+		# If `domain` is None, return records for all domains.
+		if domain is not None and qname != domain and not qname.endswith("." + domain): continue
+
+		# Turn the fully qualified domain name in the YAML file into
+		# our short form (None => domain, or a relative QNAME) if
+		# domain is not None.
+		if domain is not None:
+			if qname == domain:
+				qname = None
+			else:
+				qname = qname[0:len(qname)-len("." + domain)]
+
+		yield (qname, rtype, value)
+
+def write_custom_dns_config(config, env):
+	# We get a list of (qname, rtype, value) triples. Convert this into a
+	# nice dictionary format for storage on disk.
+	from collections import OrderedDict
+	config = list(config)
+	dns = OrderedDict()
+	seen_qnames = set()
+
+	# Process the qnames in the order we see them.
+	for qname in [rec[0] for rec in config]:
+		if qname in seen_qnames: continue
+		seen_qnames.add(qname)
+
+		records = [(rec[1], rec[2]) for rec in config if rec[0] == qname]
+		if len(records) == 1 and records[0][0] == "A":
+			dns[qname] = records[0][1]
+		else:
+			dns[qname] = OrderedDict()
+			seen_rtypes = set()
+
+			# Process the rtypes in the order we see them.
+			for rtype in [rec[0] for rec in records]:
+				if rtype in seen_rtypes: continue
+				seen_rtypes.add(rtype)
+
+				values = [rec[1] for rec in records if rec[0] == rtype]
+				if len(values) == 1:
+					values = values[0]
+				dns[qname][rtype] = values
+	
+	# Write.	
+	config_yaml = rtyaml.dump(dns)
+	with open(os.path.join(env['STORAGE_ROOT'], 'dns/custom.yaml'), "w") as f:
+		f.write(config_yaml)
+
+def set_custom_dns_record(qname, rtype, value, action, env):
 	# validate qname
 	for zone, fn in get_dns_zones(env):
 		# It must match a zone apex or be a subdomain of a zone
@@ -689,15 +706,17 @@ def set_custom_dns_record(qname, rtype, value, env):
 			break
 	else:
 		# No match.
-		raise ValueError("%s is not a domain name or a subdomain of a domain name managed by this box." % qname)
+		if qname != "_secondary_nameserver":
+			raise ValueError("%s is not a domain name or a subdomain of a domain name managed by this box." % qname)
 
 	# validate rtype
 	rtype = rtype.upper()
-	if value is not None:
+	if value is not None and qname != "_secondary_nameserver":
 		if rtype in ("A", "AAAA"):
-			v = ipaddress.ip_address(value)
-			if rtype == "A" and not isinstance(v, ipaddress.IPv4Address): raise ValueError("That's an IPv6 address.")
-			if rtype == "AAAA" and not isinstance(v, ipaddress.IPv6Address): raise ValueError("That's an IPv4 address.")
+			if value != "local": # "local" is a special flag for us
+				v = ipaddress.ip_address(value) # raises a ValueError if there's a problem
+				if rtype == "A" and not isinstance(v, ipaddress.IPv4Address): raise ValueError("That's an IPv6 address.")
+				if rtype == "AAAA" and not isinstance(v, ipaddress.IPv6Address): raise ValueError("That's an IPv4 address.")
 		elif rtype in ("CNAME", "TXT", "SRV", "MX"):
 			# anything goes
 			pass
@@ -705,69 +724,65 @@ def set_custom_dns_record(qname, rtype, value, env):
 			raise ValueError("Unknown record type '%s'." % rtype)
 
 	# load existing config
-	config = get_custom_dns_config(env)
+	config = list(get_custom_dns_config(env))
 
 	# update
-	if qname not in config:
-		if value is None:
-			# Is asking to delete a record that does not exist.
-			return False
-		elif rtype == "A":
-			# Add this record using the short form 'qname: value'.
-			config[qname] = value
-		else:
-			# Add this record. This is the qname's first record.
-			config[qname] = { rtype: value }
-	else:
-		if isinstance(config[qname], str):
-			# This is a short-form 'qname: value' implicit-A record.
-			if value is None and rtype != "A":
-				# Is asking to delete a record that doesn't exist.
+	newconfig = []
+	made_change = False
+	needs_add = True
+	for _qname, _rtype, _value in config:
+		if action == "add":
+			if (_qname, _rtype, _value) == (qname, rtype, value):
+				# Record already exists. Bail.
 				return False
-			elif value is None and rtype == "A":
-				# Delete record.
-				del config[qname]
-			elif rtype == "A":
-				# Update, keeping short form.
-				if config[qname] == "value":
-					# No change.
-					return False
-				config[qname] = value
-			else:
-				# Expand short form so we can add a new record type.
-				config[qname] = { "A": config[qname], rtype: value }
-		else:
-			# This is the qname: { ... } (dict) format.
-			if value is None:
-				if rtype not in config[qname]:
-					# Is asking to delete a record that doesn't exist.
-					return False
+		elif action == "set":
+			if (_qname, _rtype) == (qname, rtype):
+				if _value == value:
+					# Flag that the record already exists, don't
+					# need to add it.
+					needs_add = False
 				else:
-					# Delete the record. If it's the last record, delete the domain.
-					del config[qname][rtype]
-					if len(config[qname]) == 0:
-						del config[qname]
-			else:
-				# Update the record.
-				if config[qname].get(rtype) == "value":
-					# No change.
-					return False
-				config[qname][rtype] = value
+					# Drop any other values for this (qname, rtype).
+					made_change = True
+					continue
+		elif action == "remove":
+			if (_qname, _rtype, _value) == (qname, rtype, value):
+				# Drop this record.
+				made_change = True
+				continue
+			if value == None and (_qname, _rtype) == (qname, rtype):
+				# Drop all qname-rtype records.
+				made_change = True
+				continue
+		else:
+			raise ValueError("Invalid action: " + action)
 
-	# serialize & save
-	write_custom_dns_config(config, env)
+		# Preserve this record.
+		newconfig.append((_qname, _rtype, _value))
 
-	return True
+	if action in ("add", "set") and needs_add and value is not None:
+		newconfig.append((qname, rtype, value))
+		made_change = True
+
+	if made_change:
+		# serialize & save
+		write_custom_dns_config(newconfig, env)
+
+	return made_change
 
 ########################################################################
 
+def get_secondary_dns(custom_dns):
+	for qname, rtype, value in custom_dns:
+		if qname == "_secondary_nameserver":
+			return value
+	return None
+
 def set_secondary_dns(hostname, env):
-	config = get_custom_dns_config(env)
 
 	if hostname in (None, ""):
 		# Clear.
-		if "_secondary_nameserver" in config:
-			del config["_secondary_nameserver"]
+		set_custom_dns_record("_secondary_nameserver", "A", None, "set", env)
 	else:
 		# Validate.
 		hostname = hostname.strip().lower()
@@ -778,10 +793,9 @@ def set_secondary_dns(hostname, env):
 			raise ValueError("Could not resolve the IP address of %s." % hostname)
 
 		# Set.
-		config["_secondary_nameserver"] = hostname
+		set_custom_dns_record("_secondary_nameserver", "A", hostname, "set", env)
 
-	# Save and apply.
-	write_custom_dns_config(config, env)
+	# Apply.
 	return do_dns_update(env)
 
 
@@ -832,7 +846,7 @@ def build_recommended_dns(env):
 	ret = []
 	domains = get_dns_domains(env)
 	zonefiles = get_dns_zones(env)
-	additional_records = get_custom_dns_config(env)
+	additional_records = list(get_custom_dns_config(env))
 	for domain, zonefile in zonefiles:
 		records = build_zone(domain, domains, additional_records, env)
 
@@ -863,8 +877,11 @@ def build_recommended_dns(env):
 if __name__ == "__main__":
 	from utils import load_environment
 	env = load_environment()
-	for zone, records in build_recommended_dns(env):
-		for record in records:
-			print("; " + record['explanation'])
-			print(record['qname'], record['rtype'], record['value'], sep="\t")
-			print()
+	if sys.argv[-1] == "--lint":
+		write_custom_dns_config(get_custom_dns_config(env), env)
+	else:
+		for zone, records in build_recommended_dns(env):
+			for record in records:
+				print("; " + record['explanation'])
+				print(record['qname'], record['rtype'], record['value'], sep="\t")
+				print()

management/mailconfig.py

@@ -2,69 +2,79 @@
 
 import subprocess, shutil, os, sqlite3, re
 import utils
+from email_validator import validate_email as validate_email_, EmailNotValidError
 
 def validate_email(email, mode=None):
-	# There are a lot of characters permitted in email addresses, but
-	# Dovecot's sqlite driver seems to get confused if there are any
-	# unusual characters in the address. Bah. Also note that since
-	# the mailbox path name is based on the email address, the address
-	# shouldn't be absurdly long and must not have a forward slash.
+	# Checks that an email address is syntactically valid. Returns True/False.
+	# Until Postfix supports SMTPUTF8, an email address may contain ASCII
+	# characters only; IDNs must be IDNA-encoded.
+	#
+	# When mode=="user", we're checking that this can be a user account name.
+	# Dovecot has tighter restrictions - letters, numbers, underscore, and
+	# dash only!
+	#
+	# When mode=="alias", we're allowing anything that can be in a Postfix
+	# alias table, i.e. omitting the local part ("@domain.tld") is OK.
 
-	if len(email) > 255: return False
-
-	if mode == 'user':
-		# For Dovecot's benefit, only allow basic characters.
-		ATEXT = r'[a-zA-Z0-9_\-]'
-	elif mode in (None, 'alias'):
-		# For aliases, we can allow any valid email address.
-		# Based on RFC 2822 and https://github.com/SyrusAkbary/validate_email/blob/master/validate_email.py,
-		# these characters are permitted in email addresses.
-		ATEXT = r'[\w!#$%&\'\*\+\-/=\?\^`\{\|\}~]' # see 3.2.4
-	else:
-		raise ValueError(mode)
-
-	# per RFC 2822 3.2.4
-	DOT_ATOM_TEXT_LOCAL = ATEXT + r'+(?:\.' + ATEXT + r'+)*'
-	if mode == 'alias':
-		# For aliases, Postfix accepts '@domain.tld' format for
-		# catch-all addresses on the source side and domain aliases
-		# on the destination side. Make the local part optional.
-		DOT_ATOM_TEXT_LOCAL = '(?:' + DOT_ATOM_TEXT_LOCAL + ')?'
-
-	# as above, but we can require that the host part have at least
-	# one period in it, so use a "+" rather than a "*" at the end
-	DOT_ATOM_TEXT_HOST = ATEXT + r'+(?:\.' + ATEXT + r'+)+'
-
-	# per RFC 2822 3.4.1
-	ADDR_SPEC = '^(%s)@(%s)$' % (DOT_ATOM_TEXT_LOCAL, DOT_ATOM_TEXT_HOST)
-
-	# Check the regular expression.
-	m = re.match(ADDR_SPEC, email)
-	if not m: return False
-
-	# Check that the domain part is IDNA-encodable.
-	localpart, domainpart = m.groups()
+	# Check the syntax of the address.
 	try:
-		domainpart.encode("idna")
-	except:
+		validate_email_(email,
+			allow_smtputf8=False,
+			check_deliverability=False,
+			allow_empty_local=(mode=="alias")
+			)
+	except EmailNotValidError:
 		return False
 
+	if mode == 'user':
+		# There are a lot of characters permitted in email addresses, but
+		# Dovecot's sqlite auth driver seems to get confused if there are any
+		# unusual characters in the address. Bah. Also note that since
+		# the mailbox path name is based on the email address, the address
+		# shouldn't be absurdly long and must not have a forward slash.
+		if len(email) > 255: return False
+		if re.search(r'[^\@\.a-zA-Z0-9_\-]+', email):
+			return False
+
+	# Everything looks good.
 	return True
 
 def sanitize_idn_email_address(email):
-	# Convert an IDNA-encoded email address (domain part) into Unicode
-	# before storing in our database. Chrome may IDNA-ize <input type="email">
-	# values before POSTing, so we want to normalize before putting
-	# values into the database.
+	# The user may enter Unicode in an email address. Convert the domain part
+	# to IDNA before going into our database. Leave the local part alone ---
+	# although validate_email will reject non-ASCII characters.
+	#
+	# The domain name system only exists in ASCII, so it doesn't make sense
+	# to store domain names in Unicode. We want to store what is meaningful
+	# to the underlying protocols.
 	try:
 		localpart, domainpart = email.split("@")
-		domainpart = domainpart.encode("ascii").decode("idna")
+		domainpart = domainpart.encode("idna").decode('ascii')
 		return localpart + "@" + domainpart
 	except:
-		# Domain part is already Unicode or not IDNA-valid, so
-		# leave unchanged.
+		# Domain part is not IDNA-valid, so leave unchanged. If there
+		# are non-ASCII characters it will be filtered out by
+		# validate_email.
 		return email
 
+def prettify_idn_email_address(email):
+	# This is the opposite of sanitize_idn_email_address. We store domain
+	# names in IDNA in the database, but we want to show Unicode to the user.
+	try:
+		localpart, domainpart = email.split("@")
+		domainpart = domainpart.encode("ascii").decode('idna')
+		return localpart + "@" + domainpart
+	except:
+		# Failed to decode IDNA. Should never happen.
+		return email
+
+def is_dcv_address(email):
+	email = email.lower()
+	for localpart in ("admin", "administrator", "postmaster", "hostmaster", "webmaster"):
+		if email.startswith(localpart+"@") or email.startswith(localpart+"+"):
+			return True
+	return False
+
 def open_database(env, with_connection=False):
 	conn = sqlite3.connect(env["STORAGE_ROOT"] + "/mail/users.sqlite")
 	if not with_connection:
@@ -90,7 +100,7 @@ def get_mail_users_ex(env, with_archived=False, with_slow_info=False):
 	#       {
 	#         email: "name@domain.tld",
 	#         privileges: [ "priv1", "priv2", ... ],
-	#         status: "active",
+	#         status: "active" | "inactive",
 	#       },
 	#       ...
 	#     ]
@@ -182,7 +192,8 @@ def get_mail_aliases_ex(env):
 	#     domain: "domain.tld",
 	#     alias: [
 	#       {
-	#         source: "name@domain.tld",
+	#         source: "name@domain.tld", # IDNA-encoded
+	#         source_display: "name@domain.tld", # full Unicode
 	#         destination: ["target1@domain.com", "target2@domain.com", ...],
 	#         required: True|False
 	#       },
@@ -207,7 +218,8 @@ def get_mail_aliases_ex(env):
 			}
 		domains[domain]["aliases"].append({
 			"source": source,
-			"destination": [d.strip() for d in destination.split(",")],
+			"source_display": prettify_idn_email_address(source),
+			"destination": [prettify_idn_email_address(d.strip()) for d in destination.split(",")],
 			"required": required,
 		})
 
@@ -219,19 +231,22 @@ def get_mail_aliases_ex(env):
 		domain["aliases"].sort(key = lambda alias : (alias["required"], alias["source"]))
 	return domains
 
-def get_domain(emailaddr):
-	return emailaddr.split('@', 1)[1]
+def get_domain(emailaddr, as_unicode=True):
+	# Gets the domain part of an email address. Turns IDNA
+	# back to Unicode for display.
+	ret = emailaddr.split('@', 1)[1]
+	if as_unicode: ret = ret.encode('ascii').decode('idna')
+	return ret
 
 def get_mail_domains(env, filter_aliases=lambda alias : True):
+	# Returns the domain names (IDNA-encoded) of all of the email addresses
+	# configured on the system.
 	return set(
-		   [get_domain(addr) for addr in get_mail_users(env)]
-		 + [get_domain(source) for source, target in get_mail_aliases(env) if filter_aliases((source, target)) ]
+		   [get_domain(addr, as_unicode=False) for addr in get_mail_users(env)]
+		 + [get_domain(source, as_unicode=False) for source, target in get_mail_aliases(env) if filter_aliases((source, target)) ]
 		 )
 
 def add_mail_user(email, pw, privs, env):
-	# accept IDNA domain names but normalize to Unicode before going into database
-	email = sanitize_idn_email_address(email)
-
 	# validate email
 	if email.strip() == "":
 		return ("No email address provided.", 400)
@@ -239,7 +254,13 @@ def add_mail_user(email, pw, privs, env):
 		return ("Invalid email address.", 400)
 	elif not validate_email(email, mode='user'):
 		return ("User account email addresses may only use the ASCII letters A-Z, the digits 0-9, underscore (_), hyphen (-), and period (.).", 400)
+	elif is_dcv_address(email) and len(get_mail_users(env)) > 0:
+		# Make domain control validation hijacking a little harder to mess up by preventing the usual
+		# addresses used for DCV from being user accounts. Except let it be the first account because
+		# during box setup the user won't know the rules.
+		return ("You may not make a user account for that address because it is frequently used for domain control validation. Use an alias instead if necessary.", 400)
 
+	# validate password
 	validate_password(pw)
 
 	# validate privileges
@@ -290,9 +311,6 @@ def add_mail_user(email, pw, privs, env):
 	return kick(env, "mail user added")
 
 def set_mail_password(email, pw, env):
-	# accept IDNA domain names but normalize to Unicode before going into database
-	email = sanitize_idn_email_address(email)
-
 	# validate that password is acceptable
 	validate_password(pw)
 	
@@ -326,9 +344,6 @@ def get_mail_password(email, env):
 	return rows[0][0]
 
 def remove_mail_user(email, env):
-	# accept IDNA domain names but normalize to Unicode before going into database
-	email = sanitize_idn_email_address(email)
-
 	# remove
 	conn, c = open_database(env, with_connection=True)
 	c.execute("DELETE FROM users WHERE email=?", (email,))
@@ -342,15 +357,13 @@ def remove_mail_user(email, env):
 def parse_privs(value):
 	return [p for p in value.split("\n") if p.strip() != ""]
 
-def get_mail_user_privileges(email, env):
-	# accept IDNA domain names but normalize to Unicode before going into database
-	email = sanitize_idn_email_address(email)
-
+def get_mail_user_privileges(email, env, empty_on_error=False):
 	# get privs
 	c = open_database(env)
 	c.execute('SELECT privileges FROM users WHERE email=?', (email,))
 	rows = c.fetchall()
 	if len(rows) != 1:
+		if empty_on_error: return []
 		return ("That's not a user (%s)." % email, 400)
 	return parse_privs(rows[0][0])
 
@@ -360,9 +373,6 @@ def validate_privilege(priv):
 	return None
 
 def add_remove_mail_user_privilege(email, priv, action, env):
-	# accept IDNA domain names but normalize to Unicode before going into database
-	email = sanitize_idn_email_address(email)
-
 	# validate
 	validation = validate_privilege(priv)
 	if validation: return validation
@@ -390,32 +400,47 @@ def add_remove_mail_user_privilege(email, priv, action, env):
 	return "OK"
 
 def add_mail_alias(source, destination, env, update_if_exists=False, do_kick=True):
-	# accept IDNA domain names but normalize to Unicode before going into database
+	# convert Unicode domain to IDNA
 	source = sanitize_idn_email_address(source)
 
 	# validate source
-	if source.strip() == "":
+	source = source.strip()
+	if source == "":
 		return ("No incoming email address provided.", 400)
 	if not validate_email(source, mode='alias'):
 		return ("Invalid incoming email address (%s)." % source, 400)
 
+	# extra checks for email addresses used in domain control validation
+	is_dcv_source = is_dcv_address(source)
+
 	# validate destination
 	dests = []
 	destination = destination.strip()
-	if validate_email(destination, mode='alias'):
-		# Oostfix allows a single @domain.tld as the destination, which means
-		# the local part on the address is preserved in the rewrite.
-		dests.append(sanitize_idn_email_address(destination))
+	
+	# Postfix allows a single @domain.tld as the destination, which means
+	# the local part on the address is preserved in the rewrite. We must
+	# try to convert Unicode to IDNA first before validating that it's a
+	# legitimate alias address. Don't allow this sort of rewriting for
+	# DCV source addresses.
+	d1 = sanitize_idn_email_address(destination)
+	if validate_email(d1, mode='alias') and not is_dcv_source:
+		dests.append(d1)
+
 	else:
 		# Parse comma and \n-separated destination emails & validate. In this
 		# case, the recipients must be complete email addresses.
 		for line in destination.split("\n"):
 			for email in line.split(","):
 				email = email.strip()
-				email = sanitize_idn_email_address(email) # Unicode => IDNA
 				if email == "": continue
+				email = sanitize_idn_email_address(email) # Unicode => IDNA
 				if not validate_email(email):
 					return ("Invalid destination email address (%s)." % email, 400)
+				if is_dcv_source and not is_dcv_address(email) and "admin" not in get_mail_user_privileges(email, env, empty_on_error=True):
+					# Make domain control validation hijacking a little harder to mess up by
+					# requiring aliases for email addresses typically used in DCV to forward
+					# only to accounts that are administrators on this system.
+					return ("This alias can only have administrators of this system as destinations because the address is frequently used for domain control validation.", 400)
 				dests.append(email)
 	if len(destination) == 0:
 		return ("No destination email address(es) provided.", 400)
@@ -440,7 +465,7 @@ def add_mail_alias(source, destination, env, update_if_exists=False, do_kick=Tru
 		return kick(env, return_status)
 
 def remove_mail_alias(source, env, do_kick=True):
-	# accept IDNA domain names but normalize to Unicode before going into database
+	# convert Unicode domain to IDNA
 	source = sanitize_idn_email_address(source)
 
 	# remove

management/status_checks.py

@@ -11,7 +11,7 @@ import sys, os, os.path, re, subprocess, datetime, multiprocessing.pool
 import dns.reversename, dns.resolver
 import dateutil.parser, dateutil.tz
 
-from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config
+from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config, get_secondary_dns
 from web_update import get_web_domains, get_domain_ssl_files
 from mailconfig import get_mail_domains, get_mail_aliases
 
@@ -170,7 +170,7 @@ def check_software_updates(env, output):
 def check_system_aliases(env, output):
 	# Check that the administrator alias exists since that's where all
 	# admin email is automatically directed.
-	check_alias_exists("administrator@" + env['PRIMARY_HOSTNAME'], env, output)
+	check_alias_exists("System administrator address", "administrator@" + env['PRIMARY_HOSTNAME'], env, output)
 
 def check_free_disk_space(rounded_values, env, output):
 	# Check free disk space.
@@ -246,7 +246,8 @@ def run_domain_checks(rounded_time, env, output, pool):
 def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains):
 	output = BufferedOutput()
 
-	output.add_heading(domain)
+	# The domain is IDNA-encoded, but for display use Unicode.
+	output.add_heading(domain.encode('ascii').decode('idna'))
 
 	if domain == env["PRIMARY_HOSTNAME"]:
 		check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles)
@@ -283,7 +284,7 @@ def check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles):
 	# the nameserver, are reporting the right info --- but if the glue is incorrect this
 	# will probably fail.
 	if ns_ips == env['PUBLIC_IP'] + '/' + env['PUBLIC_IP']:
-		output.print_ok("Nameserver glue records are correct at registrar. [ns1/ns2.%s => %s]" % (env['PRIMARY_HOSTNAME'], env['PUBLIC_IP']))
+		output.print_ok("Nameserver glue records are correct at registrar. [ns1/ns2.%s ↦ %s]" % (env['PRIMARY_HOSTNAME'], env['PUBLIC_IP']))
 
 	elif ip == env['PUBLIC_IP']:
 		# The NS records are not what we expect, but the domain resolves correctly, so
@@ -300,7 +301,7 @@ def check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles):
 
 	# Check that PRIMARY_HOSTNAME resolves to PUBLIC_IP in public DNS.
 	if ip == env['PUBLIC_IP']:
-		output.print_ok("Domain resolves to box's IP address. [%s => %s]" % (env['PRIMARY_HOSTNAME'], env['PUBLIC_IP']))
+		output.print_ok("Domain resolves to box's IP address. [%s ↦ %s]" % (env['PRIMARY_HOSTNAME'], env['PUBLIC_IP']))
 	else:
 		output.print_error("""This domain must resolve to your box's IP address (%s) in public DNS but it currently resolves
 			to %s. It may take several hours for public DNS to update after a change. This problem may result from other
@@ -312,7 +313,7 @@ def check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles):
 	ipaddr_rev = dns.reversename.from_address(env['PUBLIC_IP'])
 	existing_rdns = query_dns(ipaddr_rev, "PTR")
 	if existing_rdns == domain:
-		output.print_ok("Reverse DNS is set correctly at ISP. [%s => %s]" % (env['PUBLIC_IP'], env['PRIMARY_HOSTNAME']))
+		output.print_ok("Reverse DNS is set correctly at ISP. [%s ↦ %s]" % (env['PUBLIC_IP'], env['PRIMARY_HOSTNAME']))
 	else:
 		output.print_error("""Your box's reverse DNS is currently %s, but it should be %s. Your ISP or cloud provider will have instructions
 			on setting up reverse DNS for your box at %s.""" % (existing_rdns, domain, env['PUBLIC_IP']) )
@@ -334,12 +335,12 @@ def check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles):
                         % (tlsa_qname, tlsa25, tlsa25_expected))
 
 	# Check that the hostmaster@ email address exists.
-	check_alias_exists("hostmaster@" + domain, env, output)
+	check_alias_exists("Hostmaster contact address", "hostmaster@" + domain, env, output)
 
-def check_alias_exists(alias, env, output):
+def check_alias_exists(alias_name, alias, env, output):
 	mail_alises = dict(get_mail_aliases(env))
 	if alias in mail_alises:
-		output.print_ok("%s exists as a mail alias [=> %s]" % (alias, mail_alises[alias]))
+		output.print_ok("%s exists as a mail alias. [%s ↦ %s]" % (alias_name, alias, mail_alises[alias]))
 	else:
 		output.print_error("""You must add a mail alias for %s and direct email to you or another administrator.""" % alias)
 
@@ -356,11 +357,11 @@ def check_dns_zone(domain, env, output, dns_zonefiles):
 	# the TLD, and so we're not actually checking the TLD. For that we'd need
 	# to do a DNS trace.
 	ip = query_dns(domain, "A")
-	custom_dns = get_custom_dns_config(env)
+	secondary_ns = get_secondary_dns(get_custom_dns_config(env)) or "ns2." + env['PRIMARY_HOSTNAME']
 	existing_ns = query_dns(domain, "NS")
 	correct_ns = "; ".join(sorted([
 		"ns1." + env['PRIMARY_HOSTNAME'],
-		custom_dns.get("_secondary_nameserver", "ns2." + env['PRIMARY_HOSTNAME']),
+		secondary_ns,
 		]))
 	if existing_ns.lower() == correct_ns.lower():
 		output.print_ok("Nameservers are set correctly at registrar. [%s]" % correct_ns)
@@ -470,7 +471,7 @@ def check_mail_domain(domain, env, output):
 					change. This problem may result from other issues listed here.""" % (recommended_mx,))
 
 	elif mxhost == env['PRIMARY_HOSTNAME']:
-		good_news = "Domain's email is directed to this domain. [%s => %s]" % (domain, mx)
+		good_news = "Domain's email is directed to this domain. [%s ↦ %s]" % (domain, mx)
 		if mx != recommended_mx:
 			good_news += "  This configuration is non-standard.  The recommended configuration is '%s'." % (recommended_mx,)
 		output.print_ok(good_news)
@@ -482,7 +483,7 @@ def check_mail_domain(domain, env, output):
 	# Check that the postmaster@ email address exists. Not required if the domain has a
 	# catch-all address or domain alias.
 	if "@" + domain not in dict(get_mail_aliases(env)):
-		check_alias_exists("postmaster@" + domain, env, output)
+		check_alias_exists("Postmaster contact address", "postmaster@" + domain, env, output)
 
 	# Stop if the domain is listed in the Spamhaus Domain Block List.
 	# The user might have chosen a domain that was previously in use by a spammer
@@ -502,7 +503,7 @@ def check_web_domain(domain, rounded_time, env, output):
 	if domain != env['PRIMARY_HOSTNAME']:
 		ip = query_dns(domain, "A")
 		if ip == env['PUBLIC_IP']:
-			output.print_ok("Domain resolves to this box's IP address. [%s => %s]" % (domain, env['PUBLIC_IP']))
+			output.print_ok("Domain resolves to this box's IP address. [%s ↦ %s]" % (domain, env['PUBLIC_IP']))
 		else:
 			output.print_error("""This domain should resolve to your box's IP address (%s) if you would like the box to serve
 				webmail or a website on this domain. The domain currently resolves to %s in public DNS. It may take several hours for
@@ -639,7 +640,6 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, rounded_time=Fal
 		if m:
 			cert_expiration_date = dateutil.parser.parse(m.group(1))
 
-	domain = domain.encode("idna").decode("ascii")
 	wildcard_domain = re.sub("^[^\.]+", "*", domain)
 	if domain is not None and domain not in certificate_names and wildcard_domain not in certificate_names:
 		return ("The certificate is for the wrong domain name. It is for %s."

management/templates/aliases.html

@@ -27,7 +27,7 @@
     <label for="addaliasEmail" class="col-sm-1 control-label">Alias</label>
     <div class="col-sm-10">
       <input type="email" class="form-control" id="addaliasEmail">
-      <div style="margin-top: 3px; padding-left: 3px; font-size: 90%" class="text-muted">You may use international (non-ASCII) characters, but this has not yet been well tested.</div>
+      <div style="margin-top: 3px; padding-left: 3px; font-size: 90%" class="text-muted">You may use international (non-ASCII) characters for the domain part of the email address only.</div>
     </div>
   </div>
   <div class="form-group">
@@ -98,8 +98,8 @@ function show_aliases() {
           n.attr('id', '');
 
           if (alias.required) n.addClass('alias-required');
-          n.attr('data-email', alias.source);
-          n.find('td.email').text(alias.source)
+          n.attr('data-email', alias.source_display); // this is decoded from IDNA, but will get re-coded to IDNA on the backend
+          n.find('td.email').text(alias.source_display)
           for (var j = 0; j < alias.destination.length; j++)
             n.find('td.target').append($("<div></div>").text(alias.destination[j]))
           $('#alias_table tbody').append(n);

management/templates/custom-dns.html

@@ -93,44 +93,56 @@
 
 <p>Use your box&rsquo;s DNS API to set custom DNS records on domains hosted here. For instance, you can create your own dynamic DNS service.</p>
 
-<p>Send a POST request like this:</p>
+<p>Usage:</p>
 
-<pre>curl -d "" --user {email}:{password} https://{{hostname}}/admin/dns/set/<b>qname</b>[/<b>rtype</b>[/<b>value</b>]]</pre>
+<pre>curl -X <b>VERB</b> [-d "<b>value</b>"] --user {email}:{password} https://{{hostname}}/admin/dns/custom[/<b>qname</b>[/<b>rtype</b>]]</pre>
 
-<h4>HTTP POST parameters</h4>
+<p>(Brackets denote an optional argument.)</p>
+
+<h4>Verbs</h4>
+
+<table class="table">
+<thead><th>Verb</th> <th>Usage</th></thead>
+<tr><td>GET</td> <td>Returns matching custom DNS records as a JSON array of objects. Each object has the keys <code>qname</code>, <code>rtype</code>, and <code>value</code>. The optional <code>qname</code> and <code>rtype</code> parameters in the request URL filter the records returned in the response. The request body (<code>-d "..."</code>) must be omitted.</td></tr>
+<tr><td>PUT</td> <td>Sets a custom DNS record replacing any existing records with the same <code>qname</code> and <code>rtype</code>. Use PUT (instead of POST) when you only have one value for a <code>qname</code> and <code>rtype</code>, such as typical <code>A</code> records (without round-robin).</td></tr>
+<tr><td>POST</td> <td>Adds a new custom DNS record. Use POST when you have multiple <code>TXT</code> records or round-robin <code>A</code> records. (PUT would delete previously added records.)</td></tr>
+<tr><td>DELETE</td> <td>Deletes custom DNS records. If the request body (<code>-d "..."</code>) is empty or omitted, deletes all records matching the <code>qname</code> and <code>rtype</code>. If the request body is present, deletes only the record matching the <code>qname</code>, <code>rtype</code> and value.</td></tr>
+</table>
+
+<h4>Parameters</h4>
 
 <table class="table">
 <thead><th>Parameter</th> <th>Value</th></thead>
 <tr><td>email</td> <td>The email address of any administrative user here.</td></tr>
 <tr><td>password</td> <td>That user&rsquo;s password.</td></tr>
-<tr><td>qname</td> <td>The fully qualified domain name for the record you are trying to set.</td></tr>
-<tr><td>rtype</td> <td>The resource type. <code>A</code> if omitted. Possible values: <code>A</code> (an IPv4 address), <code>AAAA</code> (an IPv6 address), <code>TXT</code> (a text string), or <code>CNAME</code> (an alias, which is a fully qualified domain name).</td></tr>
-<tr><td>value</td> <td>The new record&rsquo;s value. If omitted, the IPv4 address of the remote host is used. This is handy for dynamic DNS! To delete a record, use &ldquo;__delete__&rdquo;.</td></tr>
+<tr><td>qname</td> <td>The fully qualified domain name for the record you are trying to set. It must be one of the domain names or a subdomain of one of the domain names hosted on this box. (Add mail users or aliases to add new domains.)</td></tr>
+<tr><td>rtype</td> <td>The resource type. Defaults to <code>A</code> if omitted. Possible values: <code>A</code> (an IPv4 address), <code>AAAA</code> (an IPv6 address), <code>TXT</code> (a text string), <code>CNAME</code> (an alias, which is a fully qualified domain name &mdash; don&rsquo;t forget the final period), <code>MX</code>, or <code>SRV</code>.</td></tr>
+<tr><td>value</td> <td>For PUT, POST, and DELETE, the record&rsquo;s value. If the <code>rtype</code> is <code>A</code> or <code>AAAA</code> and <code>value</code> is empty or omitted, the IPv4 or IPv6 address of the remote host is used (be sure to use the <code>-4</code> or <code>-6</code> options to curl). This is handy for dynamic DNS!</td></tr>
 </table>
 
-<p style="margin-top: 1em">Note that <code>-d ""</code> is merely to ensure curl sends a POST request. You do not need to put anything inside the quotes. You can also pass the value using typical form encoding in the POST body.</p>
-
 <p>Strict <a href="http://tools.ietf.org/html/rfc4408">SPF</a> and <a href="https://datatracker.ietf.org/doc/draft-kucherawy-dmarc-base/?include_text=1">DMARC</a> records will be added to all custom domains unless you override them.</p>
 
 <h4>Examples:</h4>
 
+<p>Try these examples. For simplicity the examples omit the <code>--user me@mydomain.com:yourpassword</code> command line argument which you must fill in with your email address and password.</p>
+
 <pre># sets laptop.mydomain.com to point to the IP address of the machine you are executing curl on
-curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/laptop.mydomain.com
+curl -X PUT https://{{hostname}}/admin/dns/custom/laptop.mydomain.com
 
-# sets an alias
-curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/foo.mydomain.com/cname/bar.mydomain.com
+# deletes that record and all A records for that domain name
+curl -X DELETE https://{{hostname}}/admin/dns/custom/laptop.mydomain.com
 
-# clears the alias
-curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/bar.mydomain.com/cname/__delete__
+# sets a CNAME alias
+curl -X PUT -d "bar.mydomain.com." https://{{hostname}}/admin/dns/custom/foo.mydomain.com/cname
 
-# sets a TXT record using the alternate value syntax
-curl -d "value=something%20here" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/foo.mydomain.com/txt
+# deletes that CNAME and all CNAME records for that domain name
+curl -X DELETE https://{{hostname}}/admin/dns/custom/foo.mydomain.com/cname
 
-# sets a <a href="http://en.wikipedia.org/wiki/SRV_record">SRV record</a> for the "service" and "protocol" hosted on "target" server
-curl -d "" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/_service._protocol.{{hostname}}/srv/"priority weight port target"
+# adds a TXT record using POST to preserve any previous TXT records
+curl -X POST -d "some text here" https://{{hostname}}/admin/dns/custom/foo.mydomain.com/txt
 
-# sets a SRV record using the value syntax
-curl -d "value=priority weight port target" --user me@mydomain.com:###### https://{{hostname}}/admin/dns/set/_service._protocol.host/srv
+# deletes that one TXT record while preserving other TXT records
+curl -X DELETE -d "some text here" https://{{hostname}}/admin/dns/custom/foo.mydomain.com/txt
 </pre>
 
 <script>
@@ -161,7 +173,7 @@ function show_custom_dns() {
 
 function show_current_custom_dns() {
   api(
-    "/dns/set",
+    "/dns/custom",
     "GET",
     { },
     function(data) {
@@ -176,6 +188,7 @@ function show_current_custom_dns() {
         $('#custom-dns-current').find("tbody").append(tr);
         tr.attr('data-qname', data[i].qname);
         tr.attr('data-rtype', data[i].rtype);
+        tr.attr('data-value', data[i].value);
         tr.append($('<td class="long"/>').text(data[i].qname));
         tr.append($('<td/>').text(data[i].rtype));
         tr.append($('<td class="long"/>').text(data[i].value));
@@ -187,7 +200,8 @@ function show_current_custom_dns() {
 function delete_custom_dns_record(elem) {
   var qname = $(elem).parents('tr').attr('data-qname');
   var rtype = $(elem).parents('tr').attr('data-rtype');
-  do_set_custom_dns(qname, rtype, "__delete__");
+  var value = $(elem).parents('tr').attr('data-value');
+  do_set_custom_dns(qname, rtype, value, "DELETE");
   return false;
 }
 
@@ -208,7 +222,7 @@ function do_set_secondary_dns() {
     });
 }
 
-function do_set_custom_dns(qname, rtype, value) {
+function do_set_custom_dns(qname, rtype, value, method) {
   if (!qname) {
     if ($('#customdnsQname').val() != '')
       qname = $('#customdnsQname').val() + '.' + $('#customdnsZone').val();
@@ -216,21 +230,20 @@ function do_set_custom_dns(qname, rtype, value) {
       qname = $('#customdnsZone').val();
     rtype = $('#customdnsType').val();
     value = $('#customdnsValue').val();
+    method = 'POST';
   }
 
   api(
-    "/dns/set/" + qname + "/" + rtype,
-    "POST",
-    {
-      value: value
-    },
+    "/dns/custom/" + qname + "/" + rtype,
+    method,
+    value,
     function(data) {
       if (data == "") return; // nothing updated
       show_modal_error("Custom DNS", $("<pre/>").text(data));
       show_current_custom_dns();
     },
     function(err) {
-      show_modal_error("Custom DNS", $("<pre/>").text(err));
+      show_modal_error("Custom DNS (Error)", $("<pre/>").text(err));
     });
 }
 

management/templates/index.html

@@ -1,18 +1,15 @@
 <!DOCTYPE html>
-<!--[if lt IE 7]>      <html class="no-js lt-ie9 lt-ie8 lt-ie7"> <![endif]-->
-<!--[if IE 7]>         <html class="no-js lt-ie9 lt-ie8"> <![endif]-->
-<!--[if IE 8]>         <html class="no-js lt-ie9"> <![endif]-->
-<!--[if gt IE 8]><!--> <html class="no-js"> <!--<![endif]-->
+<html lang="en">
     <head>
-        <meta charset="utf-8">
         <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
-        <meta name="viewport" content="width=device-width">
+        <meta charset="utf-8">
+        <meta name="viewport" content="width=device-width, initial-scale=1">
 
         <title>{{hostname}} - Mail-in-a-Box Control Panel</title>
 
         <meta name="robots" content="noindex, nofollow">
 
-        <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap.min.css">
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap.min.css">
         <style>
             @import url(https://fonts.googleapis.com/css?family=Raleway:400,700);
             @import url(https://fonts.googleapis.com/css?family=Ubuntu:300);
@@ -73,18 +70,19 @@
 	       margin-bottom: 1em;
 	    }
         </style>
-        <link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/css/bootstrap-theme.min.css">
-        <style>
-        </style>
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap-theme.min.css">
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
     </head>
     <body>
-        <!--[if lt IE 7]>
-            <p class="chromeframe">You are using an <strong>outdated</strong> browser. Please <a href="http://browsehappy.com/">upgrade your browser</a> or <a href="http://www.google.com/chromeframe/?redirect=true">activate Google Chrome Frame</a> to improve your experience.</p>
-        <![endif]-->
-    <div class="navbar navbar-inverse navbar-fixed-top">
+
+    <!--[if lt IE 8]><p>Internet Explorer version 8 or any modern web browser is required to use this website, sorry.<![endif]-->
+    <!--[if gt IE 7]><!-->
+
+    <div class="navbar navbar-inverse navbar-fixed-top" role="navigation">
       <div class="container">
         <div class="navbar-header">
-          <button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-collapse">
+          <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse">
+            <span class="sr-only">Toggle navigation</span>
             <span class="icon-bar"></span>
             <span class="icon-bar"></span>
             <span class="icon-bar"></span>
@@ -175,9 +173,9 @@
       </footer>
     </div> <!-- /container -->
 
-        <div id="ajax_loading_indicator" style="display: none; position: fixed; left: 0; top: 0; width: 100%; height: 100%; text-align: center; background-color: rgba(255,255,255,.75)">
+        <div id="ajax_loading_indicator" style="display: none; position: fixed; left: 0; top: 0; width: 100%; height: 100%; z-index: 100000; text-align: center; background-color: rgba(255,255,255,.75)">
           <div style="margin: 20% auto">
-            <div><span class="glyphicon glyphicon-time"></span></div>
+            <div><span class="fa fa-spinner fa-pulse"></span></div>
             <div>Loading...</div>
           </div>
         </div>
@@ -200,8 +198,8 @@
           </div>
         </div>
 
-        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.10.1/jquery.min.js"></script>
-        <script src="//maxcdn.bootstrapcdn.com/bootstrap/3.2.0/js/bootstrap.min.js"></script>
+        <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
+        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/js/bootstrap.min.js"></script>
 
         <script>
 var global_modal_state = null;
@@ -233,7 +231,7 @@ $(function() {
 function show_modal_error(title, message, callback) {
   $('#global_modal h4').text(title);
   $('#global_modal .modal-body').html("<p/>");
-  if (typeof question == String) {
+  if (typeof question == 'string') {
     $('#global_modal p').text(message);
     $('#global_modal .modal-dialog').addClass("modal-sm");
   } else {
@@ -245,11 +243,12 @@ function show_modal_error(title, message, callback) {
   global_modal_funcs = [callback, callback];
   global_modal_state = null;
   $('#global_modal').modal({});
+  return false; // handy when called from onclick
 }
 
 function show_modal_confirm(title, question, verb, yes_callback, cancel_callback) {
   $('#global_modal h4').text(title);
-  if (typeof question == String) {
+  if (typeof question == 'string') {
     $('#global_modal .modal-dialog').addClass("modal-sm");
     $('#global_modal .modal-body').html("<p/>");
     $('#global_modal p').text(question);
@@ -257,11 +256,17 @@ function show_modal_confirm(title, question, verb, yes_callback, cancel_callback
     $('#global_modal .modal-dialog').removeClass("modal-sm");
     $('#global_modal .modal-body').html("").append(question);
   }
-  $('#global_modal .btn-default').show().text("Cancel");
-  $('#global_modal .btn-danger').show().text(verb);
+  if (typeof verb == 'string') {
+    $('#global_modal .btn-default').show().text("Cancel");
+    $('#global_modal .btn-danger').show().text(verb);
+  } else {
+    $('#global_modal .btn-default').show().text(verb[1]);
+    $('#global_modal .btn-danger').show().text(verb[0]);
+  }
   global_modal_funcs = [yes_callback, cancel_callback];
   global_modal_state = null;
   $('#global_modal').modal({});
+  return false; // handy when called from onclick
 }
 
 var ajax_num_executing_requests = 0;
@@ -270,7 +275,7 @@ function ajax(options) {
   function hide_loading_indicator() {
     ajax_num_executing_requests--;
     if (ajax_num_executing_requests == 0)
-      $('#ajax_loading_indicator').stop().hide(); // stop() prevents an ongoing fade from causing the thing to be shown again after this call
+      $('#ajax_loading_indicator').stop(true).hide(); // stop() prevents an ongoing fade from causing the thing to be shown again after this call
   }
   var old_success = options.success;
   var old_error = options.error;
@@ -290,6 +295,7 @@ function ajax(options) {
   };
   ajax_num_executing_requests++;
   $.ajax(options);
+  return false; // handy when called from onclick
 }
 
 var api_credentials = ["", ""];
@@ -331,6 +337,11 @@ function api(url, method, data, callback, callback_error) {
     method: method,
     cache: false,
     data: data,
+
+    // the custom DNS api sends raw POST/PUT bodies --- prevent URL-encoding
+    processData: typeof data != "string",
+    mimeType: typeof data == "string" ? "text/plain; charset=ascii" : null,
+
     beforeSend: function(xhr) {
       // We don't store user credentials in a cookie to avoid the hassle of CSRF
       // attacks. The Authorization header only gets set in our AJAX calls triggered

management/templates/login.html

@@ -1,12 +1,20 @@
 <h1 style="margin: 1em; text-align: center">{{hostname}}</h1>
 
-{% if no_admins_exist %}
+{% if no_users_exist or no_admins_exist %}
 <div class="row">
 <div class="col-md-offset-2 col-md-8">
+  {% if no_users_exist %}
+  <p class="text-danger">There are no users on this system! To make an administrative user,
+  log into this machine using SSH (like when you first set it up) and run:</p>
+  <pre>cd mailinabox
+sudo tools/mail.py user add me@{{hostname}}
+sudo tools/mail.py user make-admin me@{{hostname}}</pre>
+  {% else %}
   <p class="text-danger">There are no administrative users on this system! To make an administrative user,
   log into this machine using SSH (like when you first set it up) and run:</p>
   <pre>cd mailinabox
-sudo tools/mail.py user make-admin your@emailaddress.com</pre>
+sudo tools/mail.py user make-admin me@{{hostname}}</pre>
+  {% endif %}
   <hr>
 </div>
 </div>  

management/templates/system-backup.html

@@ -28,8 +28,6 @@
   </tbody>
 </table>
 
-<p style="margin-top: 2em"><small>The size column in the table indicates the size of the encrypted backup, but the total size on disk shown above includes storage for unencrypted intermediate files.</small></p>
-
 <script>
 function nice_size(bytes) {
   var powers = ['bytes', 'KB', 'MB', 'GB', 'TB'];
@@ -54,7 +52,7 @@ function show_system_backup() {
     "GET",
     { },
     function(r) {
-      $('#backup-location').text(r.encdirectory);
+      $('#backup-location').text(r.directory);
       $('#backup-encpassword-file').text(r.encpwfile);
 
       $('#backup-status tbody').html("");
@@ -72,7 +70,7 @@ function show_system_backup() {
         tr.append( $('<td/>').text(b.date_str + " " + r.tz) );
         tr.append( $('<td/>').text(b.date_delta + " ago") );
         tr.append( $('<td/>').text(b.full ? "full" : "increment") );
-        tr.append( $('<td style="text-align: right"/>').text( nice_size(b.encsize)) );
+        tr.append( $('<td style="text-align: right"/>').text( nice_size(b.size)) );
         if (b.deleted_in)
           tr.append( $('<td/>').text(b.deleted_in) );
         else
@@ -80,7 +78,6 @@ function show_system_backup() {
         $('#backup-status tbody').append(tr);
 
         total_disk_size += b.size;
-        total_disk_size += b.encsize;
       }
 
       $('#backup-total-size').text(nice_size(total_disk_size));

management/templates/users.html

@@ -188,6 +188,13 @@ function users_set_password(elem) {
 
 function users_remove(elem) {
   var email = $(elem).parents('tr').attr('data-email');
+
+  // can't remove yourself
+  if (api_credentials != null && email == api_credentials[0]) {
+    show_modal_error("Archive User", "You cannot archive your own account.");
+    return;
+  }
+
   show_modal_confirm(
     "Archive User",
     $("<p>Are you sure you want to archive <b>" + email + "</b>?</p> <p>The user's mailboxes will not be deleted (you can do that later), but the user will no longer be able to log into any services on this machine.</p>"),
@@ -223,7 +230,7 @@ function mod_priv(elem, add_remove) {
   var add_remove1 = add_remove.charAt(0).toUpperCase() + add_remove.substring(1);
   show_modal_confirm(
     "Modify Privileges",
-    "Are you sure you want to " + add_remove + " the " + priv + " privilege for <b>" + email + "</b>?",
+    $("<p>Are you sure you want to " + add_remove + " the " + priv + " privilege for <b>" + email + "</b>?</p>"),
     add_remove1,
     function() {
       api(

management/templates/web.html

@@ -82,7 +82,7 @@ function show_change_web_root(elem) {
   var root = $(elem).parents('tr').attr('data-custom-web-root');
   show_modal_confirm(
     'Change Root Directory for ' + domain,
-    '<p>You can change the static directory for <tt>' + domain + '</tt> to:</p> <p><tt>' + root + '</tt></p> <p>First create this directory on the server. Then click Update to scan for the directory and update web settings.',
+    $('<p>You can change the static directory for <tt>' + domain + '</tt> to:</p> <p><tt>' + root + '</tt></p> <p>First create this directory on the server. Then click Update to scan for the directory and update web settings.'),
     'Update',
     function() { do_web_update(); });
 }

management/utils.py

@@ -184,3 +184,19 @@ def du(path):
             seen.add(stat.st_ino)
             total_size += stat.st_size
     return total_size
+
+def wait_for_service(port, public, env, timeout):
+	# Block until a service on a given port (bound privately or publicly)
+	# is taking connections, with a maximum timeout.
+	import socket, time
+	start = time.perf_counter()
+	while True:
+		s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+		s.settimeout(timeout/3)
+		try:
+			s.connect(("127.0.0.1" if not public else env['PUBLIC_IP'], port))
+			return True
+		except OSError:
+			if time.perf_counter() > start+timeout:
+				return False
+		time.sleep(min(timeout/4, 1))

management/web_update.py

@@ -24,12 +24,9 @@ def get_web_domains(env):
 	# ...Unless the domain has an A/AAAA record that maps it to a different
 	# IP address than this box. Remove those domains from our list.
 	dns = get_custom_dns_config(env)
-	for domain, value in dns.items():
+	for domain, rtype, value in dns:
 		if domain not in domains: continue
-		if (isinstance(value, str) and (value != "local")) \
-		  or (isinstance(value, dict) and ("CNAME" in value)) \
-		  or (isinstance(value, dict) and ("A" in value) and (value["A"] != "local")) \
-		  or (isinstance(value, dict) and ("AAAA" in value) and (value["AAAA"] != "local")):
+		if rtype == "CNAME" or (rtype in ("A", "AAAA") and value != "local"):
 			domains.remove(domain)
 
 	# Sort the list. Put PRIMARY_HOSTNAME first so it becomes the
@@ -89,7 +86,7 @@ def make_domain_config(domain, template, template_for_primaryhost, env):
 
 	# Replace substitution strings in the template & return.
 	nginx_conf = nginx_conf.replace("$STORAGE_ROOT", env['STORAGE_ROOT'])
-	nginx_conf = nginx_conf.replace("$HOSTNAME", domain.encode("idna").decode("ascii"))
+	nginx_conf = nginx_conf.replace("$HOSTNAME", domain)
 	nginx_conf = nginx_conf.replace("$ROOT", root)
 	nginx_conf = nginx_conf.replace("$SSL_KEY", ssl_key)
 	nginx_conf = nginx_conf.replace("$SSL_CERTIFICATE", ssl_certificate)
@@ -213,7 +210,7 @@ def create_csr(domain, ssl_key, env):
                 "-key", ssl_key,
                 "-out",  "/dev/stdout",
                 "-sha256",
-                "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain.encode("idna").decode("ascii"))])
+                "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain)])
 
 def install_cert(domain, ssl_cert, ssl_chain, env):
 	if domain not in get_web_domains(env):

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.08
+	TAG=v0.09
 fi
 
 # Are we running as root?

setup/dns.sh

@@ -10,19 +10,7 @@
 source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
-# Install `nsd`, our DNS server software, and `ldnsutils` which helps
-# us sign zones for DNSSEC.
-
-# ...but first, we have to create the user because the 
-# current Ubuntu forgets to do so in the .deb
-# (see issue #25 and https://bugs.launchpad.net/ubuntu/+source/nsd/+bug/1311886)
-if id nsd > /dev/null 2>&1; then
-	true #echo "nsd user exists... good"; #NODOC
-else
-	useradd nsd;
-fi
-
-# Okay now install the packages.
+# Install the packages.
 #
 # * nsd: The non-recursive nameserver that publishes our DNS records.
 # * ldnsutils: Helper utilities for signing DNSSEC zones.
@@ -34,6 +22,35 @@ apt_install nsd ldnsutils openssh-client
 
 mkdir -p /var/run/nsd
 
+cat > /etc/nsd/nsd.conf << EOF;
+# No not edit. Overwritten by Mail-in-a-Box setup.
+server:
+  hide-version: yes
+
+  # identify the server (CH TXT ID.SERVER entry).
+  identity: ""
+
+  # The directory for zonefile: files.
+  zonesdir: "/etc/nsd/zones"
+
+  # Allows NSD to bind to IP addresses that are not (yet) added to the
+  # network interface. This allows nsd to start even if the network stack
+  # isn't fully ready, which apparently happens in some cases.
+  # See https://www.nlnetlabs.nl/projects/nsd/nsd.conf.5.html.
+  ip-transparent: yes
+
+EOF
+
+# Since we have bind9 listening on localhost for locally-generated
+# DNS queries that require a recursive nameserver, and the system
+# might have other network interfaces for e.g. tunnelling, we have
+# to be specific about the network interfaces that nsd binds to.
+for ip in $PRIVATE_IP $PRIVATE_IPV6; do
+	echo "  ip-address: $ip" >> /etc/nsd/nsd.conf;
+done
+
+echo "include: /etc/nsd/zones.conf" >> /etc/nsd/nsd.conf;
+
 # Create DNSSEC signing keys.
 
 mkdir -p "$STORAGE_ROOT/dns/dnssec";

setup/functions.sh

@@ -9,13 +9,15 @@ function hide_output {
 	$@ &> $OUTPUT
 
 	# If the command failed, show the output that was captured in the temporary file.
-	if [ $? != 0 ]; then
+	E=$?
+	if [ $E != 0 ]; then
 		# Something failed.
 		echo
 		echo FAILED: $@
 		echo -----------------------------------------
 		cat $OUTPUT
 		echo -----------------------------------------
+		exit $E
 	fi
 
 	# Remove temporary file.
@@ -180,6 +182,28 @@ function input_menu {
 	result_code=$?
 }
 
+function wget_verify {
+	# Downloads a file from the web and checks that it matches
+	# a provided hash. If the comparison fails, exit immediately.
+	URL=$1
+	HASH=$2
+	DEST=$3
+	CHECKSUM="$HASH  $DEST"
+	rm -f $DEST
+	wget -q -O $DEST $URL || exit 1
+	if ! echo "$CHECKSUM" | sha1sum --check --strict > /dev/null; then
+		echo "------------------------------------------------------------"
+		echo "Download of $URL did not match expected checksum."
+		echo "Found:"
+		sha1sum $DEST
+		echo
+		echo "Expected:"
+		echo "$CHECKSUM"
+		rm -f $DEST
+		exit 1
+	fi
+}
+
 function git_clone {
 	# Clones a git repository, checks out a particular commit or tag,
 	# and moves the repository (or a subdirectory in it) to some path.

setup/mail-dovecot.sh

@@ -1,9 +1,9 @@
 #!/bin/bash
 #
-# Dovecot (IMAP and LDA)
+# Dovecot (IMAP/POP and LDA)
 # ----------------------
 #
-# Dovecot is *both* the IMAP server (the protocol that email applications
+# Dovecot is *both* the IMAP/POP server (the protocol that email applications
 # use to query a mailbox) as well as the local delivery agent (LDA),
 # meaning it is responsible for writing emails to mailbox storage on disk.
 # You could imagine why these things would be bundled together.
@@ -21,10 +21,11 @@ source /etc/mailinabox.conf # load global vars
 # Install packages...
 
 apt_install \
-	dovecot-core dovecot-imapd dovecot-lmtpd dovecot-sqlite sqlite3 \
+	dovecot-core dovecot-imapd dovecot-pop3d dovecot-lmtpd dovecot-sqlite sqlite3 \
 	dovecot-sieve dovecot-managesieved
 
-# The `dovecot-imapd` and `dovecot-lmtpd` packages automatically enable IMAP and LMTP protocols.
+# The `dovecot-imapd`, `dovecot-pop3d`, and `dovecot-lmtpd` packages automatically
+# enable IMAP, POP and LMTP protocols.
 
 # Set basic daemon options.
 
@@ -51,7 +52,7 @@ tools/editconf.py /etc/dovecot/conf.d/10-mail.conf \
 	mail_privileged_group=mail \
 	first_valid_uid=0
 
-# ### IMAP
+# ### IMAP/POP
 
 # Require that passwords are sent over SSL only, and allow the usual IMAP authentication mechanisms.
 # The LOGIN mechanism is supposedly for Microsoft products like Outlook to do SMTP login (I guess
@@ -69,9 +70,9 @@ tools/editconf.py /etc/dovecot/conf.d/10-ssl.conf \
 	"ssl_protocols=!SSLv3 !SSLv2" \
 	"ssl_cipher_list=TLSv1+HIGH !SSLv2 !RC4 !aNULL !eNULL !3DES @STRENGTH"
 
-# Disable in-the-clear IMAP because there is no reason for a user to transmit
-# login credentials outside of an encrypted connection. Although we haven't
-# even installed the POP server, ensure it is disabled too.
+# Disable in-the-clear IMAP/POP because there is no reason for a user to transmit
+# login credentials outside of an encrypted connection. Only the over-TLS versions
+# are made available (IMAPS on port 993; POP3S on port 995).
 sed -i "s/#port = 143/port = 0/" /etc/dovecot/conf.d/10-master.conf
 sed -i "s/#port = 110/port = 0/" /etc/dovecot/conf.d/10-master.conf
 
@@ -84,6 +85,13 @@ sed -i "s/#port = 110/port = 0/" /etc/dovecot/conf.d/10-master.conf
 tools/editconf.py /etc/dovecot/conf.d/20-imap.conf \
 	imap_idle_notify_interval="4 mins"
 
+# Set POP3 UIDL
+# UIDLs are used by POP3 clients to keep track of what messages they've downloaded. 
+# For new POP3 servers, the easiest way to set up UIDLs is to use IMAP's UIDVALIDITY
+# and UID values, the default in Dovecot.
+tools/editconf.py /etc/dovecot/conf.d/20-pop3.conf \
+	pop3_uidl_format="%08Xu%08Xv"
+
 # ### LDA (LMTP)
 
 # Enable Dovecot's LDA service with the LMTP protocol. It will listen
@@ -162,8 +170,9 @@ chown -R mail.mail $STORAGE_ROOT/mail/mailboxes
 mkdir -p $STORAGE_ROOT/mail/sieve
 chown -R mail.mail $STORAGE_ROOT/mail/sieve
 
-# Allow the IMAP port in the firewall.
+# Allow the IMAP/POP ports in the firewall.
 ufw_allow imaps
+ufw_allow pop3s
 
 # Restart services.
 restart_service dovecot

setup/mail-postfix.sh

@@ -65,6 +65,8 @@ tools/editconf.py /etc/postfix/main.cf \
 # * Do not add the OpenDMAC Authentication-Results header. That should only be added
 #   on incoming mail. Omit the OpenDMARC milter by re-setting smtpd_milters to the
 #   OpenDKIM milter only. See dkim.sh.
+# * Even though we dont allow auth over non-TLS connections (smtpd_tls_auth_only below, and without auth the client cant
+#   send outbound mail), don't allow non-TLS mail submission on this port anyway to prevent accidental misconfiguration.
 # * Require the best ciphers for incoming connections per http://baldric.net/2013/12/07/tls-ciphers-in-postfix-and-dovecot/.
 #   By putting this setting here we leave opportunistic TLS on incoming mail at default cipher settings (any cipher is better than none).
 # * Give it a different name in syslog to distinguish it from the port 25 smtpd server.
@@ -75,7 +77,8 @@ tools/editconf.py /etc/postfix/master.cf -s -w \
 	"submission=inet n       -       -       -       -       smtpd
 	  -o syslog_name=postfix/submission
 	  -o smtpd_milters=inet:127.0.0.1:8891
-	  -o smtpd_tls_ciphers=high -o smtpd_tls_protocols=!SSLv2,!SSLv3
+	  -o smtpd_tls_security_level=encrypt
+	  -o smtpd_tls_ciphers=high -o smtpd_tls_exclude_ciphers=aNULL,DES,3DES,MD5,DES+MD5,RC4 -o smtpd_tls_protocols=!SSLv2,!SSLv3
 	  -o cleanup_service_name=authclean" \
 	"authclean=unix  n       -       -       -       0       cleanup
 	  -o header_checks=pcre:/etc/postfix/outgoing_mail_header_filters"
@@ -94,6 +97,8 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtpd_tls_cert_file=$STORAGE_ROOT/ssl/ssl_certificate.pem \
 	smtpd_tls_key_file=$STORAGE_ROOT/ssl/ssl_private_key.pem \
 	smtpd_tls_dh1024_param_file=$STORAGE_ROOT/ssl/dh2048.pem \
+	smtpd_tls_ciphers=medium \
+	smtpd_tls_exclude_ciphers=aNULL \
 	smtpd_tls_received_header=yes
 
 # Prevent non-authenticated users from sending mail that requires being

setup/management.sh

@@ -3,7 +3,8 @@
 source setup/functions.sh
 
 apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil
-hide_output pip3 install rtyaml
+hide_output pip3 install rtyaml "email_validator==0.1.0-rc4"
+	# email_validator is repeated in setup/questions.sh
 
 # Create a backup directory and a random key for encrypting backups.
 mkdir -p $STORAGE_ROOT/backup

setup/migrate.py

@@ -67,6 +67,35 @@ def migration_6(env):
 	basepath = os.path.join(env["STORAGE_ROOT"], 'dns/dnssec')
 	shutil.move(os.path.join(basepath, 'keys.conf'), os.path.join(basepath, 'RSASHA1-NSEC3-SHA1.conf'))
 
+def migration_7(env):
+	# I previously wanted domain names to be stored in Unicode in the database. Now I want them
+	# to be in IDNA. Affects aliases only.
+	import sqlite3
+	conn = sqlite3.connect(os.path.join(env["STORAGE_ROOT"], "mail/users.sqlite"))
+
+	# Get existing alias source addresses.
+	c = conn.cursor()
+	c.execute('SELECT source FROM aliases')
+	aliases = [ row[0] for row in c.fetchall() ]
+
+	# Update to IDNA-encoded domains.
+	for email in aliases:
+		try:
+			localpart, domainpart = email.split("@")
+			domainpart = domainpart.encode("idna").decode("ascii")
+			newemail = localpart + "@" + domainpart
+			if newemail != email:
+				c = conn.cursor()
+				c.execute("UPDATE aliases SET source=? WHERE source=?", (newemail, email))
+				if c.rowcount != 1: raise ValueError("Alias not found.")
+				print("Updated alias", email, "to", newemail)
+		except Exception as e:
+			print("Error updating IDNA alias", email, e)
+
+	# Save.
+	conn.commit()
+
+
 def get_current_migration():
 	ver = 0
 	while True:

setup/owncloud.sh

@@ -16,6 +16,7 @@ apt-get purge -qq -y owncloud*
 
 # Install ownCloud from source of this version:
 owncloud_ver=8.0.2
+owncloud_hash=a4d1fc44bc40af87948458ae8f60ee427ecd9560
 
 # Check if ownCloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
 if [ ! -d /usr/local/lib/owncloud/ ] \
@@ -32,8 +33,7 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 	fi
 
 	# Download and extract ownCloud.
-	rm -f /tmp/owncloud.zip
-	wget -qO /tmp/owncloud.zip https://download.owncloud.org/community/owncloud-$owncloud_ver.zip
+	wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
 	unzip -u -o -q /tmp/owncloud.zip -d /usr/local/lib #either extracts new or replaces current files
 	rm -f /tmp/owncloud.zip
 

setup/questions.sh

@@ -4,7 +4,14 @@ if [ -z "$NONINTERACTIVE" ]; then
 	# e.g. if we piped a bootstrapping install script to bash to get started. In that
 	# case, the nifty '[ -t 0 ]' test won't work. But with Vagrant we must suppress so we
 	# use a shell flag instead. Really supress any output from installing dialog.
-	apt_get_quiet install dialog
+	#
+	# Also install depencies needed to validate the email address.
+	echo Installing packages needed for setup...
+	apt_get_quiet install dialog python3 python3-pip  || exit 1
+
+	# email_validator is repeated in setup/management.sh
+	hide_output pip3 install "email_validator==0.1.0-rc4" || exit 1
+
 	message_box "Mail-in-a-Box Installation" \
 		"Hello and thanks for deploying a Mail-in-a-Box!
 		\n\nI'm going to ask you a few questions.

setup/spamassassin.sh

@@ -25,10 +25,14 @@ tools/editconf.py /etc/default/spamassassin \
 # Configure pyzor.
 hide_output pyzor discover
 
-# Pass messages on to docevot on port 10026.
-# This is actually the default setting but we don't want to lose track of it.
-# We've already configured Dovecot to listen on this port.
-tools/editconf.py /etc/default/spampd DESTPORT=10026
+# Configure spampd:
+# * Pass messages on to docevot on port 10026. This is actually the default setting but we don't
+#   want to lose track of it. (We've configured Dovecot to listen on this port elsewhere.)
+# * Increase the maximum message size of scanned messages from the default of 64KB to 500KB, which
+#   is Spamassassin (spamc)'s own default. Specified in KBytes.
+tools/editconf.py /etc/default/spampd \
+	DESTPORT=10026 \
+	ADDOPTS="\"--maxsize=500\""
 
 # Spamassassin normally wraps spam as an attachment inside a fresh
 # email with a report about the message. This also protects the user
@@ -71,6 +75,7 @@ chown -R spampd:spampd $STORAGE_ROOT/mail/spamassassin
 # Enable the Dovecot antispam plugin.
 # (Be careful if we use multiple plugins later.) #NODOC
 sed -i "s/#mail_plugins = .*/mail_plugins = \$mail_plugins antispam/" /etc/dovecot/conf.d/20-imap.conf
+sed -i "s/#mail_plugins = .*/mail_plugins = \$mail_plugins antispam/" /etc/dovecot/conf.d/20-pop3.conf
 
 # Configure the antispam plugin to call sa-learn-pipe.sh.
 cat > /etc/dovecot/conf.d/99-local-spampd.conf << EOF;

setup/start.sh

@@ -27,7 +27,7 @@ export LC_TYPE=en_US.UTF-8
 if [ -f /etc/mailinabox.conf ]; then
 	# Run any system migrations before proceeding. Since this is a second run,
 	# we assume we have Python already installed.
-	setup/migrate.py --migrate
+	setup/migrate.py --migrate || exit 1
 
 	# Load the old .conf file to get existing configuration options loaded
 	# into variables with a DEFAULT_ prefix.

setup/webmail.sh

@@ -32,8 +32,9 @@ apt-get purge -qq -y roundcube* #NODOC
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of vacation_sieve to track
 # whether we have the latest version.
-VERSION=1.1.0
-VACATION_SIEVE_VERSION=06a20e9d44db62259ae41fd8451f3c937d3ab4f3
+VERSION=1.1.1
+HASH=08222f382a8dd89bba7dbbad595f48443bec0aa2
+VACATION_SIEVE_VERSION=91ea6f52216390073d1f5b70b5f6bea0bfaee7e5
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/roundcubemail/version ]; then
 	# not installed yet #NODOC
@@ -45,8 +46,10 @@ fi
 if [ $needs_update == 1 ]; then
 	# install roundcube
 	echo installing Roundcube webmail $VERSION...
-	rm -f /tmp/roundcube.tgz
-	wget -qO /tmp/roundcube.tgz http://downloads.sourceforge.net/project/roundcubemail/roundcubemail/$VERSION/roundcubemail-$VERSION.tar.gz
+	wget_verify \
+		http://downloads.sourceforge.net/project/roundcubemail/roundcubemail/$VERSION/roundcubemail-$VERSION.tar.gz \
+		$HASH \
+		/tmp/roundcube.tgz
 	tar -C /usr/local/lib -zxf /tmp/roundcube.tgz
 	rm -rf /usr/local/lib/roundcubemail
 	mv /usr/local/lib/roundcubemail-$VERSION/ /usr/local/lib/roundcubemail

tools/editconf.py

@@ -54,6 +54,14 @@ while settings[0][0] == "-" and settings[0] != "--":
 		print("Invalid option.")
 		sys.exit(1)
 
+# sanity check command line
+for setting in settings:
+	try:
+		name, value = setting.split("=", 1)
+	except:
+		import subprocess
+		print("Invalid command line: ", subprocess.list2cmdline(sys.argv))
+
 # create the new config file in memory
 
 found = set()

tools/mail.py

@@ -1,6 +1,6 @@
 #!/usr/bin/python3
 
-import sys, getpass, urllib.request, urllib.error, json
+import sys, getpass, urllib.request, urllib.error, json, re
 
 def mgmt(cmd, data=None, is_json=False):
 	# The base URL for the management daemon. (Listens on IPv4 only.)
@@ -31,11 +31,14 @@ def read_password():
     while True:
         first = getpass.getpass('password: ')
         if len(first) < 4:
-            print('Passwords must be at least four characters.')
+            print("Passwords must be at least four characters.")
+            continue
+        if re.search(r'[\s]', first):
+            print("Passwords cannot contain spaces.")
             continue
         second = getpass.getpass(' (again): ')
         if first != second:
-            print('Passwords not the same. Try again.')
+            print("Passwords not the same. Try again.")
             continue
         break
     return first