v0.14

v0.14 (November 4, 2015)
------------------------

Mail:

* Spamassassin's network-based tests (Pyzor, others) and DKIM tests are now enabled. (Pyzor had always been installed but was not active due to a misconfiguration.)
* Moving spam out of the Spam folder and into Trash would incorrectly train Spamassassin that those messages were not spam.
* Automatically create the Sent and Archive folders for new users.
* The HTML5_Notifier plugin for Roundcube is now included, which when turned on in Roundcube settings provides desktop notifications for new mail.
* The Exchange/ActiveSync backend Z-Push has been updated to fix a problem with CC'd emails not being sent to the CC recipients.

Calender/Contacts:

* CalDAV/CardDAV and Exchange/ActiveSync for calendar/contacts wasn't working in some network configurations.

Web:

* When a new domain is added to the box, rather than applying a new self-signed certificate for that domain, the SSL certificate for the box's primary hostname will be used instead.
* If a custom DNS record is set on a domain or 'www'+domain, web would not be served for that domain. If the custom DNS record is just the box's IP address, that's a configuration mistake, but allow it and let web continue to be served.
* Accommodate really long domain names by increasing an nginx setting.

Control panel:

* Added an option to check for new Mail-in-a-Box versions within status checks. It is off by default so that boxes don't "phone home" without permission.
* Added a random password generator on the users page to simplify creating new accounts.
* When S3 backup credentials are set, the credentials are now no longer ever sent back from the box to the client, for better security.
* Fixed the jumpiness when a modal is displayed.
* Focus is put into the login form fields when the login form is displayed.
* Status checks now include a warning if a custom DNS record has been set on a domain that would normally serve web and as a result that domain no longer is serving web.
* Status checks now check that secondary nameservers, if specified, are actually serving the domains.
* Some errors in the control panel when there is invalid data in the database or an improperly named archived user account have been suppressed.
* Added subresource integrity attributes to all remotely-sourced resources (i.e. via CDNs) to guard against CDNs being used as an attack vector.

System:

* Tweaks to fail2ban settings.
* Fixed a spurrious warning while installing munin.

CHANGELOG.md

@@ -1,6 +1,44 @@
 CHANGELOG
 =========
 
+v0.14 (November 4, 2015)
+------------------------
+
+Mail:
+
+* Spamassassin's network-based tests (Pyzor, others) and DKIM tests are now enabled. (Pyzor had always been installed but was not active due to a misconfiguration.)
+* Moving spam out of the Spam folder and into Trash would incorrectly train Spamassassin that those messages were not spam.
+* Automatically create the Sent and Archive folders for new users.
+* The HTML5_Notifier plugin for Roundcube is now included, which when turned on in Roundcube settings provides desktop notifications for new mail.
+* The Exchange/ActiveSync backend Z-Push has been updated to fix a problem with CC'd emails not being sent to the CC recipients.
+
+Calender/Contacts:
+
+* CalDAV/CardDAV and Exchange/ActiveSync for calendar/contacts wasn't working in some network configurations.
+
+Web:
+
+* When a new domain is added to the box, rather than applying a new self-signed certificate for that domain, the SSL certificate for the box's primary hostname will be used instead.
+* If a custom DNS record is set on a domain or 'www'+domain, web would not be served for that domain. If the custom DNS record is just the box's IP address, that's a configuration mistake, but allow it and let web continue to be served.
+* Accommodate really long domain names by increasing an nginx setting.
+
+Control panel:
+
+* Added an option to check for new Mail-in-a-Box versions within status checks. It is off by default so that boxes don't "phone home" without permission.
+* Added a random password generator on the users page to simplify creating new accounts.
+* When S3 backup credentials are set, the credentials are now no longer ever sent back from the box to the client, for better security.
+* Fixed the jumpiness when a modal is displayed.
+* Focus is put into the login form fields when the login form is displayed.
+* Status checks now include a warning if a custom DNS record has been set on a domain that would normally serve web and as a result that domain no longer is serving web.
+* Status checks now check that secondary nameservers, if specified, are actually serving the domains.
+* Some errors in the control panel when there is invalid data in the database or an improperly named archived user account have been suppressed.
+* Added subresource integrity attributes to all remotely-sourced resources (i.e. via CDNs) to guard against CDNs being used as an attack vector.
+
+System:
+
+* Tweaks to fail2ban settings.
+* Fixed a spurrious warning while installing munin.
+
 v0.13b (August 30, 2015)
 ------------------------
 

README.md

@@ -54,23 +54,25 @@ Clone this repository:
 	$ cd mailinabox
 
 _Optional:_ Download my PGP key and then verify that the sources were signed
-by me. You'll get a lot of warnings, but the fingerprint should match the
-fingerprint in the key details at [https://keybase.io/joshdata](https://keybase.io/joshdata)
-and on my [personal homepage](https://razor.occams.info/). (Of course, if this repository has been compromised you can't trust these instructions anyway.)
+by me:
 
 	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
 	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported
 
-	$ git verify-tag v0.13
+	$ git verify-tag v0.14
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
 	gpg:          There is no indication that the signature belongs to the owner.
 	Primary key fingerprint: 5F4C 0E73 13CC D744 693B  2AEA B920 41F4 C10B DD81
 
+You'll get a lot of warnings, but that's OK. Check that the primary key fingerprint matchs the
+fingerprint in the key details at [https://keybase.io/joshdata](https://keybase.io/joshdata)
+and on my [personal homepage](https://razor.occams.info/). (Of course, if this repository has been compromised you can't trust these instructions.)
+
 Checkout the tag corresponding to the most recent release:
 
-	$ git checkout v0.13
+	$ git checkout v0.14
 
 Begin the installation.
 

conf/dovecot-mailboxes.conf

@@ -0,0 +1,63 @@
+## NOTE: This file is automatically generated by Mail-in-a-Box.
+##       Do not edit this file. It is continually updated by
+##       Mail-in-a-Box and your changes will be lost.
+##
+##       Mail-in-a-Box machines are not meant to be modified.
+##       If you modify any system configuration you are on
+##       your own --- please do not ask for help from us.
+
+namespace inbox {
+  # Automatically create & subscribe some folders.
+  # * Create and subscribe the INBOX folder.
+  # * Our sieve rule for spam expects that the Spam folder exists.
+  # * Z-Push must be configured with the same settings in conf/zpush/backend_imap.php (#580).
+
+  # MUA notes:
+  # * Roundcube will show an error if the user tries to delete a message before the Trash folder exists (#359).
+  # * K-9 mail will poll every 90 seconds if a Drafts folder does not exist.
+  # * Apple's OS X Mail app will create 'Sent Messages' if it doesn't see a folder with the \Sent flag (#571, #573) and won't be able to archive messages unless 'Archive' exists (#581).
+  # * Thunderbird's default in its UI is 'Archives' (plural) but it will configure new accounts to use whatever we say here (#581).
+
+  # auto:
+  # 'create' will automatically create this mailbox.
+  # 'subscribe' will both create and subscribe to the mailbox.
+
+  # special_use is a space separated list of IMAP SPECIAL-USE
+  # attributes as specified by RFC 6154:
+  # \All \Archive \Drafts \Flagged \Junk \Sent \Trash
+
+  mailbox INBOX {
+    auto = subscribe
+  }
+  mailbox Spam {
+    special_use = \Junk
+    auto = subscribe
+  }
+  mailbox Drafts {
+    special_use = \Drafts
+    auto = subscribe
+  }
+  mailbox Sent {
+    special_use = \Sent
+    auto = subscribe
+  }
+  mailbox Trash {
+    special_use = \Trash
+    auto = subscribe
+  }
+  mailbox Archive {
+    special_use = \Archive
+    auto = subscribe
+  }
+
+  # dovevot's standard mailboxes configuration file marks two sent folders
+  # with the \Sent attribute, just in case clients don't agree about which
+  # they're using. We'll keep that, plus add Junk as an alterative for Spam.
+  # These are not auto-created.
+  mailbox "Sent Messages" {
+    special_use = \Sent
+  }
+  mailbox Junk {
+    special_use = \Junk
+  }
+}

conf/fail2ban/jail.local

@@ -2,6 +2,10 @@
 
 # JAILS
 
+[ssh]
+maxretry = 7
+bantime = 3600
+
 [ssh-ddos]
 enabled  = true
 

conf/nginx-primaryonly.conf

@@ -50,7 +50,7 @@
 	location ~ ^/((caldav|carddav|webdav).*)$ {
 		# Z-Push doesn't like getting a redirect, and a plain rewrite didn't work either.
 		# Properly proxying like this seems to work fine.
-		proxy_pass https://$HOSTNAME/cloud/remote.php/$1;
+		proxy_pass https://127.0.0.1/cloud/remote.php/$1;
 	}
 	rewrite ^/.well-known/host-meta /cloud/public.php?service=host-meta last;
 	rewrite ^/.well-known/host-meta.json /cloud/public.php?service=host-meta-json last;

conf/nginx-ssl.conf

@@ -16,7 +16,9 @@
 #ssl_certificate_key /path/to/my-private-decrypted.key;
 
 # Tell browsers to require SSL (warning: difficult to change your mind)
-add_header Strict-Transport-Security max-age=31536000;
+# Handled by the management daemon because we can toggle this version or a
+# preload version.
+#add_header Strict-Transport-Security max-age=31536000;
 
 # Prefer certain ciphersuites, to enforce Forward Secrecy and avoid known vulnerabilities.
 # 
@@ -36,7 +38,7 @@ ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
 # as recommended by http://nginx.org/en/docs/http/configuring_https_servers.html
 ssl_session_cache   shared:SSL:10m;
 ssl_session_timeout 10m;
-keepalive_timeout   70;
+#keepalive_timeout   70; # in Ubuntu 14.04/nginx 1.4.6 the default is 65, so plenty good
 
 # Buffer size of 1400 bytes fits in one MTU.
 # nginx 1.5.9+ ONLY

conf/nginx.conf

@@ -31,7 +31,6 @@ server {
 
 	ssl_certificate $SSL_CERTIFICATE;
 	ssl_certificate_key $SSL_KEY;
-	include /etc/nginx/nginx-ssl.conf;
 
 	# ADDITIONAL DIRECTIVES HERE
 }

conf/zpush/backend_caldav.php

@@ -5,10 +5,12 @@
 * Descr     :   CalDAV backend configuration file
 ************************************************/
 
-define('CALDAV_SERVER', 'https://localhost');
+define('CALDAV_PROTOCOL', 'https');
+define('CALDAV_SERVER', 'localhost');
 define('CALDAV_PORT', '443');
 define('CALDAV_PATH', '/caldav/calendars/%u/');
-define('CALDAV_PERSONAL', '');
-define('CALDAV_SUPPORTS_SYNC', true);
+define('CALDAV_PERSONAL', 'PRINCIPAL');
+define('CALDAV_SUPPORTS_SYNC', false);
+define('CALDAV_MAX_SYNC_PERIOD', 2147483647);
 
 ?>

conf/zpush/backend_imap.php

@@ -10,6 +10,20 @@ define('IMAP_PORT', 993);
 define('IMAP_OPTIONS', '/ssl/norsh/novalidate-cert');
 define('IMAP_DEFAULTFROM', '');
 
+define('SYSTEM_MIME_TYPES_MAPPING', '/etc/mime.types');
+define('IMAP_AUTOSEEN_ON_DELETE', false);
+define('IMAP_FOLDER_CONFIGURED', true);
+define('IMAP_FOLDER_PREFIX', '');
+define('IMAP_FOLDER_PREFIX_IN_INBOX', false);
+// see our conf/dovecot-mailboxes.conf file for IMAP special flags settings
+define('IMAP_FOLDER_INBOX', 'INBOX');
+define('IMAP_FOLDER_SENT', 'SENT');
+define('IMAP_FOLDER_DRAFT', 'DRAFTS');
+define('IMAP_FOLDER_TRASH', 'TRASH');
+define('IMAP_FOLDER_SPAM', 'SPAM');
+define('IMAP_FOLDER_ARCHIVE', 'ARCHIVE');
+
+
 // not used
 define('IMAP_FROM_SQL_DSN', '');
 define('IMAP_FROM_SQL_USER', '');
@@ -28,12 +42,6 @@ define('IMAP_FROM_LDAP_FIELDS', serialize(array('givenname', 'sn', 'mail')));
 define('IMAP_FROM_LDAP_FROM', '#givenname #sn <#mail>');
 
 
-// copy outgoing mail to this folder. If not set z-push will try the default folders
-define('IMAP_SENTFOLDER', '');
-define('IMAP_INLINE_FORWARD', true);
-define('IMAP_EXCLUDED_FOLDERS', '');
-define('IMAP_SMTP_METHOD', 'sendmail');
-
 global $imap_smtp_params;
 $imap_smtp_params = array('host' => 'ssl://localhost', 'port' => 587, 'auth' => true, 'username' => 'imap_username', 'password' => 'imap_password');
 

management/backup.py

@@ -384,9 +384,9 @@ def backup_set_custom(env, target, target_user, target_pass, min_age):
 	
 	write_backup_config(env, config)
 
-	return "Updated backup config"
+	return "OK"
 	
-def get_backup_config(env, for_save=False):
+def get_backup_config(env, for_save=False, for_ui=False):
 	backup_root = os.path.join(env["STORAGE_ROOT"], 'backup')
 
 	# Defaults.
@@ -407,6 +407,13 @@ def get_backup_config(env, for_save=False):
 	if for_save:
 		return config
 
+	# When passing this back to the admin to show the current settings, do not include
+	# authentication details. The user will have to re-enter it.
+	if for_ui:
+		for field in ("target_user", "target_pass"):
+			if field in config:
+				del config[field]
+
 	# helper fields for the admin
 	config["file_target_directory"] = os.path.join(backup_root, 'encrypted')
 	config["enc_pw_file"] = os.path.join(backup_root, 'secret_key.txt')

management/daemon.py

@@ -318,9 +318,9 @@ def dns_get_dump():
 @app.route('/ssl/csr/<domain>', methods=['POST'])
 @authorized_personnel_only
 def ssl_get_csr(domain):
-	from web_update import get_domain_ssl_files, create_csr
-	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
-	return create_csr(domain, ssl_key, env)
+	from web_update import create_csr
+	ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
+	return create_csr(domain, ssl_private_key, env)
 
 @app.route('/ssl/install', methods=['POST'])
 @authorized_personnel_only
@@ -413,7 +413,7 @@ def backup_status():
 @authorized_personnel_only
 def backup_get_custom():
 	from backup import get_backup_config
-	return json_response(get_backup_config(env))
+	return json_response(get_backup_config(env, for_ui=True))
 
 @app.route('/system/backup/config', methods=["POST"])
 @authorized_personnel_only
@@ -426,6 +426,20 @@ def backup_set_custom():
 		request.form.get('min_age', '')
 	))
 
+@app.route('/system/privacy', methods=["GET"])
+@authorized_personnel_only
+def privacy_status_get():
+	config = utils.load_settings(env)
+	return json_response(config.get("privacy", True))
+
+@app.route('/system/privacy', methods=["POST"])
+@authorized_personnel_only
+def privacy_status_set():
+	config = utils.load_settings(env)
+	config["privacy"] = (request.form.get('value') == "private")
+	utils.write_settings(config, env)
+	return "OK"
+
 # MUNIN
 
 @app.route('/munin/')

management/dns_update.py

@@ -120,6 +120,10 @@ def do_dns_update(env, force=False):
 			# If this is the only thing that changed?
 			updated_domains.append("OpenDKIM configuration")
 
+	# Clear bind9's DNS cache so our own DNS resolver is up to date.
+	# (ignore errors with trap=True)
+	shell('check_call', ["/usr/sbin/rndc", "flush"], trap=True)
+
 	if len(updated_domains) == 0:
 		# if nothing was updated (except maybe OpenDKIM's files), don't show any output
 		return ""
@@ -847,6 +851,12 @@ def set_secondary_dns(hostnames, env):
 	return do_dns_update(env)
 
 
+def get_custom_dns_record(custom_dns, qname, rtype):
+	for qname1, rtype1, value in custom_dns:
+		if qname1 == qname and rtype1 == rtype:
+			return value
+	return None
+
 ########################################################################
 
 def justtestingdotemail(domain, records):

management/mail_log.py

@@ -1,5 +1,6 @@
 #!/usr/bin/python3
 
+from collections import defaultdict
 import re, os.path
 import dateutil.parser
 
@@ -12,6 +13,7 @@ def scan_mail_log(logger, env):
 		"imap-logins": { },
 		"postgrey": { },
 		"rejected-mail": { },
+		"activity-by-hour": { "imap-logins": defaultdict(int), "smtp-sends": defaultdict(int) },
 	}
 
 	collector["real_mail_addresses"] = set(mailconfig.get_mail_users(env)) | set(alias[0] for alias in mailconfig.get_mail_aliases(env))
@@ -45,6 +47,10 @@ def scan_mail_log(logger, env):
 			for date, sender, message in collector["rejected-mail"][k]:
 				logger.print_line(k + "\t" + str(date) + "\t" + sender + "\t" + message)
 
+	logger.add_heading("Activity by Hour")
+	for h in range(24):
+		logger.print_line("%d\t%d\t%d" % (h, collector["activity-by-hour"]["imap-logins"][h], collector["activity-by-hour"]["smtp-sends"][h] ))
+
 	if len(collector["other-services"]) > 0:
 		logger.add_heading("Other")
 		logger.print_block("Unrecognized services in the log: " + ", ".join(collector["other-services"]))
@@ -65,6 +71,9 @@ def scan_mail_log_line(line, collector):
 	elif service == "postfix/smtpd":
 		scan_postfix_smtpd_line(date, log, collector)
 
+	elif service == "postfix/submission/smtpd":
+		scan_postfix_submission_line(date, log, collector)
+
 	elif service in ("postfix/qmgr", "postfix/pickup", "postfix/cleanup",
 			"postfix/scache", "spampd", "postfix/anvil", "postfix/master",
 			"opendkim", "postfix/lmtp", "postfix/tlsmgr"):
@@ -80,6 +89,7 @@ def scan_dovecot_line(date, log, collector):
 		login, ip = m.group(1), m.group(2)
 		if ip != "127.0.0.1": # local login from webmail/zpush
 			collector["imap-logins"].setdefault(login, {})[ip] = date
+		collector["activity-by-hour"]["imap-logins"][date.hour] += 1
 
 def scan_postgrey_line(date, log, collector):
 	m = re.match("action=(greylist|pass), reason=(.*?), (?:delay=\d+, )?client_name=(.*), client_address=(.*), sender=(.*), recipient=(.*)", log)
@@ -114,6 +124,11 @@ def scan_postfix_smtpd_line(date, log, collector):
 
 			collector["rejected-mail"].setdefault(recipient, []).append( (date, sender, message) )
 
+def scan_postfix_submission_line(date, log, collector):
+	m = re.match("([A-Z0-9]+): client=(\S+), sasl_method=PLAIN, sasl_username=(\S+)", log)
+	if m:
+		procid, client, user = m.groups()
+		collector["activity-by-hour"]["smtp-sends"][date.hour] += 1
 
 if __name__ == "__main__":
 	from status_checks import ConsoleOutput

management/mailconfig.py

@@ -244,7 +244,13 @@ def get_domain(emailaddr, as_unicode=True):
 	# Gets the domain part of an email address. Turns IDNA
 	# back to Unicode for display.
 	ret = emailaddr.split('@', 1)[1]
-	if as_unicode: ret = idna.decode(ret.encode('ascii'))
+	if as_unicode:
+		try:
+			ret = idna.decode(ret.encode('ascii'))
+		except (ValueError, UnicodeError, idna.IDNAError):
+			# Looks like we have an invalid email address in
+			# the database. Now is not the time to complain.
+			pass
 	return ret
 
 def get_mail_domains(env, filter_aliases=lambda alias : True):
@@ -297,25 +303,6 @@ def add_mail_user(email, pw, privs, env):
 	# write databasebefore next step
 	conn.commit()
 
-	# Create & subscribe the user's INBOX, Trash, Spam, and Drafts folders.
-	# * Our sieve rule for spam expects that the Spam folder exists.
-	# * Roundcube will show an error if the user tries to delete a message before the Trash folder exists (#359).
-	# * K-9 mail will poll every 90 seconds if a Drafts folder does not exist, so create it
-	#   to avoid unnecessary polling.
-
-	# Check if the mailboxes exist before creating them. When creating a user that had previously
-	# been deleted, the mailboxes will still exist because they are still on disk.
-	try:
-		existing_mboxes = utils.shell('check_output', ["doveadm", "mailbox", "list", "-u", email, "-8"], capture_stderr=True).split("\n")
-	except subprocess.CalledProcessError as e:
-		c.execute("DELETE FROM users WHERE email=?", (email,))
-		conn.commit()
-		return ("Failed to initialize the user: " + e.output.decode("utf8"), 400)
-
-	for folder in ("INBOX", "Trash", "Spam", "Drafts"):
-		if folder not in existing_mboxes:
-			utils.shell('check_call', ["doveadm", "mailbox", "create", "-u", email, "-s", folder])
-
 	# Update things in case any new domains are added.
 	return kick(env, "mail user added")
 

management/status_checks.py

@@ -12,11 +12,11 @@ import dns.reversename, dns.resolver
 import dateutil.parser, dateutil.tz
 import idna
 
-from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config, get_secondary_dns
-from web_update import get_web_domains, get_default_www_redirects, get_domain_ssl_files
+from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config, get_secondary_dns, get_custom_dns_record
+from web_update import get_web_domains, get_default_www_redirects, get_ssl_certificates, get_domain_ssl_files, get_domains_with_a_records
 from mailconfig import get_mail_domains, get_mail_aliases
 
-from utils import shell, sort_domains, load_env_vars_from_file
+from utils import shell, sort_domains, load_env_vars_from_file, load_settings
 
 def run_checks(rounded_values, env, output, pool):
 	# run systems checks
@@ -149,6 +149,7 @@ def check_service(i, service, env):
 def run_system_checks(rounded_values, env, output):
 	check_ssh_password(env, output)
 	check_software_updates(env, output)
+	check_miab_version(env, output)
 	check_system_aliases(env, output)
 	check_free_disk_space(rounded_values, env, output)
 
@@ -244,23 +245,34 @@ def run_domain_checks(rounded_time, env, output, pool):
 
 	domains_to_check = mail_domains | dns_domains | web_domains
 
+	# Get the list of domains that we don't serve web for because of a custom CNAME/A record.
+	domains_with_a_records = get_domains_with_a_records(env)
+
+	ssl_certificates = get_ssl_certificates(env)
+
 	# Serial version:
 	#for domain in sort_domains(domains_to_check, env):
 	#	run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains)
 
 	# Parallelize the checks across a worker pool.
-	args = ((domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains)
+	args = ((domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains, domains_with_a_records, ssl_certificates)
 		for domain in domains_to_check)
 	ret = pool.starmap(run_domain_checks_on_domain, args, chunksize=1)
 	ret = dict(ret) # (domain, output) => { domain: output }
 	for domain in sort_domains(ret, env):
 		ret[domain].playback(output)
 
-def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains):
+def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains, domains_with_a_records, ssl_certificates):
 	output = BufferedOutput()
 
-	# The domain is IDNA-encoded, but for display use Unicode.
-	output.add_heading(idna.decode(domain.encode('ascii')))
+	# The domain is IDNA-encoded in the database, but for display use Unicode.
+	try:
+		domain_display = idna.decode(domain.encode('ascii'))
+		output.add_heading(domain_display)
+	except (ValueError, UnicodeError, idna.IDNAError) as e:
+		# Looks like we have some invalid data in our database.
+		output.add_heading(domain)
+		output.print_error("Domain name is invalid: " + str(e))
 
 	if domain == env["PRIMARY_HOSTNAME"]:
 		check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles)
@@ -272,10 +284,10 @@ def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zone
 		check_mail_domain(domain, env, output)
 
 	if domain in web_domains:
-		check_web_domain(domain, rounded_time, env, output)
+		check_web_domain(domain, rounded_time, ssl_certificates, env, output)
 
 	if domain in dns_domains:
-		check_dns_zone_suggestions(domain, env, output, dns_zonefiles)
+		check_dns_zone_suggestions(domain, env, output, dns_zonefiles, domains_with_a_records)
 
 	return (domain, output)
 
@@ -368,17 +380,24 @@ def check_dns_zone(domain, env, output, dns_zonefiles):
 
 	# We provide a DNS zone for the domain. It should have NS records set up
 	# at the domain name's registrar pointing to this box. The secondary DNS
-	# server may be customized. Unfortunately this may not check the domain's
-	# whois information -- we may be getting the NS records from us rather than
-	# the TLD, and so we're not actually checking the TLD. For that we'd need
-	# to do a DNS trace.
-	ip = query_dns(domain, "A")
-	secondary_ns = get_secondary_dns(get_custom_dns_config(env), mode="NS") or ["ns2." + env['PRIMARY_HOSTNAME']]
+	# server may be customized.
+	# (I'm not sure whether this necessarily tests the TLD's configuration,
+	# as it should, or if one successful NS line at the TLD will result in
+	# this query being answered by the box, which would mean the test is only
+	# half working.)
+
+	custom_dns_records = list(get_custom_dns_config(env)) # generator => list so we can reuse it
+	correct_ip = get_custom_dns_record(custom_dns_records, domain, "A") or env['PUBLIC_IP']
+	custom_secondary_ns = get_secondary_dns(custom_dns_records, mode="NS")
+	secondary_ns = custom_secondary_ns or ["ns2." + env['PRIMARY_HOSTNAME']]
+
 	existing_ns = query_dns(domain, "NS")
 	correct_ns = "; ".join(sorted(["ns1." + env['PRIMARY_HOSTNAME']] + secondary_ns))
+	ip = query_dns(domain, "A")
+
 	if existing_ns.lower() == correct_ns.lower():
 		output.print_ok("Nameservers are set correctly at registrar. [%s]" % correct_ns)
-	elif ip == env['PUBLIC_IP']:
+	elif ip == correct_ip:
 		# The domain resolves correctly, so maybe the user is using External DNS.
 		output.print_warning("""The nameservers set on this domain at your domain name registrar should be %s. They are currently %s.
 			If you are using External DNS, this may be OK."""
@@ -388,7 +407,33 @@ def check_dns_zone(domain, env, output, dns_zonefiles):
 			control panel to set the nameservers to %s."""
 				% (existing_ns, correct_ns) )
 
-def check_dns_zone_suggestions(domain, env, output, dns_zonefiles):
+	# Check that each custom secondary nameserver resolves the IP address.
+	
+	if custom_secondary_ns:
+		for ns in custom_secondary_ns:
+			# We must first resolve the nameserver to an IP address so we can query it.
+			ns_ip = query_dns(ns, "A")
+			if not ns_ip:
+				output.print_error("Secondary nameserver %s is not valid (it doesn't resolve to an IP address)." % ns)
+				continue
+
+			# Now query it to see what it says about this domain.
+			ip = query_dns(domain, "A", at=ns_ip, nxdomain=None)
+			if ip == correct_ip:
+				output.print_ok("Secondary nameserver %s resolved the domain correctly." % ns)
+			elif ip is None:
+				output.print_error("Secondary nameserver %s is not configured to resolve this domain." % ns)
+			else:
+				output.print_error("Secondary nameserver %s is not configured correctly. (It resolved this domain as %s. It should be %s.)" % (ns, ip, env['PUBLIC_IP']))
+
+def check_dns_zone_suggestions(domain, env, output, dns_zonefiles, domains_with_a_records):
+	# Warn if a custom DNS record is preventing this or the automatic www redirect from
+	# being served.
+	if domain in domains_with_a_records:
+		output.print_warning("""Web has been disabled for this domain because you have set a custom DNS record.""")
+	if "www." + domain in domains_with_a_records:
+		output.print_warning("""A redirect from 'www.%s' has been disabled for this domain because you have set a custom DNS record on the www subdomain.""" % domain)
+
 	# Since DNSSEC is optional, if a DS record is NOT set at the registrar suggest it.
 	# (If it was set, we did the check earlier.)
 	if query_dns(domain, "DS", nxdomain=None) is None:
@@ -399,7 +444,9 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 	# See if the domain has a DS record set at the registrar. The DS record may have
 	# several forms. We have to be prepared to check for any valid record. We've
 	# pre-generated all of the valid digests --- read them in.
-	ds_correct = open('/etc/nsd/zones/' + dns_zonefiles[domain] + '.ds').read().strip().split("\n")
+	ds_file = '/etc/nsd/zones/' + dns_zonefiles[domain] + '.ds'
+	if not os.path.exists(ds_file): return # Domain is in our database but DNS has not yet been updated.
+	ds_correct = open(ds_file).read().strip().split("\n")
 	digests = { }
 	for rr_ds in ds_correct:
 		ds_keytag, ds_alg, ds_digalg, ds_digest = rr_ds.split("\t")[4].split(" ")
@@ -509,7 +556,7 @@ def check_mail_domain(domain, env, output):
 			which may prevent recipients from receiving your mail.
 			See http://www.spamhaus.org/dbl/ and http://www.spamhaus.org/query/domain/%s.""" % (dbl, domain))
 
-def check_web_domain(domain, rounded_time, env, output):
+def check_web_domain(domain, rounded_time, ssl_certificates, env, output):
 	# See if the domain's A record resolves to our PUBLIC_IP. This is already checked
 	# for PRIMARY_HOSTNAME, for which it is required for mail specifically. For it and
 	# other domains, it is required to access its website.
@@ -525,9 +572,9 @@ def check_web_domain(domain, rounded_time, env, output):
 	# We need a SSL certificate for PRIMARY_HOSTNAME because that's where the
 	# user will log in with IMAP or webmail. Any other domain we serve a
 	# website for also needs a signed certificate.
-	check_ssl_cert(domain, rounded_time, env, output)
+	check_ssl_cert(domain, rounded_time, ssl_certificates, env, output)
 
-def query_dns(qname, rtype, nxdomain='[Not Set]'):
+def query_dns(qname, rtype, nxdomain='[Not Set]', at=None):
 	# Make the qname absolute by appending a period. Without this, dns.resolver.query
 	# will fall back a failed lookup to a second query with this machine's hostname
 	# appended. This has been causing some false-positive Spamhaus reports. The
@@ -536,9 +583,17 @@ def query_dns(qname, rtype, nxdomain='[Not Set]'):
 	if isinstance(qname, str):
 		qname += "."
 
+	# Use the default nameservers (as defined by the system, which is our locally
+	# running bind server), or if the 'at' argument is specified, use that host
+	# as the nameserver.
+	resolver = dns.resolver.get_default_resolver()
+	if at:
+		resolver = dns.resolver.Resolver()
+		resolver.nameservers = [at]
+
 	# Do the query.
 	try:
-		response = dns.resolver.query(qname, rtype)
+		response = resolver.query(qname, rtype)
 	except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer):
 		# Host did not have an answer for this query; not sure what the
 		# difference is between the two exceptions.
@@ -552,19 +607,24 @@ def query_dns(qname, rtype, nxdomain='[Not Set]'):
 	# can compare to a well known order.
 	return "; ".join(sorted(str(r).rstrip('.') for r in response))
 
-def check_ssl_cert(domain, rounded_time, env, output):
+def check_ssl_cert(domain, rounded_time, ssl_certificates, env, output):
 	# Check that SSL certificate is signed.
 
 	# Skip the check if the A record is not pointed here.
 	if query_dns(domain, "A", None) not in (env['PUBLIC_IP'], None): return
 
 	# Where is the SSL stored?
-	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
+	x = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
 
-	if not os.path.exists(ssl_certificate):
-		output.print_error("The SSL certificate file for this domain is missing.")
+	if x is None:
+		output.print_warning("""No SSL certificate is installed for this domain. Visitors to a website on
+			this domain will get a security warning. If you are not serving a website on this domain, you do
+			not need to take any action. Use the SSL Certificates page in the control panel to install a
+			SSL certificate.""")
 		return
 
+	ssl_key, ssl_certificate, ssl_via = x
+
 	# Check that the certificate is good.
 
 	cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key, rounded_time=rounded_time)
@@ -588,16 +648,13 @@ def check_ssl_cert(domain, rounded_time, env, output):
 		if domain == env['PRIMARY_HOSTNAME']:
 			output.print_error("""The SSL certificate for this domain is currently self-signed. You will get a security
 			warning when you check or send email and when visiting this domain in a web browser (for webmail or
-			static site hosting). Use the SSL Certificates page in this control panel to install a signed SSL certificate.
+			static site hosting). Use the SSL Certificates page in the control panel to install a signed SSL certificate.
 			You may choose to leave the self-signed certificate in place and confirm the security exception, but check that
 			the certificate fingerprint matches the following:""")
 			output.print_line("")
 			output.print_line("   " + fingerprint, monospace=True)
 		else:
-			output.print_warning("""The SSL certificate for this domain is currently self-signed. Visitors to a website on
-			this domain will get a security warning. If you are not serving a website on this domain, then it is
-			safe to leave the self-signed certificate in place. Use the SSL Certificates page in this control panel to
-			install a signed SSL certificate.""")
+			output.print_error("""The SSL certificate for this domain is self-signed.""")
 
 	else:
 		output.print_error("The SSL certificate has a problem: " + cert_status)
@@ -611,8 +668,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 	# for the provided domain.
 
 	from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
-	from cryptography.x509 import Certificate, DNSName, ExtensionNotFound, OID_COMMON_NAME, OID_SUBJECT_ALTERNATIVE_NAME
-	import idna
+	from cryptography.x509 import Certificate
 
 	# The ssl_certificate file may contain a chain of certificates. We'll
 	# need to split that up before we can pass anything to openssl or
@@ -627,33 +683,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 	# First check that the domain name is one of the names allowed by
 	# the certificate.
 	if domain is not None:
-		# The domain may be found in the Subject Common Name (CN). This comes back as an IDNA (ASCII)
-		# string, which is the format we store domains in - so good.
-		certificate_names = set()
-		try:
-			certificate_names.add(
-				cert.subject.get_attributes_for_oid(OID_COMMON_NAME)[0].value
-				)
-		except IndexError:
-			# No common name? Certificate is probably generated incorrectly.
-			# But we'll let it error-out when it doesn't find the domain.
-			pass
-
-		# ... or be one of the Subject Alternative Names. The cryptography library handily IDNA-decodes
-		# the names for us. We must encode back to ASCII, but wildcard certificates can't pass through
-		# IDNA encoding/decoding so we must special-case. See https://github.com/pyca/cryptography/pull/2071.
-		def idna_decode_dns_name(dns_name):
-			if dns_name.startswith("*."):
-				return "*." + idna.encode(dns_name[2:]).decode('ascii')
-			else:
-				return idna.encode(dns_name).decode('ascii')
-
-		try:
-			sans = cert.extensions.get_extension_for_oid(OID_SUBJECT_ALTERNATIVE_NAME).value.get_values_for_type(DNSName)
-			for san in sans:
-				certificate_names.add(idna_decode_dns_name(san))
-		except ExtensionNotFound:
-			pass
+		certificate_names, cert_primary_name = get_certificate_domains(cert)
 
 		# Check that the domain appears among the acceptable names, or a wildcard
 		# form of the domain name (which is a stricter check than the specs but
@@ -773,6 +803,41 @@ def load_pem(pem):
 		return load_pem_x509_certificate(pem, default_backend())
 	raise ValueError("Unsupported PEM object type: " + pem_type.decode("ascii", "replace"))
 
+def get_certificate_domains(cert):
+	from cryptography.x509 import DNSName, ExtensionNotFound, OID_COMMON_NAME, OID_SUBJECT_ALTERNATIVE_NAME
+	import idna
+
+	names = set()
+	cn = None
+
+	# The domain may be found in the Subject Common Name (CN). This comes back as an IDNA (ASCII)
+	# string, which is the format we store domains in - so good.
+	try:
+		cn = cert.subject.get_attributes_for_oid(OID_COMMON_NAME)[0].value
+		names.add(cn)
+	except IndexError:
+		# No common name? Certificate is probably generated incorrectly.
+		# But we'll let it error-out when it doesn't find the domain.
+		pass
+
+	# ... or be one of the Subject Alternative Names. The cryptography library handily IDNA-decodes
+	# the names for us. We must encode back to ASCII, but wildcard certificates can't pass through
+	# IDNA encoding/decoding so we must special-case. See https://github.com/pyca/cryptography/pull/2071.
+	def idna_decode_dns_name(dns_name):
+		if dns_name.startswith("*."):
+			return "*." + idna.encode(dns_name[2:]).decode('ascii')
+		else:
+			return idna.encode(dns_name).decode('ascii')
+
+	try:
+		sans = cert.extensions.get_extension_for_oid(OID_SUBJECT_ALTERNATIVE_NAME).value.get_values_for_type(DNSName)
+		for san in sans:
+			names.add(idna_decode_dns_name(san))
+	except ExtensionNotFound:
+		pass
+
+	return names, cn
+
 _apt_updates = None
 def list_apt_updates(apt_update=True):
 	# See if we have this information cached recently.
@@ -808,11 +873,11 @@ def list_apt_updates(apt_update=True):
 	return pkgs
 
 def what_version_is_this(env):
-	# This function runs `git describe` on the Mail-in-a-Box installation directory.
+	# This function runs `git describe --abbrev=0` on the Mail-in-a-Box installation directory.
 	# Git may not be installed and Mail-in-a-Box may not have been cloned from github,
 	# so this function may raise all sorts of exceptions.
 	miab_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-	tag = shell("check_output", ["/usr/bin/git", "describe"], env={"GIT_DIR": os.path.join(miab_dir, '.git')}).strip()
+	tag = shell("check_output", ["/usr/bin/git", "describe", "--abbrev=0"], env={"GIT_DIR": os.path.join(miab_dir, '.git')}).strip()
 	return tag
 
 def get_latest_miab_version():
@@ -821,6 +886,20 @@ def get_latest_miab_version():
 	import urllib.request
 	return re.search(b'TAG=(.*)', urllib.request.urlopen("https://mailinabox.email/bootstrap.sh?ping=1").read()).group(1).decode("utf8")
 
+def check_miab_version(env, output):
+	config = load_settings(env)
+
+	if config.get("privacy", True):
+		output.print_warning("Mail-in-a-Box version check disabled by privacy setting.")
+	else:
+		this_ver = what_version_is_this(env)
+		latest_ver = get_latest_miab_version()
+		if this_ver == latest_ver:
+			output.print_ok("Mail-in-a-Box is up to date. You are running version %s." % this_ver)
+		else:
+			output.print_error("A new version of Mail-in-a-Box is available. You are running version %s. The latest version is %s. For upgrade instructions, see https://mailinabox.email. "
+				% (this_ver, latest_ver))
+
 def run_and_output_changes(env, pool, send_via_email):
 	import json
 	from difflib import SequenceMatcher
@@ -994,7 +1073,8 @@ if __name__ == "__main__":
 		domain = env['PRIMARY_HOSTNAME']
 		if query_dns(domain, "A") != env['PUBLIC_IP']:
 			sys.exit(1)
-		ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
+		ssl_certificates = get_ssl_certificates(env)
+		ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, ssl_certificates, env)
 		if not os.path.exists(ssl_certificate):
 			sys.exit(1)
 		cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key, warn_if_expiring_soon=False)

management/templates/external-dns.html

@@ -34,6 +34,15 @@
 
 <p>If you do so, you are responsible for keeping your DNS entries up to date! If you previously enabled DNSSEC on your domain name by setting a DS record at your registrar, you will likely have to turn it off before changing nameservers.</p>
 
+
+<p class="alert" role="alert">
+	<span class="glyphicon glyphicon-info-sign"></span>
+	You may encounter zone file errors when attempting to create a TXT record with a long string.
+	<a href="http://tools.ietf.org/html/rfc4408#section-3.1.3">RFC 4408</a> states a TXT record is allowed to contain multiple strings, and this technique can be used to construct records that would exceed the 255-byte maximum length.
+	You may need to adopt this technique when adding DomainKeys. Use a tool like <code>named-checkzone</code> to validate your zone file.
+</p>
+
+
 <table id="external_dns_settings" class="table">
 	<thead>
 		<tr>

management/templates/index.html

@@ -9,17 +9,10 @@
 
         <meta name="robots" content="noindex, nofollow">
 
-        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap.min.css">
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css" integrity="sha256-MfvZlkHCEqatNoGiOXveE8FIwMzZg4W85qfrfIFBfYc=" crossorigin="anonymous">
         <style>
-            @import url(https://fonts.googleapis.com/css?family=Raleway:400,700);
-            @import url(https://fonts.googleapis.com/css?family=Ubuntu:300);
-
-	    html {
+	    body {
 		overflow-y: scroll;
-	    }
-
-            body {
-                padding-top: 50px;
                 padding-bottom: 20px;
             }
 
@@ -28,7 +21,7 @@
             }
 
             h1, h2, h3, h4 {
-              font-family: Raleway, sans-serif;
+              font-family: sans-serif;
               font-weight: bold;
             }
 
@@ -70,15 +63,14 @@
 	       margin-bottom: 1em;
 	    }
         </style>
-        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap-theme.min.css">
-        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.3.0/css/font-awesome.min.css">
+        <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap-theme.min.css" integrity="sha256-bHQiqcFbnJb1Qhh61RY9cMh6kR0gTuQY6iFOBj1yj00=" crossorigin="anonymous">
     </head>
     <body>
 
     <!--[if lt IE 8]><p>Internet Explorer version 8 or any modern web browser is required to use this website, sorry.<![endif]-->
     <!--[if gt IE 7]><!-->
 
-    <div class="navbar navbar-inverse navbar-fixed-top" role="navigation">
+    <div class="navbar navbar-inverse" role="navigation">
       <div class="container">
         <div class="navbar-header">
           <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse">
@@ -114,7 +106,6 @@
             </li>
             <li><a href="#sync_guide" onclick="return show_panel(this);">Contacts/Calendar</a></li>
             <li><a href="#web" onclick="return show_panel(this);">Web</a></li>
-            <li><a href="#version" onclick="return show_panel(this);">Version</a></li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
             <li><a href="#" onclick="do_logout(); return false;" style="color: white">Log out?</a></li>
@@ -168,10 +159,6 @@
       {% include "ssl.html" %}
       </div>
 
-      <div id="panel_version" class="admin_panel">
-      {% include "version.html" %}
-      </div>
-
       <hr>
 
       <footer>
@@ -204,8 +191,8 @@
           </div>
         </div>
 
-        <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
-        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.0/js/bootstrap.min.js"></script>
+        <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js" integrity="sha256-rsPUGdUPBXgalvIj4YKJrrUlmLXbOb6Cp7cdxn1qeUc=" crossorigin="anonymous"></script>
+        <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/js/bootstrap.min.js" integrity="sha256-Sk3nkD6mLTMOF0EOpNtsIry+s1CsaqQC1rVLTAy+0yc=" crossorigin="anonymous"></script>
 
         <script>
 var global_modal_state = null;
@@ -215,8 +202,8 @@ $(function() {
   $('#global_modal').on('shown.bs.modal', function (e) {
     // set focus to first input in the global modal's body
     var input = $('#global_modal .modal-body input');
-    if (input.length > 0) $(input[0]).focus();  
-  })  
+    if (input.length > 0) $(input[0]).focus();
+  })
   $('#global_modal .btn-danger').click(function() {
     // Don't take action now. Wait for the modal to be totally hidden
     // so that we don't attempt to show another modal while this one
@@ -276,7 +263,7 @@ function show_modal_confirm(title, question, verb, yes_callback, cancel_callback
 }
 
 var ajax_num_executing_requests = 0;
-function ajax(options) {
+function ajax_with_indicator(options) {
   setTimeout("if (ajax_num_executing_requests > 0) $('#ajax_loading_indicator').fadeIn()", 100);
   function hide_loading_indicator() {
     ajax_num_executing_requests--;
@@ -338,7 +325,7 @@ function api(url, method, data, callback, callback_error) {
       show_modal_error("Error", "Something went wrong, sorry.")
   }
 
-  ajax({
+  ajax_with_indicator({
     url: "/admin" + url,
     method: method,
     cache: false,

management/templates/login.html

@@ -57,11 +57,15 @@ sudo tools/mail.py user make-admin me@{{hostname}}</pre>
 <script>
 function do_login() {
   if ($('#loginEmail').val() == "") {
-    show_modal_error("Login Failed", "Enter your email address.")
+    show_modal_error("Login Failed", "Enter your email address.", function() {
+	$('#loginEmail').focus();
+    });
     return false;
   }
   if ($('#loginPassword').val() == "") {
-    show_modal_error("Login Failed", "Enter your email password.")
+    show_modal_error("Login Failed", "Enter your email password.", function() {
+        $('#loginPassword').focus();
+    });
     return false;
   }
 
@@ -126,4 +130,14 @@ function do_logout() {
     sessionStorage.removeItem("miab-cp-credentials");
   show_panel('login');
 }
+
+function show_login() {
+  $('#loginEmail,#loginPassword').each(function() {
+    var input = $(this);
+    if (!$.trim(input.val())) {
+      input.focus();
+      return false;
+    }
+  });
+}
 </script>

management/templates/ssl.html

@@ -18,11 +18,11 @@
   </tbody>
 </table>
 
-<p>Advanced:<br>Install a multi-domain or wildcard certificate for the <code>{{hostname}}</code> domain to have it automatically applied to any domains it is valid for.</p>
+<p>A multi-domain or wildcard certificate will be automatically applied to any domains it is valid for.</p>
 
 <h3 id="ssl_install_header">Install SSL Certificate</h3>
 
-<p>There are many places where you can get a free or cheap SSL certificate. We recommend <a href="https://www.namecheap.com/security/ssl-certificates/domain-validation.aspx">Namecheap&rsquo;s $9 certificate</a> or <a href="https://www.startssl.com/">StartSSL&rsquo;s free express lane</a>.</p>
+<p>There are many places where you can get a free or cheap SSL certificate. We recommend <a href="https://www.namecheap.com/security/ssl-certificates/domain-validation.aspx">Namecheap&rsquo;s $9 certificate</a>, <a href="https://www.startssl.com/">StartSSL&rsquo;s free express lane</a> or <a href="https://buy.wosign.com/free/">Wosign&rsquo;s free SSL</a></a>.</p>
 
 <p>Which domain are you getting an SSL certificate for?</p>
 

management/templates/system-backup.html

@@ -22,12 +22,14 @@
   </div>
   <div class="form-group backup-target-local">
     <div class="col-sm-10 col-sm-offset-2">
-      <div>Backups are stored on this machine&rsquo;s own hard disk. You are responsible for periodically using SFTP (FTP over SSH) to copy the backup files from <tt id="backup-location"></tt> to a safe location. These files are encrypted, so they are safe to store anywhere.</div>
+      <p>Backups are stored on this machine&rsquo;s own hard disk. You are responsible for periodically using SFTP (FTP over SSH) to copy the backup files from <tt id="backup-location"></tt> to a safe location. These files are encrypted, so they are safe to store anywhere.</p>
+      <p>Separately copy the encryption password from <tt class="backup-encpassword-file"></tt> to a safe and secure location. You will need this file to decrypt backup files.</p>
     </div>
   </div>
   <div class="form-group backup-target-s3">
     <div class="col-sm-10 col-sm-offset-2">
-      <div>Backups are stored in an Amazon Web Services S3 bucket. You must have an AWS account already.</div>
+      <p>Backups are stored in an Amazon Web Services S3 bucket. You must have an AWS account already.</p>
+      <p>You MUST manually copy the encryption password from <tt class="backup-encpassword-file"></tt> to a safe and secure location. You will need this file to decrypt backup files. It is NOT stored in your Amazon S3 bucket.</p>
     </div>
   </div>
   <div class="form-group backup-target-local backup-target-s3">
@@ -71,8 +73,6 @@
   </div>
 </form>
 
-<p>Copy the encryption password from <tt id="backup-encpassword-file"></tt> to a safe and secure location. You will need this file to decrypt backup files.</p>
-
 <h3>Available Backups</h3>
 
 <p>The backup location currently contains the backups listed below. The total size of the backups is currently <span id="backup-total-size"></span>.</p>
@@ -175,7 +175,7 @@ function show_custom_backup() {
         $("#backup-target-pass").val(r.target_pass);
         $("#min-age").val(r.min_age_in_days);
         $('#backup-location').text(r.file_target_directory);
-        $('#backup-encpassword-file').text(r.enc_pw_file);
+        $('.backup-encpassword-file').text(r.enc_pw_file);
         toggle_form()
       })
 }
@@ -202,11 +202,12 @@ function set_custom_backup() {
       min_age: min_age
     },
     function(r) {
-      // Responses are multiple lines of pre-formatted text.
-      show_modal_error("Backup configuration", $("<pre/>").text(r), function() { show_system_backup(); }); // refresh after modal
+      // use .text() --- it's a text response, not html
+      show_modal_error("Backup configuration", $("<p/>").text(r), function() { if (r == "OK") show_system_backup(); }); // refresh after modal on success
     },
     function(r) {
-      show_modal_error("Backup configuration (error)", r);
+      // use .text() --- it's a text response, not html
+      show_modal_error("Backup configuration", $("<p/>").text(r));
     });
   return false;
 }

management/templates/system-status.html

@@ -34,8 +34,20 @@
   font-family: monospace;
   white-space: pre-wrap;
 }
+
+#system-privacy-setting {
+  float: right;
+  max-width: 20em;
+  margin-bottom: 1em;
+}
 </style>
 
+<div id="system-privacy-setting" style="display: none">
+	<div><a onclick="return enable_privacy(!current_privacy_setting)" href="#"><span>Enable/Disable</span> New-Version Check</a></div>
+	<p style="line-height: 125%"><small>(When enabled, status checks phone-home to check for a new release of Mail-in-a-Box.)</small></p>
+</div>
+
+
 <table id="system-checks" class="table" style="max-width: 60em">
   <thead>
   </thead>
@@ -46,6 +58,18 @@
 <script>
 function show_system_status() {
   $('#system-checks tbody').html("<tr><td colspan='2' class='text-muted'>Loading...</td></tr>")
+
+  api(
+    "/system/privacy",
+    "GET",
+    { },
+    function(r) {
+      current_privacy_setting = r;
+      $('#system-privacy-setting').show();
+      $('#system-privacy-setting a span').text(r ? "Enable" : "Disable");
+      $('#system-privacy-setting p').toggle(r);
+    });
+
   api(
     "/system/status",
     "POST",
@@ -82,5 +106,20 @@ function show_system_status() {
         }
       }
     })
+
+}
+
+var current_privacy_setting = null;
+function enable_privacy(status) {
+  api(
+    "/system/privacy",
+    "POST",
+    {
+      value: (status ? "private" : "off")
+    },
+    function(res) {
+      show_system_status();
+    });
+  return false; // disable link
 }
 </script>

management/templates/users.html

@@ -31,7 +31,7 @@
   <button type="submit" class="btn btn-primary">Add User</button>
 </form>
 <ul style="margin-top: 1em; padding-left: 1.5em; font-size: 90%;">
-  <li>Passwords must be at least four characters and may not contain spaces.</li>
+  <li>Passwords must be at least four characters and may not contain spaces. For best results, <a href="#" onclick="return generate_random_password()">generate a random password</a>.</li>
   <li>Use <a href="#" onclick="return show_panel('aliases')">aliases</a> to create email addresses that forward to existing accounts.</li>
   <li>Administrators get access to this control panel.</li>
   <li>User accounts cannot contain any international (non-ASCII) characters, but <a href="#" onclick="return show_panel('aliases');">aliases</a> can.</li>
@@ -250,4 +250,13 @@ function mod_priv(elem, add_remove) {
         });
     });
 }
+
+function generate_random_password() {
+  var pw = "";
+  var charset = "ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz23456789"; // confusable characters skipped
+  for (var i = 0; i < 10; i++)
+    pw += charset.charAt(Math.floor(Math.random() * charset.length));
+  show_modal_error("Random Password", "<p>Here, try this:</p> <p><code style='font-size: 110%'>" + pw + "</code></pr");
+  return false; // cancel click
+}
 </script>

management/templates/version.html

@@ -1,36 +0,0 @@
-<style>
-</style>
-
-<h2>Mail-in-a-Box Version</h2>
-
-<p>You are running Mail-in-a-Box version <span id="miab-version" style="font-weight: bold">...</span>.</p>
-
-<p>The latest version of Mail-in-a-Box is <button id="miab-get-latest-upstream" onclick="check_latest_version()">Check</button>.</p>
-
-<p>To find the latest version and for upgrade instructions, see <a href="https://mailinabox.email/">https://mailinabox.email/</a>, <a href="https://github.com/mail-in-a-box/mailinabox/blob/master/CHANGELOG.md">release notes</a>, and <a href="https://mailinabox.email/maintenance.html#updating-mail-in-a-box">upgrade instructions</a>.</p>
-
-<script>
-function show_version() {
- $('#miab-version').text('loading...');
- api(
-    "/system/version",
-    "GET",
-    {
-    },
-    function(version) {
-      $('#miab-version').text(version);
-    });
-}
-
-function check_latest_version() {
-  $('#miab-get-latest-upstream').text('loading...');
-  api(
-   "/system/latest-upstream-version",
-   "POST",
-   {
-   },
-   function(version) {
-      $('#miab-get-latest-upstream').text(version);
-   });
-}
-</script>

management/utils.py

@@ -1,6 +1,10 @@
 import os.path
 
-CONF_DIR = os.path.join(os.path.dirname(__file__), "../conf")
+# DO NOT import non-standard modules. This module is imported by
+# migrate.py which runs on fresh machines before anything is installed
+# besides Python.
+
+# THE ENVIRONMENT FILE AT /etc/mailinabox.conf
 
 def load_environment():
     # Load settings from /etc/mailinabox.conf.
@@ -18,6 +22,26 @@ def save_environment(env):
         for k, v in env.items():
             f.write("%s=%s\n" % (k, v))
 
+# THE SETTINGS FILE AT STORAGE_ROOT/settings.yaml.
+
+def write_settings(config, env):
+    import rtyaml
+    fn = os.path.join(env['STORAGE_ROOT'], 'settings.yaml')
+    with open(fn, "w") as f:
+        f.write(rtyaml.dump(config))
+
+def load_settings(env):
+    import rtyaml
+    fn = os.path.join(env['STORAGE_ROOT'], 'settings.yaml')
+    try:
+        config = rtyaml.load(open(fn, "r"))
+        if not isinstance(config, dict): raise ValueError() # caught below
+        return config
+    except:
+        return { }
+
+# UTILITIES
+
 def safe_domain_name(name):
     # Sanitize a domain name so it is safe to use as a file name on disk.
     import urllib.parse

management/web_update.py

@@ -32,7 +32,7 @@ def get_domains_with_a_records(env):
 	domains = set()
 	dns = get_custom_dns_config(env)
 	for domain, rtype, value in dns:
-		if rtype == "CNAME" or (rtype in ("A", "AAAA") and value != "local"):
+		if rtype == "CNAME" or (rtype in ("A", "AAAA") and value not in ("local", env['PUBLIC_IP'])):
 			domains.add(domain)
 	return domains
 
@@ -60,6 +60,9 @@ def get_default_www_redirects(env):
 	return sort_domains(www_domains - web_domains - get_domains_with_a_records(env), env)
 
 def do_web_update(env):
+	# Pre-load what SSL certificates we will use for each domain.
+	ssl_certificates = get_ssl_certificates(env)
+
 	# Build an nginx configuration file.
 	nginx_conf = open(os.path.join(os.path.dirname(__file__), "../conf/nginx-top.conf")).read()
 
@@ -70,20 +73,20 @@ def do_web_update(env):
 	template3 = "\trewrite ^(.*) https://$REDIRECT_DOMAIN$1 permanent;\n"
 
 	# Add the PRIMARY_HOST configuration first so it becomes nginx's default server.
-	nginx_conf += make_domain_config(env['PRIMARY_HOSTNAME'], [template0, template1, template2], env)
+	nginx_conf += make_domain_config(env['PRIMARY_HOSTNAME'], [template0, template1, template2], ssl_certificates, env)
 
 	# Add configuration all other web domains.
 	has_root_proxy_or_redirect = get_web_domains_with_root_overrides(env)
 	for domain in get_web_domains(env):
 		if domain == env['PRIMARY_HOSTNAME']: continue # handled above
 		if domain not in has_root_proxy_or_redirect:
-			nginx_conf += make_domain_config(domain, [template0, template1], env)
+			nginx_conf += make_domain_config(domain, [template0, template1], ssl_certificates, env)
 		else:
-			nginx_conf += make_domain_config(domain, [template0], env)
+			nginx_conf += make_domain_config(domain, [template0], ssl_certificates, env)
 
 	# Add default www redirects.
 	for domain in get_default_www_redirects(env):
-		nginx_conf += make_domain_config(domain, [template0, template3], env)
+		nginx_conf += make_domain_config(domain, [template0, template3], ssl_certificates, env)
 
 	# Did the file change? If not, don't bother writing & restarting nginx.
 	nginx_conf_fn = "/etc/nginx/conf.d/local.conf"
@@ -104,18 +107,14 @@ def do_web_update(env):
 
 	return "web updated\n"
 
-def make_domain_config(domain, templates, env):
+def make_domain_config(domain, templates, ssl_certificates, env):
 	# GET SOME VARIABLES
 
 	# Where will its root directory be for static files?
 	root = get_web_root(domain, env)
 
 	# What private key and SSL certificate will we use for this domain?
-	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
-
-	# For hostnames created after the initial setup, ensure we have an SSL certificate
-	# available. Make a self-signed one now if one doesn't exist.
-	ensure_ssl_certificate_exists(domain, ssl_key, ssl_certificate, env)
+	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, ssl_certificates, env)
 
 	# ADDITIONAL DIRECTIVES.
 
@@ -135,16 +134,28 @@ def make_domain_config(domain, templates, env):
 	nginx_conf_extra += "# ssl files sha1: %s / %s\n" % (hashfile(ssl_key), hashfile(ssl_certificate))
 
 	# Add in any user customizations in YAML format.
+	hsts = "yes"
 	nginx_conf_custom_fn = os.path.join(env["STORAGE_ROOT"], "www/custom.yaml")
 	if os.path.exists(nginx_conf_custom_fn):
 		yaml = rtyaml.load(open(nginx_conf_custom_fn))
 		if domain in yaml:
 			yaml = yaml[domain]
+
+			# any proxy or redirect here?
 			for path, url in yaml.get("proxies", {}).items():
 				nginx_conf_extra += "\tlocation %s {\n\t\tproxy_pass %s;\n\t}\n" % (path, url)
 			for path, url in yaml.get("redirects", {}).items():
 				nginx_conf_extra += "\trewrite %s %s permanent;\n" % (path, url)
 
+			# override the HSTS directive type
+			hsts = yaml.get("hsts", hsts)
+
+	# Add the HSTS header.
+	if hsts == "yes":
+		nginx_conf_extra += "add_header Strict-Transport-Security max-age=31536000;\n"
+	elif hsts == "preload":
+		nginx_conf_extra += "add_header Strict-Transport-Security \"max-age=10886400; includeSubDomains; preload\";\n"
+
 	# Add in any user customizations in the includes/ folder.
 	nginx_conf_custom_include = os.path.join(env["STORAGE_ROOT"], "www", safe_domain_name(domain) + ".conf")
 	if os.path.exists(nginx_conf_custom_include):
@@ -174,77 +185,140 @@ def get_web_root(domain, env, test_exists=True):
 		if os.path.exists(root) or not test_exists: break
 	return root
 
-def get_domain_ssl_files(domain, env, allow_shared_cert=True):
-	# What SSL private key will we use? Allow the user to override this, but
-	# in many cases using the same private key for all domains would be fine.
-	# Don't allow the user to override the key for PRIMARY_HOSTNAME because
-	# that's what's in the main file.
-	ssl_key = os.path.join(env["STORAGE_ROOT"], 'ssl/ssl_private_key.pem')
-	ssl_key_is_alt = False
-	alt_key = os.path.join(env["STORAGE_ROOT"], 'ssl/%s/private_key.pem' % safe_domain_name(domain))
-	if domain != env['PRIMARY_HOSTNAME'] and os.path.exists(alt_key):
-		ssl_key = alt_key
-		ssl_key_is_alt = True
+def get_ssl_certificates(env):
+	# Scan all of the installed SSL certificates and map every domain
+	# that the certificates are good for to the best certificate for
+	# the domain.
+
+	from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
+	from cryptography.x509 import Certificate
+
+	# The certificates are all stored here:
+	ssl_root = os.path.join(env["STORAGE_ROOT"], 'ssl')
+
+	# List all of the files in the SSL directory and one level deep.
+	def get_file_list():
+		for fn in os.listdir(ssl_root):
+			fn = os.path.join(ssl_root, fn)
+			if os.path.isfile(fn):
+				yield fn
+			elif os.path.isdir(fn):
+				for fn1 in os.listdir(fn):
+					fn1 = os.path.join(fn, fn1)
+					if os.path.isfile(fn1):
+						yield fn1
+
+	# Remember stuff.
+	private_keys = { }
+	certificates = [ ]
+
+	# Scan each of the files to find private keys and certificates.
+	# We must load all of the private keys first before processing
+	# certificates so that we can check that we have a private key
+	# available before using a certificate.
+	from status_checks import load_cert_chain, load_pem
+	for fn in get_file_list():
+		try:
+			pem = load_pem(load_cert_chain(fn)[0])
+		except ValueError:
+			# Not a valid PEM format for a PEM type we care about.
+			continue
+
+		# Remember where we got this object.
+		pem._filename = fn
+
+		# Is it a private key?
+		if isinstance(pem, RSAPrivateKey):
+			private_keys[pem.public_key().public_numbers()] = pem
+
+		# Is it a certificate?
+		if isinstance(pem, Certificate):
+			certificates.append(pem)
+
+	# Process the certificates.
+	domains = { }
+	from status_checks import get_certificate_domains
+	for cert in certificates:
+		# What domains is this certificate good for?
+		cert_domains, primary_domain = get_certificate_domains(cert)
+		cert._primary_domain = primary_domain
+
+		# Is there a private key file for this certificate?
+		private_key = private_keys.get(cert.public_key().public_numbers())
+		if not private_key:
+			continue
+		cert._private_key = private_key
+
+		# Add this cert to the list of certs usable for the domains.
+		for domain in cert_domains:
+			domains.setdefault(domain, []).append(cert)
+
+	# Sort the certificates to prefer good ones.
+	import datetime
+	now = datetime.datetime.utcnow()
+	ret = { }
+	for domain, cert_list in domains.items():
+		cert_list.sort(key = lambda cert : (
+			# must be valid NOW
+			cert.not_valid_before <= now <= cert.not_valid_after,
+
+			# prefer one that is not self-signed
+			cert.issuer != cert.subject,
+
+			# prefer one with the expiration furthest into the future so
+			# that we can easily rotate to new certs as we get them
+			cert.not_valid_after,
+
+			# in case a certificate is installed in multiple paths,
+			# prefer the... lexicographically last one?
+			cert._filename,
+
+		), reverse=True)
+		cert = cert_list.pop(0)
+		ret[domain] = {
+			"private-key": cert._private_key._filename,
+			"certificate": cert._filename,
+			"primary-domain": cert._primary_domain,
+			}
+
+	return ret
+
+def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False):
+	# Get the default paths.
+	ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
+	ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
 
-	# What SSL certificate will we use?
-	ssl_certificate_primary = os.path.join(env["STORAGE_ROOT"], 'ssl/ssl_certificate.pem')
-	ssl_via = None
 	if domain == env['PRIMARY_HOSTNAME']:
-		# For PRIMARY_HOSTNAME, use the one we generated at set-up time.
-		ssl_certificate = ssl_certificate_primary
+		# The primary domain must use the server certificate because
+		# it is hard-coded in some service configuration files.
+		return ssl_private_key, ssl_certificate, None
+
+	wildcard_domain = re.sub("^[^\.]+", "*", domain)
+
+	if domain in ssl_certificates:
+		cert_info = ssl_certificates[domain]
+		cert_type = "multi-domain"
+	elif wildcard_domain in ssl_certificates:
+		cert_info = ssl_certificates[wildcard_domain]
+		cert_type = "wildcard"
+	elif not allow_missing_cert:
+		# No certificate is available for this domain! Return default files.
+		ssl_via = "Using certificate for %s." % env['PRIMARY_HOSTNAME']
+		return ssl_private_key, ssl_certificate, ssl_via
 	else:
-		# For other domains, we'll probably use a certificate in a different path.
-		ssl_certificate = os.path.join(env["STORAGE_ROOT"], 'ssl/%s/ssl_certificate.pem' % safe_domain_name(domain))
+		# No certificate is available - and warn appropriately.
+		return None
 
-		# But we can be smart and reuse the main SSL certificate if is has
-		# a Subject Alternative Name matching this domain. Don't do this if
-		# the user has uploaded a different private key for this domain.
-		if not ssl_key_is_alt and allow_shared_cert:
-			from status_checks import check_certificate
-			if check_certificate(domain, ssl_certificate_primary, None, just_check_domain=True)[0] == "OK":
-				ssl_certificate = ssl_certificate_primary
-				ssl_via = "Using multi/wildcard certificate of %s." % env['PRIMARY_HOSTNAME']
+	# 'via' is a hint to the user about which certificate is in use for the domain
+	if cert_info['certificate'] == os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'):
+		# Using the server certificate.
+		via = "Using same %s certificate as for %s." % (cert_type, env['PRIMARY_HOSTNAME'])
+	elif cert_info['primary-domain'] != domain and cert_info['primary-domain'] in ssl_certificates and cert_info == ssl_certificates[cert_info['primary-domain']]:
+		via = "Using same %s certificate as for %s." % (cert_type, cert_info['primary-domain'])
+	else:
+		via = None # don't show a hint - show expiration info instead
 
-			# For a 'www.' domain, see if we can reuse the cert of the parent.
-			elif domain.startswith('www.'):
-				ssl_certificate_parent = os.path.join(env["STORAGE_ROOT"], 'ssl/%s/ssl_certificate.pem' % safe_domain_name(domain[4:]))
-				if os.path.exists(ssl_certificate_parent) and check_certificate(domain, ssl_certificate_parent, None, just_check_domain=True)[0] == "OK":
-					ssl_certificate = ssl_certificate_parent
-					ssl_via = "Using multi/wildcard certificate of %s." % domain[4:]
-
-	return ssl_key, ssl_certificate, ssl_via
-
-def ensure_ssl_certificate_exists(domain, ssl_key, ssl_certificate, env):
-	# For domains besides PRIMARY_HOSTNAME, generate a self-signed certificate if
-	# a certificate doesn't already exist. See setup/mail.sh for documentation.
-
-	if domain == env['PRIMARY_HOSTNAME']:
-		return
-
-	# Sanity check. Shouldn't happen. A non-primary domain might use this
-	# certificate (see above), but then the certificate should exist anyway.
-	if ssl_certificate == os.path.join(env["STORAGE_ROOT"], 'ssl/ssl_certificate.pem'):
-		return
-
-	if os.path.exists(ssl_certificate):
-		return
-
-	os.makedirs(os.path.dirname(ssl_certificate), exist_ok=True)
-
-	# Generate a new self-signed certificate using the same private key that we already have.
-
-	# Start with a CSR written to a temporary file.
-	with tempfile.NamedTemporaryFile(mode="w") as csr_fp:
-		csr_fp.write(create_csr(domain, ssl_key, env))
-		csr_fp.flush() # since we won't close until after running 'openssl x509', since close triggers delete.
-
-		# And then make the certificate.
-		shell("check_call", [
-			"openssl", "x509", "-req",
-			"-days", "365",
-			"-in", csr_fp.name,
-			"-signkey", ssl_key,
-			"-out", ssl_certificate])
+	return cert_info['private-key'], cert_info['certificate'], via
 
 def create_csr(domain, ssl_key, env):
 	return shell("check_output", [
@@ -266,8 +340,8 @@ def install_cert(domain, ssl_cert, ssl_chain, env):
 
 	# Do validation on the certificate before installing it.
 	from status_checks import check_certificate
-	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env, allow_shared_cert=False)
-	cert_status, cert_status_details = check_certificate(domain, fn, ssl_key)
+	ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
+	cert_status, cert_status_details = check_certificate(domain, fn, ssl_private_key)
 	if cert_status != "OK":
 		if cert_status == "SELF-SIGNED":
 			cert_status = "This is a self-signed certificate. I can't install that."
@@ -276,7 +350,24 @@ def install_cert(domain, ssl_cert, ssl_chain, env):
 			cert_status += " " + cert_status_details
 		return cert_status
 
-	# Copy the certificate to its expected location.
+	# Where to put it?
+	if domain == env['PRIMARY_HOSTNAME']:
+		ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
+	else:
+		# Make a unique path for the certificate.
+		from status_checks import load_cert_chain, load_pem, get_certificate_domains
+		from cryptography.hazmat.primitives import hashes
+		from binascii import hexlify
+		cert = load_pem(load_cert_chain(fn)[0])
+		all_domains, cn = get_certificate_domains(cert)
+		path = "%s-%s-%s" % (
+			cn, # common name
+			cert.not_valid_after.date().isoformat().replace("-", ""), # expiration date
+			hexlify(cert.fingerprint(hashes.SHA256())).decode("ascii")[0:8], # fingerprint prefix
+			)
+		ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', path, 'ssl_certificate.pem'))
+
+	# Install the certificate.
 	os.makedirs(os.path.dirname(ssl_certificate), exist_ok=True)
 	shutil.move(fn, ssl_certificate)
 
@@ -302,9 +393,10 @@ def get_web_domains_info(env):
 	# for the SSL config panel, get cert status
 	def check_cert(domain):
 		from status_checks import check_certificate
-		ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
-		if not os.path.exists(ssl_certificate):
-			return ("danger", "No Certificate Installed")
+		ssl_certificates = get_ssl_certificates(env)
+		x = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
+		if x is None: return ("danger", "No Certificate Installed")
+		ssl_key, ssl_certificate, ssl_via = x
 		cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key)
 		if cert_status == "OK":
 			if not ssl_via:

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.13a
+	TAG=v0.14
 fi
 
 # Are we running as root?

setup/mail-dovecot.sh

@@ -56,6 +56,9 @@ tools/editconf.py /etc/dovecot/conf.d/10-mail.conf \
 	mail_privileged_group=mail \
 	first_valid_uid=0
 
+# Create, subscribe, and mark as special folders: INBOX, Drafts, Sent, Trash, Spam and Archive.
+cp conf/dovecot-mailboxes.conf /etc/dovecot/conf.d/15-mailboxes.conf
+
 # ### IMAP/POP
 
 # Require that passwords are sent over SSL only, and allow the usual IMAP authentication mechanisms.
@@ -151,6 +154,12 @@ sed -i "s/#mail_plugins = .*/mail_plugins = \$mail_plugins sieve/" /etc/dovecot/
 #
 # * `sieve_before`: The path to our global sieve which handles moving spam to the Spam folder.
 #
+# * `sieve_before2`: The path to our global sieve directory for sieve which can contain .sieve files
+# to run globally for every user before their own sieve files run.
+#
+# * `sieve_after`: The path to our global sieve directory which can contain .sieve files
+# to run globally for every user after their own sieve files run.
+#
 # * `sieve`: The path to the user's main active script. ManageSieve will create a symbolic
 # link here to the actual sieve script. It should not be in the mailbox directory
 # (because then it might appear as a folder) and it should not be in the sieve_dir
@@ -160,6 +169,8 @@ sed -i "s/#mail_plugins = .*/mail_plugins = \$mail_plugins sieve/" /etc/dovecot/
 cat > /etc/dovecot/conf.d/99-local-sieve.conf << EOF;
 plugin {
   sieve_before = /etc/dovecot/sieve-spam.sieve
+  sieve_before2 = $STORAGE_ROOT/mail/sieve/global_before
+  sieve_after = $STORAGE_ROOT/mail/sieve/global_after
   sieve = $STORAGE_ROOT/mail/sieve/%d/%n.sieve
   sieve_dir = $STORAGE_ROOT/mail/sieve/%d/%n
 }
@@ -183,6 +194,8 @@ chown -R mail.mail $STORAGE_ROOT/mail/mailboxes
 
 # Same for the sieve scripts.
 mkdir -p $STORAGE_ROOT/mail/sieve
+mkdir -p $STORAGE_ROOT/mail/sieve/global_before
+mkdir -p $STORAGE_ROOT/mail/sieve/global_after
 chown -R mail.mail $STORAGE_ROOT/mail/sieve
 
 # Allow the IMAP/POP ports in the firewall.

setup/management.sh

@@ -7,7 +7,7 @@ echo "Installing Mail-in-a-Box system management daemon..."
 # build-essential libssl-dev libffi-dev python3-dev: Required to pip install cryptography.
 apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil \
 	build-essential libssl-dev libffi-dev python3-dev python-pip
-hide_output pip3 install --upgrade rtyaml email_validator idna cryptography boto
+hide_output pip3 install --upgrade rtyaml email_validator>=1.0.0 idna>=2.0.0 cryptography>=1.0.2 boto
 
 # duplicity uses python 2 so we need to use the python 2 package of boto
 hide_output pip install --upgrade boto

setup/munin.sh

@@ -38,7 +38,7 @@ munin-node-configure --shell --remove-also 2>/dev/null | sh
 
 # Deactivate monitoring of NTP peers. Not sure why anyone would want to monitor a NTP peer. The addresses seem to change
 # (which is taken care of my munin-node-configure, but only when we re-run it.)
-find /etc/munin/plugins/ -lname /usr/share/munin/plugins/ntp_ -print0 | xargs -0 /bin/rm
+find /etc/munin/plugins/ -lname /usr/share/munin/plugins/ntp_ -print0 | xargs -0 /bin/rm -f
 
 # Deactivate monitoring of network interfaces that are not up. Otherwise we can get a lot of empty charts.
 for f in $(find /etc/munin/plugins/ \( -lname /usr/share/munin/plugins/if_ -o -lname /usr/share/munin/plugins/if_err_ -o -lname /usr/share/munin/plugins/bonding_err_ \)); do

setup/preflight.sh

@@ -19,7 +19,7 @@ fi
 
 # Check that we have enough memory.
 #
-# /proc/meminfo reports free memory in kibibytes. Our baseline will be 768 KB,
+# /proc/meminfo reports free memory in kibibytes. Our baseline will be 768 MB,
 # which is 750000 kibibytes.
 #
 # Skip the check if we appear to be running inside of Vagrant, because that's really just for testing.

setup/questions.sh

@@ -13,7 +13,7 @@ if [ -z "$NONINTERACTIVE" ]; then
 	fi
 
 	# email_validator is repeated in setup/management.sh
-	hide_output pip3 install email_validator || exit 1
+	hide_output pip3 install email_validator==1.0.0 || exit 1
 
 	message_box "Mail-in-a-Box Installation" \
 		"Hello and thanks for deploying a Mail-in-a-Box!

setup/spamassassin.sh

@@ -16,8 +16,11 @@ source setup/functions.sh # load our functions
 # ----------------------------------------
 
 # Install packages.
+# libmail-dkim-perl is needed to make the spamassassin DKIM module work.
+# For more information see Debian Bug #689414:
+# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=689414
 echo "Installing SpamAssassin..."
-apt_install spampd razor pyzor dovecot-antispam
+apt_install spampd razor pyzor dovecot-antispam libmail-dkim-perl
 
 # Allow spamassassin to download new rules.
 tools/editconf.py /etc/default/spamassassin \
@@ -42,9 +45,11 @@ echo "public.pyzor.org:24441" > /etc/spamassassin/pyzor/servers
 #   want to lose track of it. (We've configured Dovecot to listen on this port elsewhere.)
 # * Increase the maximum message size of scanned messages from the default of 64KB to 500KB, which
 #   is Spamassassin (spamc)'s own default. Specified in KBytes.
+# * Disable localmode so Pyzor, DKIM and DNS checks can be used.
 tools/editconf.py /etc/default/spampd \
 	DESTPORT=10026 \
-	ADDOPTS="\"--maxsize=500\""
+	ADDOPTS="\"--maxsize=500\"" \
+	LOCALONLY=0
 
 # Spamassassin normally wraps spam as an attachment inside a fresh
 # email with a report about the message. This also protects the user
@@ -94,6 +99,7 @@ cat > /etc/dovecot/conf.d/99-local-spampd.conf << EOF;
 plugin {
     antispam_backend = pipe
     antispam_spam_pattern_ignorecase = SPAM
+    antispam_trash_pattern_ignorecase = trash;Deleted *
     antispam_allow_append_to_spam = yes
     antispam_pipe_program_spam_args = /usr/local/bin/sa-learn-pipe.sh;--spam
     antispam_pipe_program_notspam_args = /usr/local/bin/sa-learn-pipe.sh;--ham

setup/start.sh

@@ -130,17 +130,19 @@ if management/status_checks.py --check-primary-hostname; then
 	# Show the nice URL if it appears to be resolving and has a valid certificate.
 	echo https://$PRIMARY_HOSTNAME/admin
 	echo
-	echo If you have a DNS problem use the box\'s IP address and check the SSL fingerprint:
-	echo https://$PUBLIC_IP/admin
+	echo "If you have a DNS problem put the box's IP address in the URL"
+	echo "(https://$PUBLIC_IP/admin) but then check the SSL fingerprint:"
+	openssl x509 -in $STORAGE_ROOT/ssl/ssl_certificate.pem -noout -fingerprint \
+        	| sed "s/SHA1 Fingerprint=//"
 else
 	echo https://$PUBLIC_IP/admin
 	echo
 	echo You will be alerted that the website has an invalid certificate. Check that
 	echo the certificate fingerprint matches:
 	echo
+	openssl x509 -in $STORAGE_ROOT/ssl/ssl_certificate.pem -noout -fingerprint \
+        	| sed "s/SHA1 Fingerprint=//"
+	echo
+	echo Then you can confirm the security exception and continue.
+	echo
 fi
-openssl x509 -in $STORAGE_ROOT/ssl/ssl_certificate.pem -noout -fingerprint \
-        | sed "s/SHA1 Fingerprint=//"
-echo
-echo Then you can confirm the security exception and continue.
-echo

setup/web.sh

@@ -25,13 +25,19 @@ rm -f /etc/nginx/sites-enabled/default
 # Copy in a nginx configuration file for common and best-practices
 # SSL settings from @konklone. Replace STORAGE_ROOT so it can find
 # the DH params.
+rm -f /etc/nginx/nginx-ssl.conf # we used to put it here
 sed "s#STORAGE_ROOT#$STORAGE_ROOT#" \
-	conf/nginx-ssl.conf > /etc/nginx/nginx-ssl.conf
+	conf/nginx-ssl.conf > /etc/nginx/conf.d/ssl.conf
 
 # Fix some nginx defaults.
-# The server_names_hash_bucket_size seems to prevent long domain names?
+# The server_names_hash_bucket_size seems to prevent long domain names!
+# The default, according to nginx's docs, depends on "the size of the
+# processor’s cache line." It could be as low as 32. We fixed it at
+# 64 in 2014 to accommodate a long domain name (20 characters?). But
+# even at 64, a 58-character domain name won't work (#93), so now
+# we're going up to 128.
 tools/editconf.py /etc/nginx/nginx.conf -s \
-	server_names_hash_bucket_size="64;"
+	server_names_hash_bucket_size="128;"
 
 # Tell PHP not to expose its version number in the X-Powered-By header.
 tools/editconf.py /etc/php5/fpm/php.ini -c ';' \

setup/webmail.sh

@@ -37,7 +37,8 @@ VERSION=1.1.2
 HASH=df88deae691da3ecf3e9f0aee674c1f3042ea1eb
 VACATION_SIEVE_VERSION=91ea6f52216390073d1f5b70b5f6bea0bfaee7e5
 PERSISTENT_LOGIN_VERSION=117fbd8f93b56b2bf72ad055193464803ef3bc36
-UPDATE_KEY=$VERSION:$VACATION_SIEVE_VERSION:$PERSISTENT_LOGIN_VERSION
+HTML5_NOTIFIER_VERSION=046eb388dd63b1ec77a3ee485757fc25ae9e684d
+UPDATE_KEY=$VERSION:$VACATION_SIEVE_VERSION:$PERSISTENT_LOGIN_VERSION:$HTML5_NOTIFIER_VERSION
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/roundcubemail/version ]; then
 	# not installed yet #NODOC
@@ -63,6 +64,9 @@ if [ $needs_update == 1 ]; then
 	# install roundcube persistent_login plugin
 	git_clone https://github.com/mfreiholz/Roundcube-Persistent-Login-Plugin.git $PERSISTENT_LOGIN_VERSION '' /usr/local/lib/roundcubemail/plugins/persistent_login
 
+	# install roundcube html5_notifier plugin
+	git_clone https://github.com/kitist/html5_notifier.git $HTML5_NOTIFIER_VERSION '' /usr/local/lib/roundcubemail/plugins/html5_notifier
+
 	# record the version we've installed
 	echo $UPDATE_KEY > /usr/local/lib/roundcubemail/version
 fi
@@ -96,7 +100,7 @@ cat > /usr/local/lib/roundcubemail/config/config.inc.php <<EOF;
 \$config['support_url'] = 'https://mailinabox.email/';
 \$config['product_name'] = 'Mail-in-a-Box/Roundcube Webmail';
 \$config['des_key'] = '$SECRET_KEY';
-\$config['plugins'] = array('archive', 'zipdownload', 'password', 'managesieve', 'jqueryui', 'vacation_sieve', 'persistent_login');
+\$config['plugins'] = array('html5_notifier', 'archive', 'zipdownload', 'password', 'managesieve', 'jqueryui', 'vacation_sieve', 'persistent_login');
 \$config['skin'] = 'classic';
 \$config['login_autocomplete'] = 2;
 \$config['password_charset'] = 'UTF-8';

setup/zpush.sh

@@ -22,7 +22,7 @@ apt_install \
 php5enmod imap
 
 # Copy Z-Push into place.
-TARGETHASH=d0cd5a47c53afac5c3b287006dc8a48a1c4ffcd5
+TARGETHASH=80cbe53de4ab8dd598d1f2af6f0a23fa396c529a
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/z-push/version ]; then
 	needs_update=1 #NODOC
@@ -42,6 +42,8 @@ fi
 sed -i "s^define('TIMEZONE', .*^define('TIMEZONE', '$(cat /etc/timezone)');^" /usr/local/lib/z-push/config.php
 sed -i "s/define('BACKEND_PROVIDER', .*/define('BACKEND_PROVIDER', 'BackendCombined');/" /usr/local/lib/z-push/config.php
 sed -i "s/define('USE_FULLEMAIL_FOR_LOGIN', .*/define('USE_FULLEMAIL_FOR_LOGIN', true);/" /usr/local/lib/z-push/config.php
+sed -i "s/define('LOG_MEMORY_PROFILER', .*/define('LOG_MEMORY_PROFILER', false);/" /usr/local/lib/z-push/config.php
+sed -i "s/define('BUG68532FIXED', .*/define('BUG68532FIXED', false);/" /usr/local/lib/z-push/config.php
 
 # Configure BACKEND
 rm -f /usr/local/lib/z-push/backend/combined/config.php

tools/update-subresource-integrity.py

@@ -0,0 +1,24 @@
+#!/usr/bin/python3
+# Updates subresource integrity attributes in management/templates/index.html
+# to prevent CDN-hosted resources from being used as an attack vector. Run this
+# after updating the Bootstrap and jQuery <link> and <script> to compute the
+# appropriate hash and insert it into the template.
+
+import re, urllib.request, hashlib, base64
+
+fn = "management/templates/index.html"
+
+with open(fn, 'r') as f:
+	content = f.read()
+
+def make_integrity(url):
+	resource = urllib.request.urlopen(url).read()
+	return "sha256-" + base64.b64encode(hashlib.sha256(resource).digest()).decode('ascii')
+
+content = re.sub(
+	r'<(link rel="stylesheet" href|script src)="(.*?)" integrity="(.*?)"',
+	lambda m : '<' + m.group(1) + '="' + m.group(2) + '" integrity="' + make_integrity(m.group(2)) + '"',
+	content)
+
+with open(fn, 'w') as f:
+	f.write(content)