prep for v0.07 tag

Changed MX check to respect priorities other than 10.

CHANGELOG.md

@@ -1,24 +1,46 @@
 CHANGELOG
 =========
 
-Development
------------
+v0.07 (February 28, 2015)
+-------------------------
+
+Mail:
+
+* If the box manages mail for a domain and a subdomain of that domain, outbound mail from the subdomain was not DKIM-signed and would therefore fail DMARC tests on the receiving end, possibly result in the mail heading into spam folders.
+* Auto-configuration for Mozilla Thunderbird, Evolution, KMail, and Kontact is now available.
+* Domains that only have a catch-all alias or domain alias no longer automatically create/require admin@ and postmaster@ addresses since they'll forward anyway.
+* Roundcube is updated to version 1.1.0.
+* Authentication-Results headers for DMARC are now added to incoming mail.
+
+DNS:
+
+* If a custom CNAME record is set on a 'www' subdomain, the default A/AAAA records were preventing the CNAME from working.
+* If a custom DNS A record overrides one provided by the box, the a corresponding default IPv6 record by the box is removed since it will probably be incorrect.
+* Internationalized domain names (IDNs) are now supported for DNS and web, but email is not yet tested.
+
+Web:
+
+* Static websites now deny access to certain dot (.) files and directories which typically have sensitive info: .ht*, .svn*, .git*, .hg*, .bzr*.
+* The nginx server no longer reports its version and OS for better privacy.
+* The HTTP->HTTPS redirect is now more efficient.
+* When serving a 'www.' domain, reuse the SSL certificate for the parent domain if it covers the 'www' subdomain too
+* If a custom DNS CNAME record is set on a domain, don't offer to put a website on that domain. (Same logic already applies to custom A/AAAA records.)
 
 Control panel:
 
 * Status checks now check that system services are actually running by pinging each port that should have something running on it.
-* If a custom CNAME record is set on a 'www' subdomain, the default A/AAAA records were preventing the CNAME from working.
+* The status checks are now parallelized so they may be a little faster.
+* The status check for MX records now allow any priority, in case an unusual setup is required.
+* The interface for setting website domain-specific directories is simplified.
+* The mail guide now says that to use Outlook, Outlook 2007 or later on Windows 7 and later is required.
+* External DNS settings now skip the special "_secondary_nameserver" key which is used for storing secondary NS information.
 
 Setup:
 
 * Install cron if it isn't already installed.
 * Fix a units problem in the minimum memory check.
-
-Miscellaneous:
-
-* Internationalized domain names (IDNs) are now supported for DNS and web, but email is not yet tested.
-* Domains that only have a catch-all alias or domain alias no longer automatically create/require admin@ and postmaster@ addresses since they'll forward anyway.
-
+* If you override the STORAGE_ROOT, your setting will now persist if you re-run setup.
+* Hangs due to apt wanting the user to resolve a conflict should now be fixed (apt will just clobber the problematic file now).
 
 v0.06 (January 4, 2015)
 -----------------------

conf/dovecot-checkpassword.py

@@ -1,53 +0,0 @@
-#!/usr/bin/python3
-#
-# This script implement's Dovecot's checkpassword authentication mechanism:
-# http://wiki2.dovecot.org/AuthDatabase/CheckPassword?action=show&redirect=PasswordDatabase%2FCheckPassword
-#
-# This allows us to perform our own password validation, such as for two-factor authentication,
-# which Dovecot does not have any native support for.
-#
-# We will issue an HTTP request to our management server to perform authentication.
-
-import sys, os, urllib.request, base64, json, traceback
-
-try:
-	# Read fd 3 which provides the username and password separated
-	# by NULLs and two other undocumented/empty fields.
-	creds = b''
-	while True:
-		b = os.read(3, 1024)
-		if len(b) == 0: break
-		creds += b
-	email, pw, dummy, dummy = creds.split(b'\x00')
-
-	# Call the management server's "/me" method with the
-	# provided credentials
-	req = urllib.request.Request('http://127.0.0.1:10222/me')
-	req.add_header(b'Authorization', b'Basic ' + base64.b64encode(email + b':' + pw))
-	response = urllib.request.urlopen(req)
-
-	# The response is always success and always a JSON object
-	# indicating the authentication result.
-	resp = response.read().decode('utf8')
-	resp = json.loads(resp)
-	if not isinstance(resp, dict): raise ValueError("Response is not a JSON object.")
-
-except:
-	# Handle all exceptions. Print what happens (ends up in syslog, thanks
-	# to dovecot) and return an exit status that indicates temporary failure,
-	# which is passed on to the authenticating client.
-	traceback.print_exc()
-	print(json.dumps(dict(os.environ), indent=2), file=sys.stderr)
-	sys.exit(111)
-
-if resp.get('status') != 'authorized':
-	# Indicates login failure.
-	# (sys.exit should not be inside the try block.)
-	sys.exit(1)
-
-# Signal ok by executing the indicated process, per the Dovecot
-# protocol. (Note that the second parameter is the 0th argument
-# to the called process, which is required and is typically the
-# file itself.)
-os.execl(sys.argv[1], sys.argv[1])
-

conf/mozilla-autoconfig.xml

@@ -0,0 +1,44 @@
+<?xml version="1.0"?>
+<clientConfig version="1.1">
+    <emailProvider id="PRIMARY_HOSTNAME">
+      <domain>PRIMARY_HOSTNAME</domain>
+
+      <displayName>PRIMARY_HOSTNAME (Mail-in-a-Box)</displayName>
+      <displayShortName>PRIMARY_HOSTNAME</displayShortName>
+
+      <incomingServer type="imap">
+         <hostname>PRIMARY_HOSTNAME</hostname>
+         <port>993</port>
+         <socketType>SSL</socketType>
+         <username>%EMAILADDRESS%</username>
+         <authentication>password-cleartext</authentication>
+      </incomingServer>
+
+      <outgoingServer type="smtp">
+         <hostname>PRIMARY_HOSTNAME</hostname>
+         <port>587</port>
+         <socketType>STARTTLS</socketType>
+         <username>%EMAILADDRESS%</username>
+         <authentication>password-cleartext</authentication>
+         <addThisServer>true</addThisServer>
+         <useGlobalPreferredServer>true</useGlobalPreferredServer>
+      </outgoingServer>
+
+      <documentation url="https://PRIMARY_HOSTNAME/">
+         <descr lang="en">PRIMARY_HOSTNAME website.</descr>
+      </documentation>
+    </emailProvider>
+
+    <webMail>
+      <loginPage url="https://PRIMARY_HOSTNAME/mail/" />
+      <loginPageInfo url="https://PRIMARY_HOSTNAME/mail/" >
+        <username>%EMAILADDRESS%</username>
+        <usernameField id="rcmloginuser" name="_user" />
+        <passwordField id="rcmloginpwd" name="_pass" />
+        <loginButton id="rcmloginsubmit" />
+      </loginPageInfo>
+    </webMail>
+
+    <clientConfigUpdate url="https://PRIMARY_HOSTNAME/.well-known/autoconfig/mail/config-v1.1.xml" />
+
+</clientConfig>

conf/nginx.conf

@@ -7,7 +7,15 @@ server {
 
 	server_name $HOSTNAME;
 	root /tmp/invalid-path-nothing-here;
-	rewrite ^/(.*)$ https://$HOSTNAME/$1 permanent;
+
+	# Improve privacy: Hide version an OS information on
+	# error pages and in the "Server" HTTP-Header.
+	server_tokens off;
+
+	# Redirect using the 'return' directive and the built-in
+	# variable '$request_uri' to avoid any capturing, matching
+	# or evaluation of regular expressions.
+	return 301 https://$HOSTNAME$request_uri;
 }
 
 # The secure HTTPS server.
@@ -17,6 +25,10 @@ server {
 
 	server_name $HOSTNAME;
 
+	# Improve privacy: Hide version an OS information on
+	# error pages and in the "Server" HTTP-Header.
+	server_tokens off;
+
 	ssl_certificate $SSL_CERTIFICATE;
 	ssl_certificate_key $SSL_KEY;
 	include /etc/nginx/nginx-ssl.conf;
@@ -38,6 +50,16 @@ server {
 	location = /mailinabox.mobileconfig {
 		alias /var/lib/mailinabox/mobileconfig.xml;
 	}
+	location = /.well-known/autoconfig/mail/config-v1.1.xml {
+		alias /var/lib/mailinabox/mozilla-autoconfig.xml;
+	}
+
+	# Disable viewing dotfiles (.htaccess, .svn, .git, etc.)
+	location ~ /\.(ht|svn|git|hg|bzr) {
+		log_not_found off;
+		access_log off;
+		deny all;
+	}
 
 	# Roundcube Webmail configuration.
 	rewrite ^/mail$ /mail/ redirect;

management/auth.py

@@ -1,4 +1,4 @@
-import base64, os, os.path, hmac, json
+import base64, os, os.path, hmac
 
 from flask import make_response
 
@@ -97,17 +97,6 @@ class KeyAuthService:
 			# email address does not correspond to a user.
 			pw_hash = get_mail_password(email, env)
 
-			# If 2FA is set up, get the first factor and authenticate against
-			# that first.
-			twofa = None
-			if pw_hash.startswith("{TOTP}"):
-				twofa = json.loads(pw_hash[6:])
-				pw_hash = twofa["first_factor"]
-				try:
-					pw, twofa_code = pw.split(" ", 1)
-				except:
-					twofa_code = ""
-
 			# Authenticate.
 			try:
 				# Use 'doveadm pw' to check credentials. doveadm will return
@@ -122,14 +111,6 @@ class KeyAuthService:
 				# Login failed.
 				raise ValueError("Invalid password.")
 
-			# Check second factor.
-			if twofa:
-				import oath
-				ok, drift = oath.accept_totp(twofa["secret"], twofa_code, drift=twofa["drift"])
-				if not ok:
-					raise ValueError("Invalid 2FA code.")
-
-
 		# Get privileges for authorization.
 
 		# (This call should never fail on a valid user. But if it did fail, it would

management/buy_certificate.py

@@ -1,155 +0,0 @@
-#!/usr/bin/python3
-
-# Helps you purchase a SSL certificate from Gandi.net using
-# their API.
-#
-# Before you begin:
-#  1) Create an account on Gandi.net.
-#  2) Pre-pay $16 into your account at https://www.gandi.net/prepaid/operations. Wait until the payment goes through.
-#  3) Activate your API key first on the test platform (wait a while, refresh the page) and then activate the production API at https://www.gandi.net/admin/api_key.
-
-import sys, re,  os.path, urllib.request
-import xmlrpc.client
-import rtyaml
-
-from utils import load_environment, shell
-from web_update import get_web_domains, get_domain_ssl_files, get_web_root
-from status_checks import check_certificate
-
-def buy_ssl_certificate(api_key, domain, command, env):
-	if domain != env['PRIMARY_HOSTNAME'] \
-		and domain not in get_web_domains(env):
-		raise ValueError("Domain is not %s or a domain we're serving a website for." % env['PRIMARY_HOSTNAME'])
-
-	# Initialize.
-
-	gandi = xmlrpc.client.ServerProxy('https://rpc.gandi.net/xmlrpc/')
-
-	try:
-		existing_certs = gandi.cert.list(api_key)
-	except Exception as e:
-		if "Invalid API key" in str(e):
-			print("Invalid API key. Check that you copied the API Key correctly from https://www.gandi.net/admin/api_key.")
-			sys.exit(1)
-		else:
-			raise
-
-	# Where is the SSL cert stored?
-
-	ssl_key, ssl_certificate = get_domain_ssl_files(domain, env)
-
-	# Have we already created a cert for this domain?
-
-	for cert in existing_certs:
-		if cert['cn'] == domain:
-			break
-	else:
-		# No existing cert found. Purchase one.
-		if command != 'purchase':
-			print("No certificate or order found yet. If you haven't yet purchased a certificate, run ths script again with the 'purchase' command. Otherwise wait a moment and try again.")
-			sys.exit(1)
-		else:
-			# Start an order for a single standard SSL certificate.
-			# Use DNS validation. Web-based validation won't work because they
-			# require a file on HTTP but not HTTPS w/o redirects and we don't
-			# serve anything plainly over HTTP. Email might be another way but
-			# DNS is easier to automate.
-			op = gandi.cert.create(api_key, {
-				"csr": open(ssl_csr_path).read(),
-				"dcv_method": "dns",
-				"duration": 1, # year?
-				"package": "cert_std_1_0_0",
-				})
-			print("An SSL certificate has been ordered.")
-			print()
-			print(op)
-			print()
-			print("In a moment please run this script again with the 'setup' command.")
-
-	if cert['status'] == 'pending':
-		# Get the information we need to update our DNS with a code so that
-		# Gandi can verify that we own the domain.
-
-		dcv = gandi.cert.get_dcv_params(api_key, {
-				"csr": open(ssl_csr_path).read(),
-				"cert_id": cert['id'],
-				"dcv_method": "dns",
-				"duration": 1, # year?
-				"package": "cert_std_1_0_0",
-				})
-		if dcv["dcv_method"] != "dns":
-			raise Exception("Certificate ordered with an unknown validation method.")
-
-		# Update our DNS data.
-
-		dns_config = env['STORAGE_ROOT'] + '/dns/custom.yaml'
-		if os.path.exists(dns_config):
-			dns_records = rtyaml.load(open(dns_config))
-		else:
-			dns_records = { }
-
-		qname = dcv['md5'] + '.' + domain
-		value = dcv['sha1'] + '.comodoca.com.'
-		dns_records[qname] = { "CNAME": value }
-
-		with open(dns_config, 'w') as f:
-			f.write(rtyaml.dump(dns_records))
-
-		shell('check_call', ['tools/dns_update'])
-
-		# Okay, done with this step.
-
-		print("DNS has been updated. Gandi will check within 60 minutes.")
-		print()
-		print("See https://www.gandi.net/admin/ssl/%d/details for the status of this order." % cert['id'])
-
-	elif cert['status'] == 'valid':
-		# The certificate is ready.
-
-		# Check before we overwrite something we shouldn't.
-		if os.path.exists(ssl_certificate):
-			cert_status, cert_status_details = check_certificate(None, ssl_certificate, None)
-			if cert_status != "SELF-SIGNED":
-				print("Please back up and delete the file %s so I can save your new certificate." % ssl_certificate)
-				sys.exit(1)
-
-		# Form the certificate.
-
-		# The certificate comes as a long base64-encoded string. Break in
-		# into lines in the usual way.
-		pem = "-----BEGIN CERTIFICATE-----\n"
-		pem += "\n".join(chunk for chunk in re.split(r"(.{64})", cert['cert']) if chunk != "")
-		pem += "\n-----END CERTIFICATE-----\n\n"
-
-		# Append intermediary certificates.
-		pem += urllib.request.urlopen("https://www.gandi.net/static/CAs/GandiStandardSSLCA.pem").read().decode("ascii")
-
-		# Write out.
-
-		with open(ssl_certificate, "w") as f:
-			f.write(pem)
-
-		print("The certificate has been installed in %s. Restarting services..." % ssl_certificate)
-
-		# Restart dovecot and if this is for PRIMARY_HOSTNAME.
-
-		if domain == env['PRIMARY_HOSTNAME']:
-			shell('check_call', ["/usr/sbin/service", "dovecot", "restart"])
-			shell('check_call', ["/usr/sbin/service", "postfix", "restart"])
-
-		# Restart nginx in all cases.
-
-		shell('check_call', ["/usr/sbin/service", "nginx", "restart"])
-
-	else:
-		print("The certificate has an unknown status. Please check https://www.gandi.net/admin/ssl/%d/details for the status of this order." % cert['id'])
-
-if __name__ == "__main__":
-	if len(sys.argv) < 4:
-		print("Usage: python management/buy_certificate.py gandi_api_key domain_name {purchase, setup}")
-		sys.exit(1)
-	api_key = sys.argv[1]
-	domain_name = sys.argv[2]
-	cmd = sys.argv[3]
-	buy_ssl_certificate(api_key, domain_name, cmd, load_environment())
-

management/daemon.py

@@ -7,10 +7,16 @@ from functools import wraps
 from flask import Flask, request, render_template, abort, Response
 
 import auth, utils
-from mailconfig import get_mail_users, get_mail_users_ex, get_admins, add_mail_user, set_mail_password, remove_mail_user, get_mail_password
+from mailconfig import get_mail_users, get_mail_users_ex, get_admins, add_mail_user, set_mail_password, remove_mail_user
 from mailconfig import get_mail_user_privileges, add_remove_mail_user_privilege
 from mailconfig import get_mail_aliases, get_mail_aliases_ex, get_mail_domains, add_mail_alias, remove_mail_alias
 
+# Create a worker pool for the status checks. The pool should
+# live across http requests so we don't baloon the system with
+# processes.
+import multiprocessing.pool
+pool = multiprocessing.pool.Pool(processes=10)
+
 env = utils.load_environment()
 
 auth_service = auth.KeyAuthService()
@@ -40,7 +46,6 @@ def authorized_personnel_only(viewfunc):
 		# Authorized to access an API view?
 		if "admin" in privs:
 			# Call view func.	
-			request.user_email = email
 			return viewfunc(*args, **kwargs)
 		elif not error:
 			error = "You are not an administrator."
@@ -116,81 +121,6 @@ def me():
 	# Return.
 	return json_response(resp)
 
-# ME
-
-@app.route('/me/2fa')
-@authorized_personnel_only
-def twofa_status():
-	pw = get_mail_password(request.user_email, env)
-	if pw.startswith("{SHA512-CRYPT}"):
-		method = "password-only"
-	elif pw.startswith("{TOTP}"):
-		method = "TOTP 2FA"
-	else:
-		method = "unknown"
-
-	return json_response({
-		"method": method
-		})
-
-@app.route('/me/2fa/totp/initialize', methods=['POST'])
-@authorized_personnel_only
-def twofa_initialize():
-	# Generate a Google Authenticator URI that encodes TOTP info.
-	import urllib.parse, base64, qrcode, io, binascii
-
-	secret = os.urandom(32)
-	uri = "otpauth://totp/%s:%s?secret=%s&issuer=%s&digits=%d&algorithm=%s" % (
-		urllib.parse.quote(env['PRIMARY_HOSTNAME']),
-		urllib.parse.quote(request.user_email),
-		base64.b32encode(secret).decode("ascii").lower().replace("=", ""),
-		urllib.parse.quote(env['PRIMARY_HOSTNAME']),
-		6,
-		"sha1"
-		)
-
-	image_buffer = io.BytesIO()
-	im = qrcode.make(uri)
-	im.save(image_buffer, 'png')
-
-	return json_response({
-		"uri": uri,
-		"secret": binascii.hexlify(secret).decode('ascii'),
-		"qr": base64.b64encode(image_buffer.getvalue()).decode('ascii')
-		})
-
-@app.route('/me/2fa/totp/activate', methods=['POST'])
-@authorized_personnel_only
-def twofa_activate():
-	import oath
-	ok, drift = oath.accept_totp(request.form['secret'], request.form['code'])
-	if ok:
-		# use the user's current plain password as the first_factor
-		# of 2FA.
-		existing_pw = get_mail_password(request.user_email, env)
-		if existing_pw.startswith("{TOTP}"):
-			existing_pw = json.loads(existing_pw)["first_factor"]
-
-		pw = "{TOTP}" + json.dumps({
-			"secret": request.form['secret'],
-			"drift": drift,
-			"first_factor": existing_pw,
-		})
-
-		set_mail_password(request.user_email, pw, env, already_hashed=True)
-
-		return json_response({
-			"status": "ok",
-			"message": "TOTP 2FA installed."
-		})
-
-	else:
-		return json_response({
-			"status": "fail",
-			"message": "The activation code was not right. Try again?"
-		})
-
-
 # MAIL
 
 @app.route('/mail/users')
@@ -348,7 +278,7 @@ def dns_get_dump():
 @authorized_personnel_only
 def ssl_get_csr(domain):
 	from web_update import get_domain_ssl_files, create_csr
-	ssl_key, ssl_certificate = get_domain_ssl_files(domain, env)
+	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
 	return create_csr(domain, ssl_key, env)
 
 @app.route('/ssl/install', methods=['POST'])
@@ -394,7 +324,7 @@ def system_status():
 		def print_line(self, message, monospace=False):
 			self.items[-1]["extra"].append({ "text": message, "monospace": monospace })
 	output = WebOutput()
-	run_checks(env, output)
+	run_checks(env, output, pool)
 	return json_response(output.items)
 
 @app.route('/system/updates')

management/dns_update.py

@@ -122,7 +122,7 @@ def do_dns_update(env, force=False):
 		shell('check_call', ["/usr/sbin/service", "nsd", "restart"])
 
 	# Write the OpenDKIM configuration tables.
-	if write_opendkim_tables(zonefiles, env):
+	if write_opendkim_tables(domains, env):
 		# Settings changed. Kick opendkim.
 		shell('check_call', ["/usr/sbin/service", "opendkim", "restart"])
 		if len(updated_domains) == 0:
@@ -222,7 +222,11 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 	for qname, rtype, value, explanation in defaults:
 		if value is None or value.strip() == "": continue # skip IPV6 if not set
 		if not is_zone and qname == "www": continue # don't create any default 'www' subdomains on what are themselves subdomains
-		if not has_rec(qname, rtype) and not has_rec(qname, "CNAME"):
+		# Set the default record, but not if:
+		# (1) there is not a user-set record of the same type already
+		# (2) there is not a CNAME record already, since you can't set both and who knows what takes precedence
+		# (2) there is not an A record already (if this is an A record this is a dup of (1), and if this is an AAAA record then don't set a default AAAA record if the user sets a custom A record, since the default wouldn't make sense and it should not resolve if the user doesn't provide a new AAAA record)
+		if not has_rec(qname, rtype) and not has_rec(qname, "CNAME") and not has_rec(qname, "A"):
 			records.append((qname, rtype, value, explanation))
 
 	# Append the DKIM TXT record to the zone as generated by OpenDKIM.
@@ -254,6 +258,10 @@ def build_zone(domain, all_domains, additional_records, env, is_zone=True):
 
 def get_custom_records(domain, additional_records, env):
 	for qname, value in additional_records.items():
+		# We don't count the secondary nameserver config (if present) as a record - that would just be
+		# confusing to users. Instead it is accessed/manipulated directly via (get/set)_custom_dns_config.
+		if qname == "_secondary_nameserver": continue
+
 		# Is this record for the domain or one of its subdomains?
 		# If `domain` is None, return records for all domains.
 		if domain is not None and qname != domain and not qname.endswith("." + domain): continue
@@ -612,8 +620,9 @@ def sign_zone(domain, zonefile, env):
 
 ########################################################################
 
-def write_opendkim_tables(zonefiles, env):
-	# Append a record to OpenDKIM's KeyTable and SigningTable for each domain.
+def write_opendkim_tables(domains, env):
+	# Append a record to OpenDKIM's KeyTable and SigningTable for each domain
+	# that we send mail from (zones and all subdomains).
 
 	opendkim_key_file = os.path.join(env['STORAGE_ROOT'], 'mail/dkim/mail.private')
 
@@ -632,7 +641,7 @@ def write_opendkim_tables(zonefiles, env):
 		"SigningTable":
 			"".join(
 				"*@{domain} {domain}\n".format(domain=domain)
-				for domain, zonefile in zonefiles
+				for domain in domains
 			),
 
 		# The KeyTable specifies the signing domain, the DKIM selector, and the
@@ -641,7 +650,7 @@ def write_opendkim_tables(zonefiles, env):
 		"KeyTable":
 			"".join(
 				"{domain} {domain}:mail:{key_file}\n".format(domain=domain, key_file=opendkim_key_file)
-				for domain, zonefile in zonefiles
+				for domain in domains
 			),
 	}
 

management/mailconfig.py

@@ -311,16 +311,15 @@ def add_mail_user(email, pw, privs, env):
 	# Update things in case any new domains are added.
 	return kick(env, "mail user added")
 
-def set_mail_password(email, pw, env, already_hashed=False):
+def set_mail_password(email, pw, env):
 	# accept IDNA domain names but normalize to Unicode before going into database
 	email = sanitize_idn_email_address(email)
 
 	# validate that password is acceptable
-	if not already_hashed:
-		# Validate and hash the password. Skip if we're providing
-		# a raw hashed password value.
-		validate_password(pw)
-		pw = hash_password(pw)
+	validate_password(pw)
+	
+	# hash the password
+	pw = hash_password(pw)
 
 	# update the database
 	conn, c = open_database(env, with_connection=True)

management/status_checks.py

@@ -17,12 +17,12 @@ from mailconfig import get_mail_domains, get_mail_aliases
 
 from utils import shell, sort_domains, load_env_vars_from_file
 
-def run_checks(env, output):
+def run_checks(env, output, pool):
 	# run systems checks
 	output.add_heading("System")
 
 	# check that services are running
-	if not run_services_checks(env, output):
+	if not run_services_checks(env, output, pool):
 		# If critical services are not running, stop. If bind9 isn't running,
 		# all later DNS checks will timeout and that will take forever to
 		# go through, and if running over the web will cause a fastcgi timeout.
@@ -37,13 +37,21 @@ def run_checks(env, output):
 
 	# perform other checks asynchronously
 
-	pool = multiprocessing.pool.Pool(processes=1)
-	r1 = pool.apply_async(run_network_checks, [env])
-	r2 = run_domain_checks(env)
-	r1.get().playback(output)
-	r2.playback(output)
+	run_network_checks(env, output)
+	run_domain_checks(env, output, pool)
 
-def run_services_checks(env, output):
+def get_ssh_port():
+    # Returns ssh port
+    output = shell('check_output', ['sshd', '-T'])
+    returnNext = False
+
+    for e in output.split():
+        if returnNext:
+            return int(e)
+        if e == "port":
+            returnNext = True
+
+def run_services_checks(env, output, pool):
 	# Check that system services are running.
 
 	services = [
@@ -54,11 +62,12 @@ def run_services_checks(env, output):
 		{ "name": "Postgrey", "port": 10023, "public": False, },
 		{ "name": "Spamassassin", "port": 10025, "public": False, },
 		{ "name": "OpenDKIM", "port": 8891, "public": False, },
+		{ "name": "OpenDMARC", "port": 8893, "public": False, },
 		{ "name": "Memcached", "port": 11211, "public": False, },
 		{ "name": "Sieve (dovecot)", "port": 4190, "public": True, },
 		{ "name": "Mail-in-a-Box Management Daemon", "port": 10222, "public": False, },
 
-		{ "name": "SSH Login (ssh)", "port": 22, "public": True, },
+		{ "name": "SSH Login (ssh)", "port": get_ssh_port(), "public": True, },
 		{ "name": "Public DNS (nsd4)", "port": 53, "public": True, },
 		{ "name": "Incoming Mail (SMTP/postfix)", "port": 25, "public": True, },
 		{ "name": "Outgoing Mail (SMTP 587/postfix)", "port": 587, "public": True, },
@@ -70,7 +79,6 @@ def run_services_checks(env, output):
 
 	all_running = True
 	fatal = False
-	pool = multiprocessing.pool.Pool(processes=10)
 	ret = pool.starmap(check_service, ((i, service, env) for i, service in enumerate(services)), chunksize=1)
 	for i, running, fatal2, output2 in sorted(ret):
 		all_running = all_running and running
@@ -90,13 +98,28 @@ def check_service(i, service, env):
 	s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 	s.settimeout(1)
 	try:
-		s.connect((
-			"127.0.0.1" if not service["public"] else env['PUBLIC_IP'],
-			service["port"]))
-		running = True
+		try:
+			s.connect((
+				"127.0.0.1" if not service["public"] else env['PUBLIC_IP'],
+				service["port"]))
+			running = True
+		except OSError as e1:
+			if service["public"] and service["port"] != 53:
+				# For public services (except DNS), try the private IP as a fallback.
+				s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+				s1.settimeout(1)
+				try:
+					s1.connect(("127.0.0.1", service["port"]))
+					output.print_error("%s is running but is not publicly accessible at %s:%d (%s)." % (service['name'], env['PUBLIC_IP'], service['port'], str(e1)))
+				except:
+					raise e1
+				finally:
+					s1.close()
+			else:
+				raise
 
 	except OSError as e:
-		output.print_error("%s is not running (%s)." % (service['name'], str(e)))
+		output.print_error("%s is not running (%s; port %d)." % (service['name'], str(e), service['port']))
 
 		# Why is nginx not running?
 		if service["port"] in (80, 443):
@@ -162,10 +185,9 @@ def check_free_disk_space(env, output):
 	else:
 		output.print_error(disk_msg)
 
-def run_network_checks(env):
+def run_network_checks(env, output):
 	# Also see setup/network-checks.sh.
 
-	output = BufferedOutput()
 	output.add_heading("Network")
 
 	# Stop if we cannot make an outbound connection on port 25. Many residential
@@ -193,9 +215,7 @@ def run_network_checks(env):
 			which may prevent recipients from receiving your email. See http://www.spamhaus.org/query/ip/%s."""
 			% (env['PUBLIC_IP'], zen, env['PUBLIC_IP']))
 
-	return output
-
-def run_domain_checks(env):
+def run_domain_checks(env, output, pool):
 	# Get the list of domains we handle mail for.
 	mail_domains = get_mail_domains(env)
 
@@ -215,13 +235,10 @@ def run_domain_checks(env):
 	# Parallelize the checks across a worker pool.
 	args = ((domain, env, dns_domains, dns_zonefiles, mail_domains, web_domains)
 		for domain in domains_to_check)
-	pool = multiprocessing.pool.Pool(processes=10)
 	ret = pool.starmap(run_domain_checks_on_domain, args, chunksize=1)
 	ret = dict(ret) # (domain, output) => { domain: output }
-	output = BufferedOutput()
 	for domain in sort_domains(ret, env):
 		ret[domain].playback(output)
-	return output
 
 def run_domain_checks_on_domain(domain, env, dns_domains, dns_zonefiles, mail_domains, web_domains):
 	output = BufferedOutput()
@@ -399,13 +416,17 @@ def check_dnssec(domain, env, output, dns_zonefiles, is_checking_primary=False):
 def check_mail_domain(domain, env, output):
 	# Check the MX record.
 
+	recommended_mx = "10 " + env['PRIMARY_HOSTNAME']
 	mx = query_dns(domain, "MX", nxdomain=None)
-	expected_mx = "10 " + env['PRIMARY_HOSTNAME']
 
-	if mx == expected_mx:
-		output.print_ok("Domain's email is directed to this domain. [%s => %s]" % (domain, mx))
+	if mx is None:
+		mxhost = None
+	else:
+		# query_dns returns a semicolon-delimited list
+		# of priority-host pairs.
+		mxhost = mx.split('; ')[0].split(' ')[1]
 
-	elif mx == None:
+	if mxhost == None:
 		# A missing MX record is okay on the primary hostname because
 		# the primary hostname's A record (the MX fallback) is... itself,
 		# which is what we want the MX to be.
@@ -423,15 +444,22 @@ def check_mail_domain(domain, env, output):
 			else:
 				output.print_error("""This domain's DNS MX record is not set. It should be '%s'. Mail will not
 					be delivered to this box. It may take several hours for public DNS to update after a
-					change. This problem may result from other issues listed here.""" % (expected_mx,))
+					change. This problem may result from other issues listed here.""" % (recommended_mx,))
 
+	elif mxhost == env['PRIMARY_HOSTNAME']:
+		good_news = "Domain's email is directed to this domain. [%s => %s]" % (domain, mx)
+		if mx != recommended_mx:
+			good_news += "  This configuration is non-standard.  The recommended configuration is '%s'." % (recommended_mx,)
+		output.print_ok(good_news)
 	else:
 		output.print_error("""This domain's DNS MX record is incorrect. It is currently set to '%s' but should be '%s'. Mail will not
 			be delivered to this box. It may take several hours for public DNS to update after a change. This problem may result from
-			other issues listed here.""" % (mx, expected_mx))
+			other issues listed here.""" % (mx, recommended_mx))
 
-	# Check that the postmaster@ email address exists.
-	check_alias_exists("postmaster@" + domain, env, output)
+	# Check that the postmaster@ email address exists. Not required if the domain has a
+	# catch-all address or domain alias.
+	if "@" + domain not in dict(get_mail_aliases(env)):
+		check_alias_exists("postmaster@" + domain, env, output)
 
 	# Stop if the domain is listed in the Spamhaus Domain Block List.
 	# The user might have chosen a domain that was previously in use by a spammer
@@ -494,7 +522,7 @@ def check_ssl_cert(domain, env, output):
 	if query_dns(domain, "A", None) not in (env['PUBLIC_IP'], None): return
 
 	# Where is the SSL stored?
-	ssl_key, ssl_certificate = get_domain_ssl_files(domain, env)
+	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
 
 	if not os.path.exists(ssl_certificate):
 		output.print_error("The SSL certificate file for this domain is missing.")
@@ -506,7 +534,7 @@ def check_ssl_cert(domain, env, output):
 
 	if cert_status == "OK":
 		# The certificate is ok. The details has expiry info.
-		output.print_ok("SSL certificate is signed & valid. " + cert_status_details)
+		output.print_ok("SSL certificate is signed & valid. %s %s" % (ssl_via if ssl_via else "", cert_status_details))
 
 	elif cert_status == "SELF-SIGNED":
 		# Offer instructions for purchasing a signed certificate.
@@ -748,21 +776,25 @@ class BufferedOutput:
 		for attr, args, kwargs in self.buf:
 			getattr(output, attr)(*args, **kwargs)
 
+
 if __name__ == "__main__":
 	import sys
 	from utils import load_environment
 	env = load_environment()
 	if len(sys.argv) == 1:
-		run_checks(env, ConsoleOutput())
+		pool = multiprocessing.pool.Pool(processes=10)
+		run_checks(env, ConsoleOutput(), pool)
 	elif sys.argv[1] == "--check-primary-hostname":
 		# See if the primary hostname appears resolvable and has a signed certificate.
 		domain = env['PRIMARY_HOSTNAME']
 		if query_dns(domain, "A") != env['PUBLIC_IP']:
 			sys.exit(1)
-		ssl_key, ssl_certificate = get_domain_ssl_files(domain, env)
+		ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
 		if not os.path.exists(ssl_certificate):
 			sys.exit(1)
 		cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key)
 		if cert_status != "OK":
 			sys.exit(1)
 		sys.exit(0)
+
+

management/templates/2fa.html

@@ -1,79 +0,0 @@
-<style>
-</style>
-
-<h2>Two-Factor Authentication</h2>
-
-<p>Two-factor authentication (2FA) is <i>something you know</i> and <i>something you have</i>.</p>
-
-<p>Regular password-based logins are one-factor (something you know). 2FA makes an account more secure by guarding against a lost or guessed password, since you also need a special device to access your account. You can turn on 2FA for your account here.</p>
-
-<p>Your authentication method is currently: <strong id="2fa_current"> </strong></p>
-
-<h3>TOTP</h3>
-
-<p>TOTP is a time-based one-time password method of two-factor authentication.</p>
-
-<p>You will need a TOTP-compatible device, such as any Android device with the <a href="https://play.google.com/store/apps/details?id=org.fedorahosted.freeotp">FreeOTP Authenticator</a> app. We&rsquo;ll generate a QR code that you import into your device or app. After you generate the QR code, you&rsquo;ll activate 2FA by entering your first activation code provided by your device or app.</p>
-
-  <p><button onclick="totp_initialize()">Generate QR Code</button></p>
-  <div id="totp-form" class="row" style="display: none">
-    <div class="col-sm-6">
-      <center>QR Code</center>
-      <img id="totp_qr_code" src="" class="img-responsive">
-    </div>
-    <div class="col-sm-6">
-      <form class="form" role="form" onsubmit="totp_activate(); return false;">
-        <div class="form-group">
-          <label for="inputTOTP" class="control-label">Activation Code</label>
-          <p><input class="form-control" id="inputTOTP" placeholder="enter 6-digit code"></p>
-          <p><input type="submit" class="btn btn-primary" value="Activate"></p>
-        </div>
-      </form>
-    </div>
-  </div>
-
-<p>When using TOTP 2FA, your password becomes your previous plain password plus a space plus the code generated by your TOTP device.</p>
-
-<script>
-function show_2fa() {
- $('#2fa_current').text('loading...');
- api(
-    "/me/2fa",
-    "GET",
-    {
-    },
-    function(response) {
-      $('#2fa_current').text(response.method);
-    });
-}
-
-var secret = null;
-
-function totp_initialize() {
- api(
-    "/me/2fa/totp/initialize",
-    "POST",
-    {
-    },
-    function(response) {
-      $('#totp_qr_code').attr('src', 'data:image/png;base64,' + response.qr);
-      $('#totp-form').fadeIn();
-      secret = response.secret;
-    });
-}
-
-function totp_activate() {
- api(
-    "/me/2fa/totp/activate",
-    "POST",
-    {
-      "secret": secret,
-      "code": $('#inputTOTP').val()
-    },
-    function(response) {
-      show_modal_error("Two-Factor Authentication", $("<pre/>").text(response.message));
-      if (response.status == "OK")
-        $('#totp-form').fadeOut();
-    });
-}
-</script>

management/templates/index.html

@@ -115,12 +115,6 @@
             </li>
             <li><a href="#sync_guide" onclick="return show_panel(this);">Contacts/Calendar</a></li>
             <li><a href="#web" onclick="return show_panel(this);">Web</a></li>
-            <li class="dropdown">
-              <a href="#" class="dropdown-toggle" data-toggle="dropdown">You <b class="caret"></b></a>
-              <ul class="dropdown-menu">
-                <li><a href="#2fa" onclick="return show_panel(this);">Two-Factor Authentication</a></li>
-              </ul>
-            </li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
             <li><a href="#" onclick="do_logout(); return false;" style="color: white">Log out?</a></li>
@@ -174,10 +168,6 @@
       {% include "ssl.html" %}
       </div>
 
-      <div id="panel_2fa" class="admin_panel">
-      {% include "2fa.html" %}
-      </div>
-
       <hr>
 
       <footer>

management/templates/mail-guide.html

@@ -40,7 +40,7 @@
 
 				<h4>Exchange/ActiveSync settings</h4>
 
-					<p>On iOS devices and devices on this <a href="http://z-push.org/compatibility/">compatibility list</a>, you may set up your mail as an Exchange or ActiveSync server. However, we&rsquo;ve found this to be more buggy than using IMAP. If you encounter any problems, please use the manual settings above.</p>
+					<p>On iOS devices, devices on this <a href="http://z-push.org/compatibility/">compatibility list</a>, or using Outlook 2007 or later on Windows 7 and later, you may set up your mail as an Exchange or ActiveSync server. However, we&rsquo;ve found this to be more buggy than using IMAP as described above. If you encounter any problems, please use the manual settings above.</p>
 
 					<table class="table">
 					<tr><th>Server</th> <td>{{hostname}}</td></tr>

management/templates/web.html

@@ -7,7 +7,7 @@
 
 <h3>Uploading web files</h3>
 
-<p>You can replace the default website with your own HTML pages and other static files. This control panel won&rsquo;t help you design a website, but once you have <tt>.html</tt> files you can upload it following these instructions:</p>
+<p>You can replace the default website with your own HTML pages and other static files. This control panel won&rsquo;t help you design a website, but once you have <tt>.html</tt> files you can upload them following these instructions:</p>
 
 <ol>
 <li>Ensure that any domains you are publishing a website for have no problems on the <a href="#system_status" onclick="return show_panel(this);">Status Checks</a> page.</li>
@@ -18,41 +18,24 @@
 
 <li>Upload your <tt>.html</tt> or other files to the directory <tt>{{storage_root}}/www/default</tt> on this machine. They will appear directly and immediately on the web.</li>
 
-<li>The websites set up on this machine are listed in the table below with where to put the files for each website (if you have customized that, see next section).</li>
+<li>The websites set up on this machine are listed in the table below with where to put the files for each website.</li>
 
-<table id="web_domains_existing" class="table" style="margin-bottom: 2em; width: auto;">
+<table id="web_domains_existing" class="table" style="margin-bottom: 1em; width: auto;">
         <thead>
                 <tr>
                         <th>Site</th>
                         <th>Directory for Files</th>
+			<th/>
                 </tr>
         </thead>
         <tbody>
         </tbody>
 </table>
 
-<li>If you want to have this box host a static website on a domain that is not listed in the table, create a dummy <a href="#users" onclick="return show_panel(this);">mail user</a> or <a href="#aliases" onclick="return show_panel(this);">alias</a> on the domain first.</li>
+<p>To add a domain to this table, create a dummy <a href="#users" onclick="return show_panel(this);">mail user</a> or <a href="#aliases" onclick="return show_panel(this);">alias</a> on the domain first and see the <a href="https://mailinabox.email/guide.html#domain-name-configuration">setup guide</a> for adding nameserver records to the new domain at your registrar (but <i>not</i> glue records).</p>
 
 </ol>
 
-<h3>Different sites for different domains</h3>
-
-<p>Create one of the directories shown in the table below to create a space for different files for one of the websites.</p>
-
-<p>After you create one of these directories, click <button id="web_update" class="btn btn-primary" onclick="do_web_update()">Web Update</button> to restart the box&rsquo;s web server so that it sees the new website file location.</p>
-
-<table id="web_domains_custom" class="table" style="margin-bottom: 2em; width: auto;">
-        <thead>
-                <tr>
-                        <th>Site</th>
-                        <th>Create Directory</th>
-                </tr>
-        </thead>
-        <tbody>
-        </tbody>
-</table>
-
-
 <script>
 function show_web() {
  api(
@@ -64,24 +47,18 @@ function show_web() {
 	var tb = $('#web_domains_existing tbody');
 	tb.text('');
 	for (var i = 0; i < domains.length; i++) {
-		var row = $("<tr><th class='domain'><a href=''></a></th><td class='directory'><tt/></td></tr>");
+		if (!domains[i].static_enabled) continue;
+		var row = $("<tr><th class='domain'><a href=''></a></th><td class='directory'><tt/></td> <td class='change-root hidden'><button class='btn btn-default btn-xs' onclick='show_change_web_root(this)'>Change</button></td></tr>");
 		tb.append(row);
+		row.attr('data-domain', domains[i].domain);
+		row.attr('data-custom-web-root', domains[i].custom_root);
 		row.find('.domain a').text('https://' + domains[i].domain);
 		row.find('.domain a').attr('href', 'https://' + domains[i].domain);
 		row.find('.directory tt').text(domains[i].root);
+		if (domains[i].root != domains[i].custom_root)
+			row.find('.change-root').removeClass('hidden');
 	}
 
-	tb = $('#web_domains_custom tbody');
-	tb.text('');
-	for (var i = 0; i < domains.length; i++) {
-		if (domains[i].root != domains[i].custom_root) {
-			var row = $("<tr><th class='domain'><a href=''></a></th><td class='directory'><tt></td></tr>");
-			tb.append(row);
-			row.find('.domain a').text('https://' + domains[i].domain);
-			row.find('.domain a').attr('href', 'https://' + domains[i].domain);
-			row.find('.directory tt').text(domains[i].custom_root);
-		}
-	}
     });
 }
 
@@ -99,4 +76,14 @@ function do_web_update() {
       show_modal_error("Web Update", data, function() { show_web() });
     });
 }
+
+function show_change_web_root(elem) {
+  var domain = $(elem).parents('tr').attr('data-domain');
+  var root = $(elem).parents('tr').attr('data-custom-web-root');
+  show_modal_confirm(
+    'Change Root Directory for ' + domain,
+    '<p>You can change the static directory for <tt>' + domain + '</tt> to:</p> <p><tt>' + root + '</tt></p> <p>First create this directory on the server. Then click Update to scan for the directory and update web settings.',
+    'Update',
+    function() { do_web_update(); });
+}
 </script>

management/web_update.py

@@ -17,9 +17,8 @@ def get_web_domains(env):
 	domains.add(env['PRIMARY_HOSTNAME'])
 
 	# Also serve web for all mail domains so that we might at least
-	# provide Webfinger and ActiveSync auto-discover of email settings
-	# (though the latter isn't really working). These will require that
-	# an SSL cert be installed.
+	# provide auto-discover of email settings, and also a static website
+	# if the user wants to make one. These will require an SSL cert.
 	domains |= get_mail_domains(env)
 
 	# ...Unless the domain has an A/AAAA record that maps it to a different
@@ -28,6 +27,7 @@ def get_web_domains(env):
 	for domain, value in dns.items():
 		if domain not in domains: continue
 		if (isinstance(value, str) and (value != "local")) \
+		  or (isinstance(value, dict) and ("CNAME" in value)) \
 		  or (isinstance(value, dict) and ("A" in value) and (value["A"] != "local")) \
 		  or (isinstance(value, dict) and ("AAAA" in value) and (value["AAAA"] != "local")):
 			domains.remove(domain)
@@ -75,7 +75,7 @@ def make_domain_config(domain, template, template_for_primaryhost, env):
 	root = get_web_root(domain, env)
 
 	# What private key and SSL certificate will we use for this domain?
-	ssl_key, ssl_certificate = get_domain_ssl_files(domain, env)
+	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
 
 	# For hostnames created after the initial setup, ensure we have an SSL certificate
 	# available. Make a self-signed one now if one doesn't exist.
@@ -149,6 +149,7 @@ def get_domain_ssl_files(domain, env, allow_shared_cert=True):
 
 	# What SSL certificate will we use?
 	ssl_certificate_primary = os.path.join(env["STORAGE_ROOT"], 'ssl/ssl_certificate.pem')
+	ssl_via = None
 	if domain == env['PRIMARY_HOSTNAME']:
 		# For PRIMARY_HOSTNAME, use the one we generated at set-up time.
 		ssl_certificate = ssl_certificate_primary
@@ -163,8 +164,16 @@ def get_domain_ssl_files(domain, env, allow_shared_cert=True):
 			from status_checks import check_certificate
 			if check_certificate(domain, ssl_certificate_primary, None)[0] == "OK":
 				ssl_certificate = ssl_certificate_primary
+				ssl_via = "Using multi/wildcard certificate of %s." % env['PRIMARY_HOSTNAME']
 
-	return ssl_key, ssl_certificate
+			# For a 'www.' domain, see if we can reuse the cert of the parent.
+			elif domain.startswith('www.'):
+				ssl_certificate_parent = os.path.join(env["STORAGE_ROOT"], 'ssl/%s/ssl_certificate.pem' % safe_domain_name(domain[4:]))
+				if os.path.exists(ssl_certificate_parent) and check_certificate(domain, ssl_certificate_parent, None)[0] == "OK":
+					ssl_certificate = ssl_certificate_parent
+					ssl_via = "Using multi/wildcard certificate of %s." % domain[4:]
+
+	return ssl_key, ssl_certificate, ssl_via
 
 def ensure_ssl_certificate_exists(domain, ssl_key, ssl_certificate, env):
 	# For domains besides PRIMARY_HOSTNAME, generate a self-signed certificate if
@@ -219,7 +228,7 @@ def install_cert(domain, ssl_cert, ssl_chain, env):
 
 	# Do validation on the certificate before installing it.
 	from status_checks import check_certificate
-	ssl_key, ssl_certificate = get_domain_ssl_files(domain, env, allow_shared_cert=False)
+	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env, allow_shared_cert=False)
 	cert_status, cert_status_details = check_certificate(domain, fn, ssl_key)
 	if cert_status != "OK":
 		if cert_status == "SELF-SIGNED":
@@ -250,18 +259,28 @@ def install_cert(domain, ssl_cert, ssl_chain, env):
 	return "\n".join(r for r in ret if r.strip() != "")
 
 def get_web_domains_info(env):
+	# load custom settings so we can tell what domains have a redirect or proxy set up on '/',
+	# which means static hosting is not happening
+	custom_settings = { }
+	nginx_conf_custom_fn = os.path.join(env["STORAGE_ROOT"], "www/custom.yaml")
+	if os.path.exists(nginx_conf_custom_fn):
+		custom_settings = rtyaml.load(open(nginx_conf_custom_fn))
+	def has_root_proxy_or_redirect(domain):
+		return custom_settings.get(domain, {}).get('redirects', {}).get('/') or custom_settings.get(domain, {}).get('proxies', {}).get('/')
+
+	# for the SSL config panel, get cert status
 	def check_cert(domain):
 		from status_checks import check_certificate
-		ssl_key, ssl_certificate = get_domain_ssl_files(domain, env)
+		ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, env)
 		if not os.path.exists(ssl_certificate):
 			return ("danger", "No Certificate Installed")
 		cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key)
 		if cert_status == "OK":
-			if domain == env['PRIMARY_HOSTNAME'] or ssl_certificate != get_domain_ssl_files(env['PRIMARY_HOSTNAME'], env)[1]:
+			if not ssl_via:
 				return ("success", "Signed & valid. " + cert_status_details)
 			else:
 				# This is an alternate domain but using the same cert as the primary domain.
-				return ("success", "Signed & valid. Using multi/wildcard certificate of %s." % env['PRIMARY_HOSTNAME'])
+				return ("success", "Signed & valid. " + ssl_via)
 		elif cert_status == "SELF-SIGNED":
 			return ("warning", "Self-signed. Get a signed certificate to stop warnings.")
 		else:
@@ -273,6 +292,7 @@ def get_web_domains_info(env):
 			"root": get_web_root(domain, env),
 			"custom_root": get_web_root(domain, env, test_exists=False),
 			"ssl_certificate": check_cert(domain),
+			"static_enabled": not has_root_proxy_or_redirect(domain),
 		}
 		for domain in get_web_domains(env)
 	]

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.06
+	TAG=v0.07
 fi
 
 # Are we running as root?

setup/dkim.sh

@@ -10,7 +10,7 @@ source setup/functions.sh # load our functions
 source /etc/mailinabox.conf # load global vars
 
 # Install DKIM...
-apt_install opendkim opendkim-tools
+apt_install opendkim opendkim-tools opendmarc
 
 # Make sure configuration directories exist.
 mkdir -p /etc/opendkim;
@@ -48,15 +48,25 @@ fi
 chown -R opendkim:opendkim $STORAGE_ROOT/mail/dkim
 chmod go-rwx $STORAGE_ROOT/mail/dkim
 
-# Add OpenDKIM as a milter to postfix, which is how it intercepts outgoing
-# mail to perform the signing (by adding a mail header).
-# Be careful. If we add other milters later, it needs to be concatenated on the smtpd_milters line. #NODOC
+tools/editconf.py /etc/opendmarc.conf -s \
+	"Syslog=true" \
+	"Socket=inet:8893@[127.0.0.1]"
+
+# Add OpenDKIM and OpenDMARC as milters to postfix, which is how OpenDKIM
+# intercepts outgoing mail to perform the signing (by adding a mail header)
+# and how they both intercept incoming mail to add Authentication-Results
+# headers. The order possibly/probably matters: OpenDMARC relies on the
+# OpenDKIM Authentication-Results header already being present.
+#
+# Be careful. If we add other milters later, this needs to be concatenated
+# on the smtpd_milters line.
 tools/editconf.py /etc/postfix/main.cf \
-	smtpd_milters=inet:127.0.0.1:8891 \
+	"smtpd_milters=inet:127.0.0.1:8891 inet:127.0.0.1:8893"\
 	non_smtpd_milters=\$smtpd_milters \
 	milter_default_action=accept
 
 # Restart services.
 restart_service opendkim
+restart_service opendmarc
 restart_service postfix
 

setup/functions.sh

@@ -22,6 +22,20 @@ function hide_output {
 	rm -f $OUTPUT
 }
 
+function apt_get_quiet {
+	# Run apt-get in a totally non-interactive mode.
+	#
+	# Somehow all of these options are needed to get it to not ask the user
+	# questions about a) whether to proceed (-y), b) package options (noninteractive),
+	# and c) what to do about files changed locally (we don't cause that to happen but
+	# some VM providers muck with their images; -o).
+	#
+	# Although we could pass -qq to apt-get to make output quieter, many packages write to stdout
+	# and stderr things that aren't really important. Use our hide_output function to capture
+	# all of that and only show it if there is a problem (i.e. if apt_get returns a failure exit status).
+	DEBIAN_FRONTEND=noninteractive hide_output apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" "$@"
+}
+
 function apt_install {
 	# Report any packages already installed.
 	PACKAGES=$@
@@ -46,18 +60,10 @@ function apt_install {
 		echo installing $TO_INSTALL...
 	fi
 
-	# 'DEBIAN_FRONTEND=noninteractive' is to prevent dbconfig-common from asking you questions.
-	#
-	# Although we could pass -qq to apt-get to make output quieter, many packages write to stdout
-	# and stderr things that aren't really important. Use our hide_output function to capture
-	# all of that and only show it if there is a problem (i.e. if apt_get returns a failure exit status).
-	#
-	# Also note that we still include the whole original package list in the apt-get command in
+	# We still include the whole original package list in the apt-get command in
 	# case it wants to upgrade anything, I guess? Maybe we can remove it. Doesn't normally make
 	# a difference.
-	DEBIAN_FRONTEND=noninteractive \
-	hide_output \
-	apt-get -y install $PACKAGES
+	apt_get_quiet install $PACKAGES
 }
 
 function get_default_hostname {

setup/mail-users.sh

@@ -26,55 +26,31 @@ fi
 
 # ### User Authentication
 
-# Disable all of the built-in authentication mechanisms. (We formerly uncommented
-# a line to include auth-sql.conf.ext but we no longer use that.)
-sed -i "s/#*\(\!include auth-.*.conf.ext\)/#\1/"  /etc/dovecot/conf.d/10-auth.conf
+# Have Dovecot query our database, and not system users, for authentication.
+sed -i "s/#*\(\!include auth-system.conf.ext\)/#\1/"  /etc/dovecot/conf.d/10-auth.conf
+sed -i "s/#\(\!include auth-sql.conf.ext\)/\1/"  /etc/dovecot/conf.d/10-auth.conf
 
-# Legacy: Delete our old sql conf files.
-rm -f /etc/dovecot/conf.d/auth-sql.conf.ext /etc/dovecot/dovecot-sql.conf.ext
-
-# Specify how Dovecot should perform user authentication (passdb) and how it knows
-# where user mailboxes are stored (userdb).
-#
-# For passwords, we would normally have Dovecot query our mail user database
-# directly. The way to do that is commented out below. Instead, in order to
-# provide our own authentication framework so we can handle two-factor auth,
-# we will use a custom system that hooks into the Mail-in-a-Box management daemon.
-#
-# The user part of this is standard. The mailbox path and Unix system user are the
-# same for all mail users, modulo string substitution for the mailbox path that
-# Dovecot handles.
-cat > /etc/dovecot/conf.d/10-auth-mailinabox.conf << EOF;
+# Specify how the database is to be queried for user authentication (passdb)
+# and where user mailboxes are stored (userdb).
+cat > /etc/dovecot/conf.d/auth-sql.conf.ext << EOF;
 passdb {
-  driver = checkpassword
-  args = /usr/local/bin/dovecot-checkpassword
+  driver = sql
+  args = /etc/dovecot/dovecot-sql.conf.ext
 }
 userdb {
   driver = static
   args = uid=mail gid=mail home=$STORAGE_ROOT/mail/mailboxes/%d/%n
 }
 EOF
-chmod 0600 /etc/dovecot/conf.d/10-auth-mailinabox.conf
 
-# Copy dovecot-checkpassword into place.
-cp conf/dovecot-checkpassword.py /usr/local/bin/dovecot-checkpassword
-chown dovecot.dovecot /usr/local/bin/dovecot-checkpassword
-chmod 700 /usr/local/bin/dovecot-checkpassword
-
-# If we were having Dovecot query our database directly, which we did
-# originally, `/etc/dovecot/conf.d/10-auth-mailinabox.conf` would say:
-#
-#     passdb {
-#       driver = sql
-#       args = /etc/dovecot/dovecot-sql.conf.ext
-#     }
-#
-# and then `/etc/dovecot/dovecot-sql.conf.ext` (chmod 0600) would contain:
-#
-#    driver = sqlite
-#    connect = $db_path
-#    default_pass_scheme = SHA512-CRYPT
-#    password_query = SELECT email as user, password FROM users WHERE email='%u';
+# Configure the SQL to query for a user's password.
+cat > /etc/dovecot/dovecot-sql.conf.ext << EOF;
+driver = sqlite
+connect = $db_path
+default_pass_scheme = SHA512-CRYPT
+password_query = SELECT email as user, password FROM users WHERE email='%u';
+EOF
+chmod 0600 /etc/dovecot/dovecot-sql.conf.ext # per Dovecot instructions
 
 # Have Dovecot provide an authorization service that Postfix can access & use.
 cat > /etc/dovecot/conf.d/99-local-auth.conf << EOF;

setup/management.sh

@@ -5,9 +5,6 @@ source setup/functions.sh
 apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil
 hide_output pip3 install rtyaml
 
-# For two-factor authentication, the management server uses:
-hide_output pip3 install git+https://github.com/mail-in-a-box/python-oath qrcode pillow
-
 # Create a backup directory and a random key for encrypting backups.
 mkdir -p $STORAGE_ROOT/backup
 if [ ! -f $STORAGE_ROOT/backup/secret_key.txt ]; then
@@ -33,5 +30,8 @@ $(pwd)/management/backup.py
 EOF
 chmod +x /etc/cron.daily/mailinabox-backup
 
-# Start it.
+# Start it. Remove the api key file first so that start.sh
+# can wait for it to be created to know that the management
+# server is ready.
+rm -f /var/lib/mailinabox/api.key
 restart_service mailinabox

setup/network-checks.sh

@@ -1,6 +1,6 @@
 # Install the 'host', 'sed', and and 'nc' tools. This script is run before
 # the rest of the system setup so we may not yet have things installed.
-hide_output apt-get -y install bind9-host sed netcat-openbsd
+apt_get_quiet install bind9-host sed netcat-openbsd
 
 # Stop if the PRIMARY_HOSTNAME is listed in the Spamhaus Domain Block List.
 # The user might have chosen a name that was previously in use by a spammer

setup/questions.sh

@@ -4,7 +4,7 @@ if [ -z "$NONINTERACTIVE" ]; then
 	# e.g. if we piped a bootstrapping install script to bash to get started. In that
 	# case, the nifty '[ -t 0 ]' test won't work. But with Vagrant we must suppress so we
 	# use a shell flag instead. Really supress any output from installing dialog.
-	hide_output apt-get -y install dialog
+	apt_get_quiet install dialog
 	message_box "Mail-in-a-Box Installation" \
 		"Hello and thanks for deploying a Mail-in-a-Box!
 		\n\nI'm going to ask you a few questions.

setup/start.sh

@@ -87,17 +87,33 @@ if [ -z "$SKIP_NETWORK_CHECKS" ]; then
 	. setup/network-checks.sh
 fi
 
+# For the first time (if the config file (/etc/mailinabox.conf) not exists):
 # Create the user named "user-data" and store all persistent user
 # data (mailboxes, etc.) in that user's home directory.
+#
+# If the config file exists:
+# Apply the existing configuration options for STORAGE_USER/ROOT
+if [ -z "$STORAGE_USER" ]; then
+	STORAGE_USER=$([[ -z "$DEFAULT_STORAGE_USER" ]] && echo "user-data" || echo "$DEFAULT_STORAGE_USER")
+fi
+
 if [ -z "$STORAGE_ROOT" ]; then
-	STORAGE_USER=user-data
-	if [ ! -d /home/$STORAGE_USER ]; then useradd -m $STORAGE_USER; fi
-	STORAGE_ROOT=/home/$STORAGE_USER
+	STORAGE_ROOT=$([[ -z "$DEFAULT_STORAGE_ROOT" ]] && echo "/home/$STORAGE_USER" || echo "$DEFAULT_STORAGE_ROOT")
+fi
+
+# Create the STORAGE_USER if it not exists
+if ! id -u $STORAGE_USER >/dev/null 2>&1; then
+	useradd -m $STORAGE_USER
+fi
+
+# Create the STORAGE_ROOT if it not exists
+if [ ! -d $STORAGE_ROOT ]; then
 	mkdir -p $STORAGE_ROOT
 	echo $(setup/migrate.py --current) > $STORAGE_ROOT/mailinabox.version
 	chown $STORAGE_USER.$STORAGE_USER $STORAGE_ROOT/mailinabox.version
 fi
 
+
 # Save the global options in /etc/mailinabox.conf so that standalone
 # tools know where to look for data.
 cat > /etc/mailinabox.conf << EOF;
@@ -126,10 +142,13 @@ source setup/owncloud.sh
 source setup/zpush.sh
 source setup/management.sh
 
-# Write the DNS and nginx configuration files.
-sleep 5 # wait for the daemon to start
-curl -s -d POSTDATA --user $(</var/lib/mailinabox/api.key): http://127.0.0.1:10222/dns/update
-curl -s -d POSTDATA --user $(</var/lib/mailinabox/api.key): http://127.0.0.1:10222/web/update
+# Ping the management daemon to write the DNS and nginx configuration files.
+while [ ! -f /var/lib/mailinabox/api.key ]; do
+	echo Waiting for the Mail-in-a-Box management daemon to start...
+	sleep 2
+done
+tools/dns_update
+tools/web_update
 
 # If there aren't any mail users yet, create one.
 source setup/firstuser.sh

setup/system.sh

@@ -9,7 +9,7 @@ source setup/functions.sh # load our functions
 
 echo Updating system packages...
 hide_output apt-get update
-hide_output apt-get -y upgrade
+apt_get_quiet upgrade
 
 # Install basic utilities.
 #

setup/web.sh

@@ -53,6 +53,16 @@ cat conf/ios-profile.xml \
 	 > /var/lib/mailinabox/mobileconfig.xml
 chmod a+r /var/lib/mailinabox/mobileconfig.xml
 
+# Create the Mozilla Auto-configuration file which is exposed via the
+# nginx configuration at /.well-known/autoconfig/mail/config-v1.1.xml.
+# The format of the file is documented at:
+# https://wiki.mozilla.org/Thunderbird:Autoconfiguration:ConfigFileFormat
+# and https://developer.mozilla.org/en-US/docs/Mozilla/Thunderbird/Autoconfiguration/FileFormat/HowTo.
+cat conf/mozilla-autoconfig.xml \
+	| sed "s/PRIMARY_HOSTNAME/$PRIMARY_HOSTNAME/" \
+	 > /var/lib/mailinabox/mozilla-autoconfig.xml
+chmod a+r /var/lib/mailinabox/mozilla-autoconfig.xml
+
 # make a default homepage
 if [ -d $STORAGE_ROOT/www/static ]; then mv $STORAGE_ROOT/www/static $STORAGE_ROOT/www/default; fi # migration #NODOC
 mkdir -p $STORAGE_ROOT/www/default

setup/webmail.sh

@@ -30,7 +30,7 @@ apt_install \
 apt-get purge -qq -y roundcube* #NODOC
 
 # Install Roundcube from source if it is not already present or if it is out of date.
-VERSION=1.0.3
+VERSION=1.1.0
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/roundcubemail/version ]; then
 	# not installed yet #NODOC
@@ -40,7 +40,7 @@ elif [[ $VERSION != `cat /usr/local/lib/roundcubemail/version` ]]; then
 	needs_update=1 #NODOC
 fi
 if [ $needs_update == 1 ]; then
-	echo installing roudcube webmail $VERSION...
+	echo installing roundcube webmail $VERSION...
 	rm -f /tmp/roundcube.tgz
 	wget -qO /tmp/roundcube.tgz http://downloads.sourceforge.net/project/roundcubemail/roundcubemail/$VERSION/roundcubemail-$VERSION.tar.gz
 	tar -C /usr/local/lib -zxf /tmp/roundcube.tgz