v0.11b to fix missing package for apt-add-repository

---------------------

Advisories:
* Users can no longer spoof arbitrary email addresses in outbound mail. When sending mail, the email address configured in your mail client must match the SMTP login username being used, or the email address must be an alias with the SMTP login username listed as one of the alias's targets.
* This update replaces your DKIM signing key with a stronger key. Because of DNS caching/propagation, mail sent within a few hours after this update could be marked as spam by recipients. If you use External DNS, you will need to update your DNS records.
* The box will now install software from a new Mail-in-a-Box PPA on Launchpad.net, where we are distributing two of our own packages: a patched postgrey and dovecot-lucene.

Mail:
* Greylisting will now let some reputable senders pass through immediately.
* Searching mail (via IMAP) will now be much faster using the dovecot lucene full text search plugin.
* Users can no longer spoof arbitrary email addresses in outbound mail (see above).
* Fix for deleting admin@ and postmaster@ addresses.
* Roundcube is updated to version 1.1.2, plugins updated.
* Exchange/ActiveSync autoconfiguration was not working on all devices (e.g. iPhone) because of a case-sensitive URL.
* The DKIM signing key has been increased to 2048 bits, from 1024, replacing the existing key.

Web:
* 'www' subdomains now automatically redirect to their parent domain (but you'll need to install an SSL certificate).
* OCSP no longer uses Google Public DNS.
* The installed PHP version is no longer exposed through HTTP response headers, for better security.

DNS:
* Default IPv6 AAAA records were missing since version 0.09.

Control panel:
* Resetting a user's password now forces them to log in again everywhere.
* Status checks were not working if an ssh server was not installed.
* SSL certificate validation now uses the Python cryptography module in some places where openssl was used.
* There is a new tab to show the installed version of Mail-in-a-Box and to fetch the latest released version.

System:
* The munin system monitoring tool is now installed and accessible at /admin/munin.
* ownCloud updated to version 8.0.4. The ownCloud installation step now is reslient to download problems. The ownCloud configuration file is now stored in STORAGE_ROOT to fix loss of data when moving STORAGE_ROOT to a new machine.
* The setup scripts now run `apt-get update` prior to installing anything to ensure the apt database is in sync with the packages actually available.

CHANGELOG.md

@@ -1,18 +1,27 @@
 CHANGELOG
 =========
 
-In Development
+v0.11 (June 29, 2015)
---------------
+---------------------
+
+Advisories:
+* Users can no longer spoof arbitrary email addresses in outbound mail. When sending mail, the email address configured in your mail client must match the SMTP login username being used, or the email address must be an alias with the SMTP login username listed as one of the alias's targets.
+* This update replaces your DKIM signing key with a stronger key. Because of DNS caching/propagation, mail sent within a few hours after this update could be marked as spam by recipients. If you use External DNS, you will need to update your DNS records.
+* The box will now install software from a new Mail-in-a-Box PPA on Launchpad.net, where we are distributing two of our own packages: a patched postgrey and dovecot-lucene.
 
 Mail:
 * Greylisting will now let some reputable senders pass through immediately.
 * Searching mail (via IMAP) will now be much faster using the dovecot lucene full text search plugin.
+* Users can no longer spoof arbitrary email addresses in outbound mail (see above).
 * Fix for deleting admin@ and postmaster@ addresses.
 * Roundcube is updated to version 1.1.2, plugins updated.
+* Exchange/ActiveSync autoconfiguration was not working on all devices (e.g. iPhone) because of a case-sensitive URL.
+* The DKIM signing key has been increased to 2048 bits, from 1024, replacing the existing key.
 
 Web:
 * 'www' subdomains now automatically redirect to their parent domain (but you'll need to install an SSL certificate).
 * OCSP no longer uses Google Public DNS.
+* The installed PHP version is no longer exposed through HTTP response headers, for better security.
 
 DNS:
 * Default IPv6 AAAA records were missing since version 0.09.
@@ -20,10 +29,13 @@ DNS:
 Control panel:
 * Resetting a user's password now forces them to log in again everywhere.
 * Status checks were not working if an ssh server was not installed.
+* SSL certificate validation now uses the Python cryptography module in some places where openssl was used.
+* There is a new tab to show the installed version of Mail-in-a-Box and to fetch the latest released version.
 
 System:
 * The munin system monitoring tool is now installed and accessible at /admin/munin.
-* ownCloud updated to version 8.0.4.
+* ownCloud updated to version 8.0.4. The ownCloud installation step now is reslient to download problems. The ownCloud configuration file is now stored in STORAGE_ROOT to fix loss of data when moving STORAGE_ROOT to a new machine.
+* The setup scripts now run `apt-get update` prior to installing anything to ensure the apt database is in sync with the packages actually available.
 
 
 v0.10 (June 1, 2015)

README.md

@@ -57,7 +57,7 @@ I sign the release tags on git. To verify that a tag is signed by me, you can pe
 	$ cd mailinabox
 
 	# Verify the tag.
-	$ git verify-tag v0.10
+	$ git verify-tag v0.11
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
@@ -80,3 +80,4 @@ The History
 * In August 2013 I began Mail-in-a-Box by combining my own mail server configuration with the setup in ["NSA-proof your email in 2 hours"](http://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/) and making the setup steps reproducible with bash scripts.
 * Mail-in-a-Box was a semifinalist in the 2014 [Knight News Challenge](https://www.newschallenge.org/challenge/2014/submissions/mail-in-a-box), but it was not selected as a winner.
 * Mail-in-a-Box hit the front page of Hacker News in [April](https://news.ycombinator.com/item?id=7634514) 2014, [September](https://news.ycombinator.com/item?id=8276171) 2014, and [May](https://news.ycombinator.com/item?id=9624267) 2015.
+* FastCompany mentioned Mail-in-a-Box a [roundup of privacy projects](http://www.fastcompany.com/3047645/your-own-private-cloud) on June 26, 2015.

conf/nginx-alldomains.conf

@@ -55,7 +55,7 @@
 		# file upload limit to match the corresponding Postfix limit.
 		client_max_body_size 128M;
 	}
-	location /autodiscover/autodiscover.xml {
+	location ~* ^/autodiscover/autodiscover.xml$ {
 		include fastcgi_params;
 		fastcgi_param SCRIPT_FILENAME /usr/local/lib/z-push/autodiscover/autodiscover.php;
 		fastcgi_param PHP_VALUE "include_path=.:/usr/share/php:/usr/share/pear:/usr/share/awl/inc";

conf/nginx-ssl.conf

@@ -69,6 +69,6 @@ ssl_dhparam STORAGE_ROOT/ssl/dh2048.pem;
 # 8.8.8.8 and 8.8.4.4 below are Google's public IPv4 DNS servers. 
 # nginx will use them to talk to the CA.
 ssl_stapling on;
-ssl_stapling_verify off;
+ssl_stapling_verify on;
 resolver 127.0.0.1 valid=86400;
 resolver_timeout 10;

management/daemon.py

@@ -340,6 +340,24 @@ def web_update():
 
 # System
 
+@app.route('/system/version', methods=["GET"])
+@authorized_personnel_only
+def system_version():
+	from status_checks import what_version_is_this
+	try:
+		return what_version_is_this(env)
+	except Exception as e:
+		return (str(e), 500)
+
+@app.route('/system/latest-upstream-version', methods=["POST"])
+@authorized_personnel_only
+def system_latest_upstream_version():
+	from status_checks import get_latest_miab_version
+	try:
+		return get_latest_miab_version()
+	except Exception as e:
+		return (str(e), 500)
+
 @app.route('/system/status', methods=["POST"])
 @authorized_personnel_only
 def system_status():

management/dns_update.py

@@ -250,8 +250,8 @@ def build_zone(domain, all_domains, additional_records, www_redirect_domains, en
 	# Skip if the user has set a DKIM record already.
 	opendkim_record_file = os.path.join(env['STORAGE_ROOT'], 'mail/dkim/mail.txt')
 	with open(opendkim_record_file) as orf:
-		m = re.match(r'(\S+)\s+IN\s+TXT\s+\( "([^"]+)"\s+"([^"]+)"\s*\)', orf.read(), re.S)
+		m = re.match(r'(\S+)\s+IN\s+TXT\s+\( ((?:"[^"]+"\s+)+)\)', orf.read(), re.S)
-		val = m.group(2) + m.group(3)
+		val = "".join(re.findall(r'"([^"]+)"', m.group(2)))
 		if not has_rec(m.group(1), "TXT", prefix="v=DKIM1; "):
 			records.append((m.group(1), "TXT", val, "Recommended. Provides a way for recipients to verify that this machine sent @%s mail." % domain))
 
@@ -373,9 +373,16 @@ $TTL 1800           ; default time to live
 			zone += subdomain
 		zone += "\tIN\t" + querytype + "\t"
 		if querytype == "TXT":
-			value = value.replace('\\', '\\\\') # escape backslashes
+			# Divide into 255-byte max substrings.
-			value = value.replace('"', '\\"') # escape quotes
+			v2 = ""
-			value = '"' + value + '"' # wrap in quotes
+			while len(value) > 0:
+				s = value[0:255]
+				value = value[255:]
+				s = s.replace('\\', '\\\\') # escape backslashes
+				s = s.replace('"', '\\"') # escape quotes
+				s = '"' + s + '"' # wrap in quotes
+				v2 += s + " "
+			value = v2
 		zone += value + "\n"
 
 	# DNSSEC requires re-signing a zone periodically. That requires

management/status_checks.py

@@ -605,103 +605,101 @@ def check_ssl_cert(domain, rounded_time, env, output):
 			output.print_line(cert_status_details)
 			output.print_line("")
 
-def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=True, rounded_time=False):
+def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=True, rounded_time=False, just_check_domain=False):
-	# Use openssl verify to check the status of a certificate.
+	# Check that the ssl_certificate & ssl_private_key files are good
+	# for the provided domain.
 
-	# First check that the certificate is for the right domain. The domain
+	from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
-	# must be found in the Subject Common Name (CN) or be one of the
+	from cryptography.x509 import Certificate, DNSName, ExtensionNotFound, OID_COMMON_NAME, OID_SUBJECT_ALTERNATIVE_NAME
-	# Subject Alternative Names. A wildcard might also appear as the CN
-	# or in the SAN list, so check for that tool.
-	retcode, cert_dump = shell('check_output', [
-		"openssl", "x509",
-		"-in", ssl_certificate,
-		"-noout", "-text", "-nameopt", "rfc2253",
-		], trap=True)
 
-	# If the certificate is catastrophically bad, catch that now and report it.
+	# The ssl_certificate file may contain a chain of certificates. We'll
-	# More information was probably written to stderr (which we aren't capturing),
+	# need to split that up before we can pass anything to openssl or
-	# but it is probably not helpful to the user anyway.
+	# parse them in Python. Parse it with the cryptography library.
-	if retcode != 0:
+	try:
-		return ("The SSL certificate appears to be corrupted or not a PEM-formatted SSL certificate file. (%s)" % ssl_certificate, None)
+		ssl_cert_chain = load_cert_chain(ssl_certificate)
+		cert = load_pem(ssl_cert_chain[0])
+		if not isinstance(cert, Certificate): raise ValueError("This is not a certificate file.")
+	except ValueError as e:
+		return ("There is a problem with the certificate file: %s" % str(e), None)
 
-	cert_dump = cert_dump.split("\n")
+	# First check that the domain name is one of the names allowed by
-	certificate_names = set()
+	# the certificate.
-	cert_expiration_date = None
+	if domain is not None:
-	while len(cert_dump) > 0:
+		# The domain must be found in the Subject Common Name (CN)...
-		line = cert_dump.pop(0)
+		certificate_names = set()
+		try:
+			certificate_names.add(
+				cert.subject.get_attributes_for_oid(OID_COMMON_NAME)[0].value
+				)
+		except IndexError:
+			# No common name? Certificate is probably generated incorrectly.
+			# But we'll let it error-out when it doesn't find the domain.
+			pass
 
-		# Grab from the Subject Common Name. We include the indentation
+		# ... or be one of the Subject Alternative Names.
-		# at the start of the line in case maybe the cert includes the
+		try:
-		# common name of some other referenced entity (which would be
+			sans = cert.extensions.get_extension_for_oid(OID_SUBJECT_ALTERNATIVE_NAME).value.get_values_for_type(DNSName)
-		# indented, I hope).
+			for san in sans:
-		m = re.match("        Subject: CN=([^,]+)", line)
+				certificate_names.add(san)
-		if m:
+		except ExtensionNotFound:
-			certificate_names.add(m.group(1))
+			pass
-	
-		# Grab from the Subject Alternative Name, which is a comma-delim
-		# list of names, like DNS:mydomain.com, DNS:otherdomain.com.
-		m = re.match("            X509v3 Subject Alternative Name:", line)
-		if m:
-			names = re.split(",\s*", cert_dump.pop(0).strip())
-			for n in names:
-				m = re.match("DNS:(.*)", n)
-				if m:
-					certificate_names.add(m.group(1))
 
-		# Grab the expiration date for testing later.
+		# Check that the domain appears among the acceptable names, or a wildcard
-		m = re.match("            Not After : (.*)", line)
+		# form of the domain name (which is a stricter check than the specs but
-		if m:
+		# should work in normal cases).
-			cert_expiration_date = dateutil.parser.parse(m.group(1))
+		wildcard_domain = re.sub("^[^\.]+", "*", domain)
+		if domain not in certificate_names and wildcard_domain not in certificate_names:
+			return ("The certificate is for the wrong domain name. It is for %s."
+				% ", ".join(sorted(certificate_names)), None)
 
-	wildcard_domain = re.sub("^[^\.]+", "*", domain)
+	# Second, check that the certificate matches the private key.
-	if domain is not None and domain not in certificate_names and wildcard_domain not in certificate_names:
-		return ("The certificate is for the wrong domain name. It is for %s."
-			% ", ".join(sorted(certificate_names)), None)
-
-	# Second, check that the certificate matches the private key. Get the modulus of the
-	# private key and of the public key in the certificate. They should match. The output
-	# of each command looks like "Modulus=XXXXX".
 	if ssl_private_key is not None:
-		private_key_modulus = shell('check_output', [
+		priv_key = load_pem(open(ssl_private_key, 'rb').read())
-			"openssl", "rsa",
+		if not isinstance(priv_key, RSAPrivateKey):
-			"-inform", "PEM",
+			return ("The private key file %s is not a private key file." % ssl_private_key, None)
-			"-noout", "-modulus",
+
-			"-in", ssl_private_key])
+		if priv_key.public_key().public_numbers() != cert.public_key().public_numbers():
-		cert_key_modulus = shell('check_output', [
+			return ("The certificate does not correspond to the private key at %s." % ssl_private_key, None)
-			"openssl", "x509",
+
-			"-in", ssl_certificate,
+		# We could also use the openssl command line tool to get the modulus
-			"-noout", "-modulus"])
+		# listed in each file. The output of each command below looks like "Modulus=XXXXX".
-		if private_key_modulus != cert_key_modulus:
+		# $ openssl rsa -inform PEM -noout -modulus -in ssl_private_key
-			return ("The certificate installed at %s does not correspond to the private key at %s." % (ssl_certificate, ssl_private_key), None)
+		# $ openssl x509 -in ssl_certificate -noout -modulus
+
+	# Third, check if the certificate is self-signed. Return a special flag string.
+	if cert.issuer == cert.subject:
+		return ("SELF-SIGNED", None)
+
+	# When selecting which certificate to use for non-primary domains, we check if the primary
+	# certificate or a www-parent-domain certificate is good for the domain. There's no need
+	# to run extra checks beyond this point.
+	if just_check_domain:
+		return ("OK", None)
+
+	# Check that the certificate hasn't expired. The datetimes returned by the
+	# certificate are 'naive' and in UTC. We need to get the current time in UTC.
+	now = datetime.datetime.utcnow()
+	if not(cert.not_valid_before <= now <= cert.not_valid_after):
+		return ("The certificate has expired or is not yet valid. It is valid from %s to %s." % (cert.not_valid_before, cert.not_valid_after), None)
 
 	# Next validate that the certificate is valid. This checks whether the certificate
 	# is self-signed, that the chain of trust makes sense, that it is signed by a CA
 	# that Ubuntu has installed on this machine's list of CAs, and I think that it hasn't
 	# expired.
 
-	# In order to verify with openssl, we need to split out any
+	# The certificate chain has to be passed separately and is given via STDIN.
-	# intermediary certificates in the chain (if any) from our
-	# certificate (at the top). They need to be passed separately.
-
-	cert = open(ssl_certificate).read()
-	m = re.match(r'(-*BEGIN CERTIFICATE-*.*?-*END CERTIFICATE-*)(.*)', cert, re.S)
-	if m == None:
-		return ("The certificate file is an invalid PEM certificate.", None)
-	mycert, chaincerts = m.groups()
-
 	# This command returns a non-zero exit status in most cases, so trap errors.
-
 	retcode, verifyoutput = shell('check_output', [
 		"openssl",
 		"verify", "-verbose",
 		"-purpose", "sslserver", "-policy_check",]
-		+ ([] if chaincerts.strip() == "" else ["-untrusted", "/dev/stdin"])
+		+ ([] if len(ssl_cert_chain) == 1 else ["-untrusted", "/dev/stdin"])
 		+ [ssl_certificate],
-		input=chaincerts.encode('ascii'),
+		input=b"\n\n".join(ssl_cert_chain[1:]),
 		trap=True)
 
 	if "self signed" in verifyoutput:
-		# Certificate is self-signed.
+		# Certificate is self-signed. Probably we detected this above.
 		return ("SELF-SIGNED", None)
 
 	elif retcode != 0:
@@ -716,7 +714,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 		# good.
 
 		# But is it expiring soon?
-		now = datetime.datetime.now(dateutil.tz.tzlocal())
+		cert_expiration_date = cert.not_valid_after
 		ndays = (cert_expiration_date-now).days
 		if not rounded_time or ndays < 7:
 			expiry_info = "The certificate expires in %d days on %s." % (ndays, cert_expiration_date.strftime("%x"))
@@ -733,6 +731,30 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 		# Return the special OK code.
 		return ("OK", expiry_info)
 
+def load_cert_chain(pemfile):
+	# A certificate .pem file may contain a chain of certificates.
+	# Load the file and split them apart.
+	re_pem = rb"(-+BEGIN (?:.+)-+[\r\n](?:[A-Za-z0-9+/=]{1,64}[\r\n])+-+END (?:.+)-+[\r\n])"
+	with open(pemfile, "rb") as f:
+		pem = f.read() + b"\n" # ensure trailing newline
+		pemblocks = re.findall(re_pem, pem)
+		if len(pemblocks) == 0:
+			raise ValueError("File does not contain valid PEM data.")
+		return pemblocks
+
+def load_pem(pem):
+	# Parse a "---BEGIN .... END---" PEM string and return a Python object for it
+	# using classes from the cryptography package.
+	from cryptography.x509 import load_pem_x509_certificate
+	from cryptography.hazmat.primitives import serialization
+	from cryptography.hazmat.backends import default_backend
+	pem_type = re.match(b"-+BEGIN (.*?)-+\n", pem).group(1)
+	if pem_type == b"RSA PRIVATE KEY":
+		return serialization.load_pem_private_key(pem, password=None, backend=default_backend())
+	if pem_type == b"CERTIFICATE":
+		return load_pem_x509_certificate(pem, default_backend())
+	raise ValueError("Unsupported PEM object type: " + pem_type.decode("ascii", "replace"))
+
 _apt_updates = None
 def list_apt_updates(apt_update=True):
 	# See if we have this information cached recently.
@@ -767,6 +789,20 @@ def list_apt_updates(apt_update=True):
 
 	return pkgs
 
+def what_version_is_this(env):
+	# This function runs `git describe` on the Mail-in-a-Box installation directory.
+	# Git may not be installed and Mail-in-a-Box may not have been cloned from github,
+	# so this function may raise all sorts of exceptions.
+	miab_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+	tag = shell("check_output", ["/usr/bin/git", "describe"], env={"GIT_DIR": os.path.join(miab_dir, '.git')}).strip()
+	return tag
+
+def get_latest_miab_version():
+	# This pings https://mailinabox.email/bootstrap.sh and extracts the tag named in
+	# the script to determine the current product version.
+	import urllib.request
+	return re.search(b'TAG=(.*)', urllib.request.urlopen("https://mailinabox.email/bootstrap.sh?ping=1").read()).group(1).decode("utf8")
+
 def run_and_output_changes(env, pool, send_via_email):
 	import json
 	from difflib import SequenceMatcher
@@ -947,3 +983,6 @@ if __name__ == "__main__":
 		if cert_status != "OK":
 			sys.exit(1)
 		sys.exit(0)
+
+	elif sys.argv[1] == "--version":
+		print(what_version_is_this(env))

management/templates/index.html

@@ -114,6 +114,7 @@
             </li>
             <li><a href="#sync_guide" onclick="return show_panel(this);">Contacts/Calendar</a></li>
             <li><a href="#web" onclick="return show_panel(this);">Web</a></li>
+            <li><a href="#version" onclick="return show_panel(this);">Version</a></li>
           </ul>
           <ul class="nav navbar-nav navbar-right">
             <li><a href="#" onclick="do_logout(); return false;" style="color: white">Log out?</a></li>
@@ -167,6 +168,10 @@
       {% include "ssl.html" %}
       </div>
 
+      <div id="panel_version" class="admin_panel">
+      {% include "version.html" %}
+      </div>
+
       <hr>
 
       <footer>

management/templates/version.html

@@ -0,0 +1,36 @@
+<style>
+</style>
+
+<h2>Mail-in-a-Box Version</h2>
+
+<p>You are running Mail-in-a-Box version <span id="miab-version" style="font-weight: bold">...</span>.</p>
+
+<p>The latest version of Mail-in-a-Box is <button id="miab-get-latest-upstream" onclick="check_latest_version()">Check</button>.</p>
+
+<p>To find the latest version and for upgrade instructions, see <a href="https://mailinabox.email/">https://mailinabox.email/</a>, <a href="https://github.com/mail-in-a-box/mailinabox/blob/master/CHANGELOG.md">release notes</a>, and <a href="https://mailinabox.email/maintenance.html#updating-mail-in-a-box">upgrade instructions</a>.</p>
+
+<script>
+function show_version() {
+ $('#miab-version').text('loading...');
+ api(
+    "/system/version",
+    "GET",
+    {
+    },
+    function(version) {
+      $('#miab-version').text(version);
+    });
+}
+
+function check_latest_version() {
+  $('#miab-get-latest-upstream').text('loading...');
+  api(
+   "/system/latest-upstream-version",
+   "POST",
+   {
+   },
+   function(version) {
+      $('#miab-get-latest-upstream').text(version);
+   });
+}
+</script>

management/web_update.py

@@ -201,14 +201,14 @@ def get_domain_ssl_files(domain, env, allow_shared_cert=True):
 		# the user has uploaded a different private key for this domain.
 		if not ssl_key_is_alt and allow_shared_cert:
 			from status_checks import check_certificate
-			if check_certificate(domain, ssl_certificate_primary, None)[0] == "OK":
+			if check_certificate(domain, ssl_certificate_primary, None, just_check_domain=True)[0] == "OK":
 				ssl_certificate = ssl_certificate_primary
 				ssl_via = "Using multi/wildcard certificate of %s." % env['PRIMARY_HOSTNAME']
 
 			# For a 'www.' domain, see if we can reuse the cert of the parent.
 			elif domain.startswith('www.'):
 				ssl_certificate_parent = os.path.join(env["STORAGE_ROOT"], 'ssl/%s/ssl_certificate.pem' % safe_domain_name(domain[4:]))
-				if os.path.exists(ssl_certificate_parent) and check_certificate(domain, ssl_certificate_parent, None)[0] == "OK":
+				if os.path.exists(ssl_certificate_parent) and check_certificate(domain, ssl_certificate_parent, None, just_check_domain=True)[0] == "OK":
 					ssl_certificate = ssl_certificate_parent
 					ssl_via = "Using multi/wildcard certificate of %s." % domain[4:]
 

security.md

@@ -85,7 +85,11 @@ If the recipient's domain name supports DNSSEC and has published a [DANE TLSA](h
 
 ### Domain Policy Records
 
-Domain policy records allow recipient MTAs to detect when the _domain_ part of incoming mail has been spoofed. All outbound mail is signed with [DKIM](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) and "quarantine" [DMARC](https://en.wikipedia.org/wiki/DMARC) records are automatically set in DNS. Receiving MTAs that implement DMARC will automatically quarantine mail that is "From:" a domain hosted by the box but which was not sent by the box. (Strong [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records are also automatically set in DNS.) ([source](management/dns_update.py))
+Domain policy records allow recipient MTAs to detect when the _domain_ part of of the sender address in incoming mail has been spoofed. All outbound mail is signed with [DKIM](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) and "quarantine" [DMARC](https://en.wikipedia.org/wiki/DMARC) records are automatically set in DNS. Receiving MTAs that implement DMARC will automatically quarantine mail that is "From:" a domain hosted by the box but which was not sent by the box. (Strong [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) records are also automatically set in DNS.) ([source](management/dns_update.py))
+
+### User Policy
+
+While domain policy records prevent other servers from sending mail with a "From:" header that matches a domain hosted on the box (see above), those policy records do not guarnatee that the user portion of the sender email address matches the actual sender. In enterprise environments where the box may host the mail of untrusted users, it is important to guard against users impersonating other users. The box restricts the envelope sender address that users may put into outbound mail to either a) their own email address (their SMTP login username) or b) any alias that they are listed as a direct recipient of. Note that the envelope sender address is not the same as the "From:" header.
 
 Incoming Mail
 -------------

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.10
+	TAG=v0.11b
 fi
 
 # Are we running as root?

setup/dkim.sh

@@ -35,13 +35,18 @@ RequireSafeKeys         false
 EOF
 fi
 
-# Create a new DKIM key. This creates
+# Create a new DKIM key. This creates mail.private and mail.txt
-# mail.private and mail.txt in $STORAGE_ROOT/mail/dkim. The former
+# in $STORAGE_ROOT/mail/dkim. The former is the private key and
-# is the actual private key and the latter is the suggested DNS TXT
+# the latter is the suggested DNS TXT entry which we'll include
-# entry which we'll want to include in our DNS setup.
+# in our DNS setup. Note tha the files are named after the
+# 'selector' of the key, which we can change later on to support
+# key rotation.
+#
+# A 1024-bit key is seen as a minimum standard by several providers
+# such as Google. But they and others use a 2048 bit key, so we'll
+# do the same. Keys beyond 2048 bits may exceed DNS record limits.
 if [ ! -f "$STORAGE_ROOT/mail/dkim/mail.private" ]; then
-	# Should we specify -h rsa-sha256?
+	opendkim-genkey -b 2048 -r -s mail -D $STORAGE_ROOT/mail/dkim
-	opendkim-genkey -r -s mail -D $STORAGE_ROOT/mail/dkim
 fi
 
 # Ensure files are owned by the opendkim user and are private otherwise.

setup/mail-postfix.sh

@@ -160,6 +160,7 @@ tools/editconf.py /etc/postfix/main.cf virtual_transport=lmtp:[127.0.0.1]:10025
 #
 # * `reject_non_fqdn_sender`: Reject not-nice-looking return paths.
 # * `reject_unknown_sender_domain`: Reject return paths with invalid domains.
+# * `reject_authenticated_sender_login_mismatch`: Reject if mail FROM address does not match the client SASL login
 # * `reject_rhsbl_sender`: Reject return paths that use blacklisted domains.
 # * `permit_sasl_authenticated`: Authenticated users (i.e. on port 587) can skip further checks.
 # * `permit_mynetworks`: Mail that originates locally can skip further checks.
@@ -173,7 +174,7 @@ tools/editconf.py /etc/postfix/main.cf virtual_transport=lmtp:[127.0.0.1]:10025
 # whitelisted) then postfix does a DEFER_IF_REJECT, which results in all "unknown user" sorts of messages turning into #NODOC
 # "450 4.7.1 Client host rejected: Service unavailable". This is a retry code, so the mail doesn't properly bounce. #NODOC
 tools/editconf.py /etc/postfix/main.cf \
-	smtpd_sender_restrictions="reject_non_fqdn_sender,reject_unknown_sender_domain,reject_rhsbl_sender dbl.spamhaus.org" \
+	smtpd_sender_restrictions="reject_non_fqdn_sender,reject_unknown_sender_domain,reject_authenticated_sender_login_mismatch,reject_rhsbl_sender dbl.spamhaus.org" \
 	smtpd_recipient_restrictions=permit_sasl_authenticated,permit_mynetworks,"reject_rbl_client zen.spamhaus.org",reject_unlisted_recipient,"check_policy_service inet:127.0.0.1:10023"
 
 # Postfix connects to Postgrey on the 127.0.0.1 interface specifically. Ensure that

setup/mail-users.sh

@@ -69,6 +69,22 @@ tools/editconf.py /etc/postfix/main.cf \
 	smtpd_sasl_path=private/auth \
 	smtpd_sasl_auth_enable=yes
 
+# ### Sender Validation
+
+# Use a Sqlite3 database to set login maps. This is used with
+# reject_authenticated_sender_login_mismatch to see if user is
+# allowed to send mail using FROM field specified in the request.
+tools/editconf.py /etc/postfix/main.cf \
+	smtpd_sender_login_maps=sqlite:/etc/postfix/sender-login-maps.cf
+
+# SQL statement to set login map which includes the case when user is
+# sending email using a valid alias.
+# This is the same as virtual-alias-maps.cf, See below
+cat > /etc/postfix/sender-login-maps.cf << EOF;
+dbpath=$db_path
+query = SELECT destination from (SELECT destination, 0 as priority FROM aliases WHERE source='%s' UNION SELECT email as destination, 1 as priority FROM users WHERE email='%s') ORDER BY priority LIMIT 1;
+EOF
+
 # ### Destination Validation
 
 # Use a Sqlite3 database to check whether a destination email address exists,
@@ -92,13 +108,25 @@ query = SELECT 1 FROM users WHERE email='%s'
 EOF
 
 # SQL statement to rewrite an email address if an alias is present.
-# Aliases have precedence over users, but that's counter-intuitive for
+#
-# catch-all aliases ("@domain.com") which should *not* catch mail users.
+# Postfix makes multiple queries for each incoming mail. It first
-# To fix this, not only query the aliases table but also the users
+# queries the whole email address, then just the user part in certain
-# table, i.e. turn users into aliases from themselves to themselves.
+# locally-directed cases (but we don't use this), then just `@`+the
+# domain part. The first query that returns something wins. See
+# http://www.postfix.org/virtual.5.html.
+#
+# virtual-alias-maps has precedence over virtual-mailbox-maps, but
+# we don't want catch-alls and domain aliases to catch mail for users
+# that have been defined on those domains. To fix this, we not only
+# query the aliases table but also the users table when resolving
+# aliases, i.e. we turn users into aliases from themselves to
+# themselves. That means users will match in postfix's first query
+# before postfix gets to the third query for catch-alls/domain alises.
+#
 # If there is both an alias and a user for the same address either
 # might be returned by the UNION, so the whole query is wrapped in
-# another select that prioritizes the alias definition.
+# another select that prioritizes the alias definition to preserve
+# postfix's preference for aliases for whole email addresses.
 cat > /etc/postfix/virtual-alias-maps.cf << EOF;
 dbpath=$db_path
 query = SELECT destination from (SELECT destination, 0 as priority FROM aliases WHERE source='%s' UNION SELECT email as destination, 1 as priority FROM users WHERE email='%s') ORDER BY priority LIMIT 1;

setup/management.sh

@@ -2,8 +2,10 @@
 
 source setup/functions.sh
 
-apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil
+# build-essential libssl-dev libffi-dev python3-dev: Required to pip install cryptography.
-hide_output pip3 install rtyaml "email_validator==0.1.0-rc5"
+apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil \
+	build-essential libssl-dev libffi-dev python3-dev
+hide_output pip3 install rtyaml email_validator cryptography
 	# email_validator is repeated in setup/questions.sh
 
 # Create a backup directory and a random key for encrypting backups.

setup/migrate.py

@@ -95,6 +95,11 @@ def migration_7(env):
 	# Save.
 	conn.commit()
 
+def migration_8(env):
+	# Delete DKIM keys. We had generated 1024-bit DKIM keys.
+	# By deleting the key file we'll automatically generate
+	# a new key, which will be 2048 bits.
+	os.unlink(os.path.join(env['STORAGE_ROOT'], 'mail/dkim/mail.private'))
 
 def get_current_migration():
 	ver = 0

setup/owncloud.sh

@@ -34,16 +34,17 @@ fi
 if [ ! -d /usr/local/lib/owncloud/ ] \
 	|| ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then
 
+	# Download and verify
+	echo "installing ownCloud..."
+	wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
+
 	# Clear out the existing ownCloud.
-	if [ ! -d /usr/local/lib/owncloud/ ]; then
+	if [ -d /usr/local/lib/owncloud/ ]; then
-		echo installing ownCloud...
-	else
 		echo "upgrading ownCloud to $owncloud_ver (backing up existing ownCloud directory to /tmp/owncloud-backup-$$)..."
 		mv /usr/local/lib/owncloud /tmp/owncloud-backup-$$
 	fi
 
-	# Download and extract ownCloud.
+	# Extract ownCloud
-	wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
 	unzip -u -o -q /tmp/owncloud.zip -d /usr/local/lib #either extracts new or replaces current files
 	rm -f /tmp/owncloud.zip
 

setup/questions.sh

@@ -13,7 +13,7 @@ if [ -z "$NONINTERACTIVE" ]; then
 	fi
 
 	# email_validator is repeated in setup/management.sh
-	hide_output pip3 install "email_validator==0.1.0-rc5" || exit 1
+	hide_output pip3 install email_validator || exit 1
 
 	message_box "Mail-in-a-Box Installation" \
 		"Hello and thanks for deploying a Mail-in-a-Box!

setup/system.sh

@@ -11,12 +11,21 @@ source setup/functions.sh # load our functions
 # text search plugin for (and by) dovecot, which is not available in
 # Ubuntu currently.
 #
-# Add that to the system's list of repositories:
+# Add that to the system's list of repositories using add-apt-repository.
+# But add-apt-repository may not be installed. If it's not available,
+# then install it. But we have to run apt-get update before we try to
+# install anything so the package index is up to date. After adding the
+# PPA, we have to run apt-get update *again* to load the PPA's index,
+# so this must precede the apt-get update line below.
+
+if [ ! -f /usr/bin/add-apt-repository ]; then
+	echo "Installing add-apt-repository..."
+	hide_output apt-get update
+	apt_install software-properties-common
+fi
 
 hide_output add-apt-repository -y ppa:mail-in-a-box/ppa
 
-# The apt-get update in the next step will pull in the PPA's index.
-
 # ### Update Packages
 
 # Update system packages to make sure we have the latest upstream versions of things from Ubuntu.

tools/parse-nginx-log-bootstrap-accesses.py

@@ -25,6 +25,7 @@ for fn in glob.glob("/var/log/nginx/access.log*"):
 	with f:
 		for line in f:
 			# Find lines that are GETs on /bootstrap.sh by either curl or wget.
+			# (Note that we purposely skip ...?ping=1 requests which is the admin panel querying us for updates.)
 			m = re.match(rb"(?P<ip>\S+) - - \[(?P<date>.*?)\] \"GET /bootstrap.sh HTTP/.*\" 200 \d+ .* \"(?:curl|wget)", line, re.I)
 			if m:
 				date, time = m.group("date").decode("ascii").split(":", 1)