v0.12c release to work-around Sourceforge outage

fixes #482

CHANGELOG.md

@@ -1,6 +1,24 @@
 CHANGELOG
 =========
 
+v0.12 (July 3, 2015)
+--------------------
+
+This is a minor update to v0.11, which was a major update. Please read v0.11's advisories.
+
+* v0.12b was posted shortly after the initial posting of v0.12 correcting a minor regression in v0.12 related to creating aliases targetting multiple addresses.
+
+* v0.12c was posted on July 19, 2015 to work around the current Sourceforge.net outage: pyzor's remote server is now hard-coded rather than accessing a file hosted on Sourceforge, and roundcube is now downloaded from a Mail-in-a-Box mirror rather than from Sourceforge.
+
+* The administrator@ alias was incorrectly created starting with v0.11. If your first install was v0.11, check that the administrator@ alias forwards mail to you.
+* Intrusion detection rules (fail2ban) are relaxed (i.e. less is blocked).
+* SSL certificates could not be installed for the new automatic 'www.' redirect domains.
+* PHP's default character encoding is changed from no default to UTF8. The effect of this change is unclear but should prevent possible future text conversion issues.
+* User-installed SSL private keys in the BEGIN PRIVATE KEY format were not accepted.
+* SSL certificates with SAN domains with IDNA encoding were broken in v0.11.
+* Some IDNA functionality was using IDNA 2003 rather than IDNA 2008.
+
+
 v0.11 (June 29, 2015)
 ---------------------
 
@@ -8,6 +26,7 @@ Advisories:
 * Users can no longer spoof arbitrary email addresses in outbound mail. When sending mail, the email address configured in your mail client must match the SMTP login username being used, or the email address must be an alias with the SMTP login username listed as one of the alias's targets.
 * This update replaces your DKIM signing key with a stronger key. Because of DNS caching/propagation, mail sent within a few hours after this update could be marked as spam by recipients. If you use External DNS, you will need to update your DNS records.
 * The box will now install software from a new Mail-in-a-Box PPA on Launchpad.net, where we are distributing two of our own packages: a patched postgrey and dovecot-lucene.
+* v0.11b was posted shortly after the initial posting of v0.11 correcting a missing dependency for the new PPA.
 
 Mail:
 * Greylisting will now let some reputable senders pass through immediately.

README.md

@@ -57,13 +57,16 @@ I sign the release tags on git. To verify that a tag is signed by me, you can pe
 	$ cd mailinabox
 
 	# Verify the tag.
-	$ git verify-tag v0.11
+	$ git verify-tag v0.12c
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
 	gpg:          There is no indication that the signature belongs to the owner.
 	Primary key fingerprint: 5F4C 0E73 13CC D744 693B  2AEA B920 41F4 C10B DD81
 
+	# Check out the tag.
+	$ git checkout v0.12c
+
 The key ID and fingerprint above should match my [Keybase.io key](https://keybase.io/joshdata) and the fingerprint I publish on [my homepage](https://razor.occams.info/).
 
 The Acknowledgements

conf/fail2ban/jail.local

@@ -1,34 +1,19 @@
-# Fail2Ban configuration file.
-# For Mail-in-a-Box
-[DEFAULT]
+# Fail2Ban configuration file for Mail-in-a-Box
 
-# bantime in seconds
-bantime  = 60
+[DEFAULT]
 
 # This should ban dumb brute-force attacks, not oblivious users.
 findtime = 30
 maxretry = 20
 
-#
 # JAILS
-#
-
-[ssh]
-
-enabled  = true
-logpath  = /var/log/auth.log
-maxretry = 20
 
 [ssh-ddos]
-
 enabled  = true
-maxretry = 20
 
 [sasl]
-
 enabled  = true
 
 [dovecot]
-
 enabled = true
 filter  = dovecotimap

management/mailconfig.py

@@ -3,6 +3,7 @@
 import subprocess, shutil, os, sqlite3, re
 import utils
 from email_validator import validate_email as validate_email_, EmailNotValidError
+import idna
 
 def validate_email(email, mode=None):
 	# Checks that an email address is syntactically valid. Returns True/False.
@@ -52,11 +53,13 @@ def sanitize_idn_email_address(email):
 	# to the underlying protocols.
 	try:
 		localpart, domainpart = email.split("@")
-		domainpart = domainpart.encode("idna").decode('ascii')
+		domainpart = idna.encode(domainpart).decode('ascii')
 		return localpart + "@" + domainpart
-	except:
-		# Domain part is not IDNA-valid, so leave unchanged. If there
-		# are non-ASCII characters it will be filtered out by
+	except (ValueError, idna.IDNAError):
+		# ValueError: String does not have a single @-sign, so it is not
+		# a valid email address. IDNAError: Domain part is not IDNA-valid.
+		# Validation is not this function's job, so return value unchanged. 
+		# If there are non-ASCII characters it will be filtered out by
 		# validate_email.
 		return email
 
@@ -65,10 +68,11 @@ def prettify_idn_email_address(email):
 	# names in IDNA in the database, but we want to show Unicode to the user.
 	try:
 		localpart, domainpart = email.split("@")
-		domainpart = domainpart.encode("ascii").decode('idna')
+		domainpart = idna.decode(domainpart.encode("ascii"))
 		return localpart + "@" + domainpart
-	except:
-		# Failed to decode IDNA. Should never happen.
+	except (ValueError, UnicodeError, idna.IDNAError):
+		# Failed to decode IDNA, or the email address does not have a
+		# single @-sign. Should never happen.
 		return email
 
 def is_dcv_address(email):
@@ -238,7 +242,7 @@ def get_domain(emailaddr, as_unicode=True):
 	# Gets the domain part of an email address. Turns IDNA
 	# back to Unicode for display.
 	ret = emailaddr.split('@', 1)[1]
-	if as_unicode: ret = ret.encode('ascii').decode('idna')
+	if as_unicode: ret = idna.decode(ret.encode('ascii'))
 	return ret
 
 def get_mail_domains(env, filter_aliases=lambda alias : True):
@@ -543,6 +547,7 @@ def kick(env, mail_result=None):
 
 		# Doesn't exist.
 		administrator = get_system_administrator(env)
+		if source == administrator: return # don't make an alias from the administrator to itself --- this alias must be created manually
 		add_mail_alias(source, administrator, env, do_kick=False)
 		results.append("added alias %s (=> %s)\n" % (source, administrator))
 

management/status_checks.py

@@ -10,6 +10,7 @@ import sys, os, os.path, re, subprocess, datetime, multiprocessing.pool
 
 import dns.reversename, dns.resolver
 import dateutil.parser, dateutil.tz
+import idna
 
 from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config, get_secondary_dns
 from web_update import get_web_domains, get_default_www_redirects, get_domain_ssl_files
@@ -259,7 +260,7 @@ def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zone
 	output = BufferedOutput()
 
 	# The domain is IDNA-encoded, but for display use Unicode.
-	output.add_heading(domain.encode('ascii').decode('idna'))
+	output.add_heading(idna.decode(domain.encode('ascii')))
 
 	if domain == env["PRIMARY_HOSTNAME"]:
 		check_primary_hostname_dns(domain, env, output, dns_domains, dns_zonefiles)
@@ -611,6 +612,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 
 	from cryptography.hazmat.primitives.asymmetric.rsa import RSAPrivateKey
 	from cryptography.x509 import Certificate, DNSName, ExtensionNotFound, OID_COMMON_NAME, OID_SUBJECT_ALTERNATIVE_NAME
+	import idna
 
 	# The ssl_certificate file may contain a chain of certificates. We'll
 	# need to split that up before we can pass anything to openssl or
@@ -625,7 +627,8 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 	# First check that the domain name is one of the names allowed by
 	# the certificate.
 	if domain is not None:
-		# The domain must be found in the Subject Common Name (CN)...
+		# The domain may be found in the Subject Common Name (CN). This comes back as an IDNA (ASCII)
+		# string, which is the format we store domains in - so good.
 		certificate_names = set()
 		try:
 			certificate_names.add(
@@ -636,11 +639,19 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 			# But we'll let it error-out when it doesn't find the domain.
 			pass
 
-		# ... or be one of the Subject Alternative Names.
+		# ... or be one of the Subject Alternative Names. The cryptography library handily IDNA-decodes
+		# the names for us. We must encode back to ASCII, but wildcard certificates can't pass through
+		# IDNA encoding/decoding so we must special-case. See https://github.com/pyca/cryptography/pull/2071.
+		def idna_decode_dns_name(dns_name):
+			if dns_name.startswith("*."):
+				return "*." + idna.encode(dns_name[2:]).decode('ascii')
+			else:
+				return idna.encode(dns_name).decode('ascii')
+				
 		try:
 			sans = cert.extensions.get_extension_for_oid(OID_SUBJECT_ALTERNATIVE_NAME).value.get_values_for_type(DNSName)
 			for san in sans:
-				certificate_names.add(san)
+				certificate_names.add(idna_decode_dns_name(san))
 		except ExtensionNotFound:
 			pass
 
@@ -654,7 +665,11 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 
 	# Second, check that the certificate matches the private key.
 	if ssl_private_key is not None:
+		try:
 			priv_key = load_pem(open(ssl_private_key, 'rb').read())
+		except ValueError as e:
+			return ("The private key file %s is not a private key file: %s" % (ssl_private_key, str(e)), None)
+
 		if not isinstance(priv_key, RSAPrivateKey):
 			return ("The private key file %s is not a private key file." % ssl_private_key, None)
 
@@ -748,8 +763,11 @@ def load_pem(pem):
 	from cryptography.x509 import load_pem_x509_certificate
 	from cryptography.hazmat.primitives import serialization
 	from cryptography.hazmat.backends import default_backend
-	pem_type = re.match(b"-+BEGIN (.*?)-+\n", pem).group(1)
-	if pem_type == b"RSA PRIVATE KEY":
+	pem_type = re.match(b"-+BEGIN (.*?)-+\n", pem)
+	if pem_type is None:
+		raise ValueError("File is not a valid PEM-formatted file.")
+	pem_type = pem_type.group(1)
+	if pem_type in (b"RSA PRIVATE KEY", b"PRIVATE KEY"):
 		return serialization.load_pem_private_key(pem, password=None, backend=default_backend())
 	if pem_type == b"CERTIFICATE":
 		return load_pem_x509_certificate(pem, default_backend())

management/web_update.py

@@ -254,7 +254,7 @@ def create_csr(domain, ssl_key, env):
                 "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain)])
 
 def install_cert(domain, ssl_cert, ssl_chain, env):
-	if domain not in get_web_domains(env):
+	if domain not in get_web_domains(env) + get_default_www_redirects(env):
 		return "Invalid domain name."
 
 	# Write the combined cert+chain to a temporary path and validate that it is OK.

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.11
+	TAG=v0.12c
 fi
 
 # Are we running as root?

setup/management.sh

@@ -5,7 +5,7 @@ source setup/functions.sh
 # build-essential libssl-dev libffi-dev python3-dev: Required to pip install cryptography.
 apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil \
 	build-essential libssl-dev libffi-dev python3-dev
-hide_output pip3 install rtyaml email_validator cryptography
+hide_output pip3 install --upgrade rtyaml email_validator idna cryptography
 	# email_validator is repeated in setup/questions.sh
 
 # Create a backup directory and a random key for encrypting backups.

setup/spamassassin.sh

@@ -22,8 +22,19 @@ apt_install spampd razor pyzor dovecot-antispam
 tools/editconf.py /etc/default/spamassassin \
 	CRON=1
 
-# Configure pyzor.
-hide_output pyzor discover
+# Configure pyzor, which is a client to a live database of hashes of
+# spam emails. Set the pyzor configuration directory to something sane.
+# The default is ~/.pyzor. We used to use that, so we'll kill that old
+# directory. Then write the public pyzor server to its servers file.
+# That will prevent an automatic download on first use, and also means
+# we can skip 'pyzor discover', both of which are currently broken by
+# something happening on Sourceforge (#496).
+rm -rf ~/.pyzor
+tools/editconf.py /etc/spamassassin/local.cf -s \
+	pyzor_options="--homedir /etc/spamassassin/pyzor"
+mkdir -p /etc/spamassassin/pyzor
+echo "public.pyzor.org:24441" > /etc/spamassassin/pyzor/servers
+# check with: pyzor --homedir /etc/mail/spamassassin/pyzor ping
 
 # Configure spampd:
 # * Pass messages on to docevot on port 10026. This is actually the default setting but we don't

setup/system.sh

@@ -11,12 +11,21 @@ source setup/functions.sh # load our functions
 # text search plugin for (and by) dovecot, which is not available in
 # Ubuntu currently.
 #
-# Add that to the system's list of repositories:
+# Add that to the system's list of repositories using add-apt-repository.
+# But add-apt-repository may not be installed. If it's not available,
+# then install it. But we have to run apt-get update before we try to
+# install anything so the package index is up to date. After adding the
+# PPA, we have to run apt-get update *again* to load the PPA's index,
+# so this must precede the apt-get update line below.
+
+if [ ! -f /usr/bin/add-apt-repository ]; then
+	echo "Installing add-apt-repository..."
+	hide_output apt-get update
+	apt_install software-properties-common
+fi
 
 hide_output add-apt-repository -y ppa:mail-in-a-box/ppa
 
-# The apt-get update in the next step will pull in the PPA's index.
-
 # ### Update Packages
 
 # Update system packages to make sure we have the latest upstream versions of things from Ubuntu.

setup/web.sh

@@ -36,6 +36,10 @@ tools/editconf.py /etc/nginx/nginx.conf -s \
 tools/editconf.py /etc/php5/fpm/php.ini -c ';' \
 	expose_php=Off
 
+# Set PHPs default charset to UTF-8, since we use it. See #367.
+tools/editconf.py /etc/php5/fpm/php.ini -c ';' \
+        default_charset="UTF-8"
+
 # Bump up PHP's max_children to support more concurrent connections
 tools/editconf.py /etc/php5/fpm/pool.d/www.conf -c ';' \
 	pm.max_children=8

setup/webmail.sh

@@ -49,7 +49,7 @@ if [ $needs_update == 1 ]; then
 	# install roundcube
 	echo installing Roundcube webmail $VERSION...
 	wget_verify \
-		http://downloads.sourceforge.net/project/roundcubemail/roundcubemail/$VERSION/roundcubemail-$VERSION.tar.gz \
+		https://mailinabox.email/mirror/roundcubemail-$VERSION.tar.gz \
 		$HASH \
 		/tmp/roundcube.tgz
 	tar -C /usr/local/lib -zxf /tmp/roundcube.tgz

tools/editconf.py

@@ -14,6 +14,8 @@
 #
 # NAME VALUE
 #
+# If the -c option is given, then the supplied character becomes the comment character
+#
 # If the -w option is given, then setting lines continue onto following
 # lines while the lines start with whitespace, e.g.:
 #
@@ -24,7 +26,7 @@ import sys, re
 
 # sanity check
 if len(sys.argv) < 3:
-	print("usage: python3 editconf.py /etc/file.conf [-s] [-w] [-t] NAME=VAL [NAME=VAL ...]")
+	print("usage: python3 editconf.py /etc/file.conf [-s] [-w] [-c <CHARACTER>] [-t] NAME=VAL [NAME=VAL ...]")
 	sys.exit(1)
 
 # parse command line arguments