v0.17b

Fixes #741.

CHANGELOG.md

@@ -1,7 +1,72 @@
 CHANGELOG
 =========
 
-v0.15 Release Candidate
+v0.17b (March 1, 2016)
+----------------------
+
+ownCloud moved their source code to a new location, breaking our installation script.
+
+v0.17 (February 25, 2016)
+-------------------------
+
+Mail:
+
+* Roundcube updated to version 1.1.4.
+* When there's a problem delivering an outgoing message, a new 'warning' bounce will come after 3 hours and the box will stop trying after 2 days (instead of 5).
+* On multi-homed machines, Postfix now binds to the right network interface when sending outbound mail so that SPF checks on the receiving end will pass.
+* Mail sent from addresses on subdomains of other domains hosted by this box would not be DKIM-signed and so would fail DMARC checks by recipients, since version v0.15.
+
+Control panel:
+
+* TLS certificate provisioning would crash if DNS propagation was in progress and a challenge failed; might have shown the wrong error when provisioning fails.
+* Backup times were displayed with the wrong time zone.
+* Thresholds for displaying messages when the system is running low on memory have been reduced from 30% to 20% for a warning and from 15% to 10% for an error.
+* Other minor fixes.
+
+System:
+
+* Backups to some AWS S3 regions broke in version 0.15 because we reverted the version of boto. That's now fixed.
+* On low-usage systems, don't hold backups for quite so long by taking a full backup more often.
+* Nightly status checks might fail on systems not configured with a default Unicode locale.
+* If domains need a TLS certificate and the user hasn't installed one yet using Let's Encrypt, the administrator would get a nightly email with weird interactive text asking them to agree to Let's Encrypt's ToS. Now just say that the provisioning can't be done automatically.
+* Reduce the number of background processes used by the management daemon to lower memory consumption.
+
+Setup:
+
+* The first screen now warns users not to install on a machine used for other things.
+
+v0.16 (January 30, 2016)
+------------------------
+
+This update primarily adds automatic SSL (now "TLS") certificate provisioning from Let's Encrypt (https://letsencrypt.org/).
+* The Sieve port is now open so tools like the Thunderbird Sieve program can be used to edit mail filters.
+
+Control Panel:
+
+* The SSL certificates (now referred to as "TLS ccertificates") page now supports provisioning free certificates from Let's Encrypt.
+* Report free memory usage.
+* Fix a crash when the git directory is not checked out to a tag.
+* When IPv6 is enabled, check that all domains (besides the system hostname) resolve over IPv6.
+* When a domain doesn't resolve to the box, don't bother checking if the TLS certificate is valid.
+* Remove rounded border on the menu bar.
+
+Other:
+
+* The Sieve port is now open so tools like the Thunderbird Sieve extension can be used to edit mail filters.
+* .be domains now offer DNSSEC options supported by the TLD
+* The daily backup will now email the administrator if there is a problem.
+* Expiring TLS certificates are now automatically renewed via Let's Encrypt.
+* File ownership for installed Roundcube files is fixed.
+* Typos fixed.
+
+v0.15a (January 9, 2016)
+------------------------
+
+Mail:
+
+* Sending mail through Exchange/ActiveSync (Z-Push) had been broken since v0.14 in some setups. This is now fixed.
+
+v0.15 (January 1, 2016)
 -----------------------
 
 Mail:
@@ -34,7 +99,6 @@ System:
 * Fix pip command that might have not installed all necessary Python packages.
 * The control panel and backup would not work on Google Compute Engine because GCE installs a conflicting boto package.
 * Added a new command `management/backup.py --restore` to restore files from a backup to a target directory (command line arguments are passed to `duplicity restore`).
-* Update Roundcube to version 1.1.3.
 
 v0.14 (November 4, 2015)
 ------------------------

README.md

@@ -59,20 +59,20 @@ by me:
 	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
 	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported
 
-	$ git verify-tag v0.14
+	$ git verify-tag v0.17b
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
 	gpg:          There is no indication that the signature belongs to the owner.
 	Primary key fingerprint: 5F4C 0E73 13CC D744 693B  2AEA B920 41F4 C10B DD81
 
-You'll get a lot of warnings, but that's OK. Check that the primary key fingerprint matchs the
+You'll get a lot of warnings, but that's OK. Check that the primary key fingerprint matches the
 fingerprint in the key details at [https://keybase.io/joshdata](https://keybase.io/joshdata)
 and on my [personal homepage](https://razor.occams.info/). (Of course, if this repository has been compromised you can't trust these instructions.)
 
 Checkout the tag corresponding to the most recent release:
 
-	$ git checkout v0.14
+	$ git checkout v0.17b
 
 Begin the installation.
 

conf/management-initscript

@@ -27,9 +27,9 @@ EXEC_AS_USER=root
 
 # Ensure Python reads/writes files in UTF-8. If the machine
 # triggers some other locale in Python, like ASCII encoding,
-# Python may not be able to read/write files. Here and in
+# Python may not be able to read/write files. Set also
 # setup/start.sh (where the locale is also installed if not
-# already present).
+# already present) and management/daily_tasks.sh.
 export LANGUAGE=en_US.UTF-8
 export LC_ALL=en_US.UTF-8
 export LANG=en_US.UTF-8

conf/nginx.conf

@@ -1,6 +1,8 @@
 ## $HOSTNAME
 
-# Redirect all HTTP to HTTPS.
+# Redirect all HTTP to HTTPS *except* the ACME challenges (Let's Encrypt TLS certificate
+# domain validation challenges) path, which must be served over HTTP per the ACME spec
+# (due to some Apache vulnerability).
 server {
 	listen 80;
 	listen [::]:80;
@@ -12,10 +14,19 @@ server {
 	# error pages and in the "Server" HTTP-Header.
 	server_tokens off;
 
-	# Redirect using the 'return' directive and the built-in
-	# variable '$request_uri' to avoid any capturing, matching
-	# or evaluation of regular expressions.
-	return 301 https://$HOSTNAME$request_uri;
+	location / {
+		# Redirect using the 'return' directive and the built-in
+		# variable '$request_uri' to avoid any capturing, matching
+		# or evaluation of regular expressions.
+		return 301 https://$HOSTNAME$request_uri;
+	}
+
+	location /.well-known/acme-challenge/ {
+		# This path must be served over HTTP for ACME domain validation.
+		# We map this to a special path where our TLS cert provisioning
+		# tool knows to store challenge response files.
+		alias $STORAGE_ROOT/ssl/lets_encrypt/acme_challenges/;
+	}
 }
 
 # The secure HTTPS server.

conf/zpush/backend_imap.php

@@ -41,6 +41,7 @@ define('IMAP_FROM_LDAP_QUERY', '(mail=#username@#domain)');
 define('IMAP_FROM_LDAP_FIELDS', serialize(array('givenname', 'sn', 'mail')));
 define('IMAP_FROM_LDAP_FROM', '#givenname #sn <#mail>');
 
+define('IMAP_SMTP_METHOD', 'sendmail');
 
 global $imap_smtp_params;
 $imap_smtp_params = array('host' => 'ssl://localhost', 'port' => 587, 'auth' => true, 'username' => 'imap_username', 'password' => 'imap_password');

management/backup.py

@@ -1,14 +1,12 @@
 #!/usr/bin/python3
 
 # This script performs a backup of all user data:
-# 1) System services are stopped while a copy of user data is made.
-# 2) An incremental encrypted backup is made using duplicity into the
-#    directory STORAGE_ROOT/backup/encrypted. The password used for
-#    encryption is stored in backup/secret_key.txt.
+# 1) System services are stopped.
+# 2) An incremental encrypted backup is made using duplicity.
 # 3) The stopped services are restarted.
-# 5) STORAGE_ROOT/backup/after-backup is executd if it exists.
+# 4) STORAGE_ROOT/backup/after-backup is executd if it exists.
 
-import os, os.path, shutil, glob, re, datetime
+import os, os.path, shutil, glob, re, datetime, sys
 import dateutil.parser, dateutil.relativedelta, dateutil.tz
 import rtyaml
 
@@ -44,10 +42,10 @@ def backup_status(env):
 	# Get duplicity collection status and parse for a list of backups.
 	def parse_line(line):
 		keys = line.strip().split()
-		date = dateutil.parser.parse(keys[1])
+		date = dateutil.parser.parse(keys[1]).astimezone(dateutil.tz.tzlocal())
 		return {
 			"date": keys[1],
-			"date_str": date.strftime("%x %X"),
+			"date_str": date.strftime("%x %X") + " " + now.tzname(),
 			"date_delta": reldate(date, now, "the future?"),
 			"full": keys[0] == "full",
 			"size": 0, # collection-status doesn't give us the size
@@ -65,8 +63,8 @@ def backup_status(env):
 		trap=True)
 	if code != 0:
 		# Command failed. This is likely due to an improperly configured remote
-		# destination for the backups.
-		return { }
+		# destination for the backups or the last backup job terminated unexpectedly.
+		raise Exception("Something is wrong with the backup: " + collection_status)
 	for line in collection_status.split('\n'):
 		if line.startswith(" full") or line.startswith(" inc"):
 			backup = parse_line(line)
@@ -83,50 +81,66 @@ def backup_status(env):
 	# This is relied on by should_force_full() and the next step.
 	backups = sorted(backups.values(), key = lambda b : b["date"], reverse=True)
 
-	# Get the average size of incremental backups and the size of the
-	# most recent full backup.
+	# Get the average size of incremental backups, the size of the
+	# most recent full backup, and the date of the most recent
+	# backup and the most recent full backup.
 	incremental_count = 0
 	incremental_size = 0
+	first_date = None
 	first_full_size = None
+	first_full_date = None
 	for bak in backups:
+		if first_date is None:
+			first_date = dateutil.parser.parse(bak["date"])
 		if bak["full"]:
 			first_full_size = bak["size"]
+			first_full_date = dateutil.parser.parse(bak["date"])
 			break
 		incremental_count += 1
 		incremental_size += bak["size"]
 
-	# Predict how many more increments until the next full backup,
-	# and add to that the time we hold onto backups, to predict
-	# how long the most recent full backup+increments will be held
-	# onto. Round up since the backup occurs on the night following
-	# when the threshold is met.
+	# When will the most recent backup be deleted? It won't be deleted if the next
+	# backup is incremental, because the increments rely on all past increments.
+	# So first guess how many more incremental backups will occur until the next
+	# full backup. That full backup frees up this one to be deleted. But, the backup
+	# must also be at least min_age_in_days old too.
 	deleted_in = None
 	if incremental_count > 0 and first_full_size is not None:
-		deleted_in = "approx. %d days" % round(config["min_age_in_days"] + (.5 * first_full_size - incremental_size) / (incremental_size/incremental_count) + .5)
+		# How many days until the next incremental backup? First, the part of
+		# the algorithm based on increment sizes:
+		est_days_to_next_full = (.5 * first_full_size - incremental_size) / (incremental_size/incremental_count)
+		est_time_of_next_full = first_date + datetime.timedelta(days=est_days_to_next_full)
 
-	# When will a backup be deleted?
+		# ...And then the part of the algorithm based on full backup age:
+		est_time_of_next_full = min(est_time_of_next_full, first_full_date + datetime.timedelta(days=config["min_age_in_days"]*10+1))
+
+		# It still can't be deleted until it's old enough.
+		est_deleted_on = max(est_time_of_next_full, first_date + datetime.timedelta(days=config["min_age_in_days"]))
+
+		deleted_in = "approx. %d days" % round((est_deleted_on-now).total_seconds()/60/60/24 + .5)
+
+	# When will a backup be deleted? Set the deleted_in field of each backup.
 	saw_full = False
-	days_ago = now - datetime.timedelta(days=config["min_age_in_days"])
 	for bak in backups:
 		if deleted_in:
-			# Subsequent backups are deleted when the most recent increment
-			# in the chain would be deleted.
+			# The most recent increment in a chain and all of the previous backups
+			# it relies on are deleted at the same time.
 			bak["deleted_in"] = deleted_in
 		if bak["full"]:
-			# Reset when we get to a full backup. A new chain start next.
+			# Reset when we get to a full backup. A new chain start *next*.
 			saw_full = True
 			deleted_in = None
 		elif saw_full and not deleted_in:
-			# Mark deleted_in only on the first increment after a full backup.
-			deleted_in = reldate(days_ago, dateutil.parser.parse(bak["date"]), "on next daily backup")
+			# We're now on backups prior to the most recent full backup. These are
+			# free to be deleted as soon as they are min_age_in_days old.
+			deleted_in = reldate(now, dateutil.parser.parse(bak["date"]) + datetime.timedelta(days=config["min_age_in_days"]), "on next daily backup")
 			bak["deleted_in"] = deleted_in
 
 	return {
-		"tz": now.tzname(),
 		"backups": backups,
 	}
 
-def should_force_full(env):
+def should_force_full(config, env):
 	# Force a full backup when the total size of the increments
 	# since the last full backup is greater than half the size
 	# of that full backup.
@@ -138,8 +152,14 @@ def should_force_full(env):
 			inc_size += bak["size"]
 		else:
 			# ...until we reach the most recent full backup.
-			# Return if we should to a full backup.
-			return inc_size > .5*bak["size"]
+			# Return if we should to a full backup, which is based
+			# on the size of the increments relative to the full
+			# backup, as well as the age of the full backup.
+			if inc_size > .5*bak["size"]:
+				return True
+			if dateutil.parser.parse(bak["date"]) + datetime.timedelta(days=config["min_age_in_days"]*10+1) < datetime.datetime.now(dateutil.tz.tzlocal()):
+				return True
+			return False
 	else:
 		# If we got here there are no (full) backups, so make one.
 		# (I love for/else blocks. Here it's just to show off.)
@@ -217,12 +237,26 @@ def perform_backup(full_backup):
 	# will fail. Otherwise do a full backup when the size of
 	# the increments since the most recent full backup are
 	# large.
-	full_backup = full_backup or should_force_full(env)
+	try:
+		full_backup = full_backup or should_force_full(config, env)
+	except Exception as e:
+		# This was the first call to duplicity, and there might
+		# be an error already.
+		print(e)
+		sys.exit(1)
 
 	# Stop services.
-	shell('check_call', ["/usr/sbin/service", "php5-fpm", "stop"])
-	shell('check_call', ["/usr/sbin/service", "postfix", "stop"])
-	shell('check_call', ["/usr/sbin/service", "dovecot", "stop"])
+	def service_command(service, command, quit=None):
+		# Execute silently, but if there is an error then display the output & exit.
+		code, ret = shell('check_output', ["/usr/sbin/service", service, command], capture_stderr=True, trap=True)
+		if code != 0:
+			print(ret)
+			if quit:
+				sys.exit(code)
+
+	service_command("php5-fpm", "stop", quit=True)
+	service_command("postfix", "stop", quit=True)
+	service_command("dovecot", "stop", quit=True)
 
 	# Run a backup of STORAGE_ROOT (but excluding the backups themselves!).
 	# --allow-source-mismatch is needed in case the box's hostname is changed
@@ -231,6 +265,7 @@ def perform_backup(full_backup):
 		shell('check_call', [
 			"/usr/bin/duplicity",
 			"full" if full_backup else "incr",
+			"--verbosity", "warning", "--no-print-statistics",
 			"--archive-dir", backup_cache_dir,
 			"--exclude", backup_root,
 			"--volsize", "250",
@@ -242,9 +277,9 @@ def perform_backup(full_backup):
 			get_env(env))
 	finally:
 		# Start services again.
-		shell('check_call', ["/usr/sbin/service", "dovecot", "start"])
-		shell('check_call', ["/usr/sbin/service", "postfix", "start"])
-		shell('check_call', ["/usr/sbin/service", "php5-fpm", "start"])
+		service_command("dovecot", "start", quit=False)
+		service_command("postfix", "start", quit=False)
+		service_command("php5-fpm", "start", quit=False)
 
 	# Once the migrated backup is included in a new backup, it can be deleted.
 	if os.path.isdir(migrated_unencrypted_backup_dir):
@@ -256,6 +291,7 @@ def perform_backup(full_backup):
 		"/usr/bin/duplicity",
 		"remove-older-than",
 		"%dD" % config["min_age_in_days"],
+		"--verbosity", "error",
 		"--archive-dir", backup_cache_dir,
 		"--force",
 		config["target"]
@@ -270,6 +306,7 @@ def perform_backup(full_backup):
 	shell('check_call', [
 		"/usr/bin/duplicity",
 		"cleanup",
+		"--verbosity", "error",
 		"--archive-dir", backup_cache_dir,
 		"--force",
 		config["target"]

management/daemon.py

@@ -15,7 +15,7 @@ from mailconfig import get_mail_aliases, get_mail_aliases_ex, get_mail_domains,
 # live across http requests so we don't baloon the system with
 # processes.
 import multiprocessing.pool
-pool = multiprocessing.pool.Pool(processes=10)
+pool = multiprocessing.pool.Pool(processes=5)
 
 env = utils.load_environment()
 
@@ -327,6 +327,33 @@ def dns_get_dump():
 
 # SSL
 
+@app.route('/ssl/status')
+@authorized_personnel_only
+def ssl_get_status():
+	from ssl_certificates import get_certificates_to_provision
+	from web_update import get_web_domains_info, get_web_domains
+
+	# What domains can we provision certificates for? What unexpected problems do we have?
+	provision, cant_provision = get_certificates_to_provision(env, show_extended_problems=False)
+	
+	# What's the current status of TLS certificates on all of the domain?
+	domains_status = get_web_domains_info(env)
+	domains_status = [{ "domain": d["domain"], "status": d["ssl_certificate"][0], "text": d["ssl_certificate"][1] } for d in domains_status ]
+
+	# Warn the user about domain names not hosted here because of other settings.
+	for domain in set(get_web_domains(env, exclude_dns_elsewhere=False)) - set(get_web_domains(env)):
+		domains_status.append({
+			"domain": domain,
+			"status": "not-applicable",
+			"text": "The domain's website is hosted elsewhere.",
+		})
+
+	return json_response({
+		"can_provision": utils.sort_domains(provision, env),
+		"cant_provision": [{ "domain": domain, "problem": cant_provision[domain] } for domain in utils.sort_domains(cant_provision, env) ],
+		"status": domains_status,
+	})
+
 @app.route('/ssl/csr/<domain>', methods=['POST'])
 @authorized_personnel_only
 def ssl_get_csr(domain):
@@ -346,6 +373,17 @@ def ssl_install_cert():
 		return "Invalid domain name."
 	return install_cert(domain, ssl_cert, ssl_chain, env)
 
+@app.route('/ssl/provision', methods=['POST'])
+@authorized_personnel_only
+def ssl_provision_certs():
+	from ssl_certificates import provision_certificates
+	agree_to_tos_url = request.form.get('agree_to_tos_url')
+	status = provision_certificates(env,
+		agree_to_tos_url=agree_to_tos_url,
+		jsonable=True)
+	return json_response(status)
+
+
 # WEB
 
 @app.route('/web/domains')
@@ -422,7 +460,10 @@ def do_updates():
 @authorized_personnel_only
 def backup_status():
 	from backup import backup_status
-	return json_response(backup_status(env))
+	try:
+		return json_response(backup_status(env))
+	except Exception as e:
+		return json_response({ "error": str(e) })
 
 @app.route('/system/backup/config', methods=["GET"])
 @authorized_personnel_only

management/daily_tasks.sh

@@ -1,8 +1,19 @@
 #!/bin/bash
 # This script is run daily (at 3am each night).
 
+# Set character encoding flags to ensure that any non-ASCII
+# characters don't cause problems. See setup/start.sh and
+# the management daemon startup script.
+export LANGUAGE=en_US.UTF-8
+export LC_ALL=en_US.UTF-8
+export LANG=en_US.UTF-8
+export LC_TYPE=en_US.UTF-8
+
 # Take a backup.
-management/backup.py
+management/backup.py | management/email_administrator.py "Backup Status"
+
+# Provision any new certificates for new domains or domains with expiring certificates.
+management/ssl_certificates.py --headless | management/email_administrator.py "Error Provisioning TLS Certificate"
 
 # Run status checks and email the administrator if anything changed.
-management/status_checks.py --show-changes --smtp
+management/status_checks.py --show-changes | management/email_administrator.py "Status Checks Change Notice"

management/dns_update.py

@@ -91,7 +91,7 @@ def do_dns_update(env, force=False):
 		shell('check_call', ["/usr/sbin/service", "nsd", "restart"])
 
 	# Write the OpenDKIM configuration tables for all of the domains.
-	if write_opendkim_tables([domain for domain, zonefile in zonefiles], env):
+	if write_opendkim_tables(get_mail_domains(env), env):
 		# Settings changed. Kick opendkim.
 		shell('check_call', ["/usr/sbin/service", "opendkim", "restart"])
 		if len(updated_domains) == 0:
@@ -502,11 +502,12 @@ zone:
 
 def dnssec_choose_algo(domain, env):
 	if '.' in domain and domain.rsplit('.')[-1] in \
-		("email", "guide", "fund"):
+		("email", "guide", "fund", "be"):
 		# At GoDaddy, RSASHA256 is the only algorithm supported
 		# for .email and .guide.
 		# A variety of algorithms are supported for .fund. This
 		# is preferred.
+		# Gandi tells me that .be does not support RSASHA1-NSEC3-SHA1
 		return "RSASHA256"
 
 	# For any domain we were able to sign before, don't change the algorithm

management/email_administrator.py

@@ -0,0 +1,42 @@
+#!/usr/bin/python3
+
+# Reads in STDIN. If the stream is not empty, mail it to the system administrator.
+
+import sys
+
+import smtplib
+from email.message import Message
+
+from utils import load_environment
+
+# Load system environment info.
+env = load_environment()
+
+# Process command line args.
+subject = sys.argv[1]
+
+# Administrator's email address.
+admin_addr = "administrator@" + env['PRIMARY_HOSTNAME']
+
+# Read in STDIN.
+content = sys.stdin.read().strip()
+
+# If there's nothing coming in, just exit.
+if content == "":
+	sys.exit(0)
+
+# create MIME message
+msg = Message()
+msg['From'] = "\"%s\" <%s>" % (env['PRIMARY_HOSTNAME'], admin_addr)
+msg['To'] = admin_addr
+msg['Subject'] = "[%s] %s" % (env['PRIMARY_HOSTNAME'], subject)
+msg.set_payload(content, "UTF-8")
+
+# send
+smtpclient = smtplib.SMTP('localhost', 25)
+smtpclient.ehlo()
+smtpclient.sendmail(
+        admin_addr, # MAIL FROM
+        admin_addr, # RCPT TO
+        msg.as_string())
+smtpclient.quit()

management/ssl_certificates.py

@@ -1,8 +1,13 @@
+#!/usr/bin/python3
 # Utilities for installing and selecting SSL certificates.
 
 import os, os.path, re, shutil
 
-from utils import shell, safe_domain_name
+from utils import shell, safe_domain_name, sort_domains
+
+import idna
+
+# SELECTING SSL CERTIFICATES FOR USE IN WEB
 
 def get_ssl_certificates(env):
 	# Scan all of the installed SSL certificates and map every domain
@@ -17,6 +22,8 @@ def get_ssl_certificates(env):
 
 	# List all of the files in the SSL directory and one level deep.
 	def get_file_list():
+		if not os.path.exists(ssl_root):
+			return
 		for fn in os.listdir(ssl_root):
 			fn = os.path.join(ssl_root, fn)
 			if os.path.isfile(fn):
@@ -82,10 +89,27 @@ def get_ssl_certificates(env):
 			# prefer one that is not self-signed
 			cert.issuer != cert.subject,
 
+			###########################################################
+			# The above lines ensure that valid certificates are chosen
+			# over invalid certificates. The lines below choose between
+			# multiple valid certificates available for this domain.
+			###########################################################
+
 			# prefer one with the expiration furthest into the future so
 			# that we can easily rotate to new certs as we get them
 			cert.not_valid_after,
 
+			###########################################################
+			# We always choose the certificate that is good for the
+			# longest period of time. This is important for how we
+			# provision certificates for Let's Encrypt. To ensure that
+			# we don't re-provision every night, we have to ensure that
+			# if we choose to provison a certificate that it will
+			# *actually* be used so the provisioning logic knows it
+			# doesn't still need to provision a certificate for the
+			# domain.
+			###########################################################
+
 			# in case a certificate is installed in multiple paths,
 			# prefer the... lexicographically last one?
 			cert._filename,
@@ -96,46 +120,426 @@ def get_ssl_certificates(env):
 			"private-key": cert._private_key._filename,
 			"certificate": cert._filename,
 			"primary-domain": cert._primary_domain,
+			"certificate_object": cert,
 			}
 
 	return ret
 
-def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False):
-	# Get the default paths.
+def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False, raw=False):
+	# Get the system certificate info.
 	ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
 	ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
+	system_certificate = {
+		"private-key": ssl_private_key,
+		"certificate": ssl_certificate,
+		"primary-domain": env['PRIMARY_HOSTNAME'],
+		"certificate_object": load_pem(load_cert_chain(ssl_certificate)[0]),
+	}
 
 	if domain == env['PRIMARY_HOSTNAME']:
 		# The primary domain must use the server certificate because
 		# it is hard-coded in some service configuration files.
-		return ssl_private_key, ssl_certificate, None
+		return system_certificate
 
 	wildcard_domain = re.sub("^[^\.]+", "*", domain)
-
 	if domain in ssl_certificates:
-		cert_info = ssl_certificates[domain]
-		cert_type = "multi-domain"
+		return ssl_certificates[domain]
 	elif wildcard_domain in ssl_certificates:
-		cert_info = ssl_certificates[wildcard_domain]
-		cert_type = "wildcard"
+		return ssl_certificates[wildcard_domain]
 	elif not allow_missing_cert:
-		# No certificate is available for this domain! Return default files.
-		ssl_via = "Using certificate for %s." % env['PRIMARY_HOSTNAME']
-		return ssl_private_key, ssl_certificate, ssl_via
+		# No valid certificate is available for this domain! Return default files.
+		return system_certificate
 	else:
-		# No certificate is available - and warn appropriately.
+		# No valid certificate is available for this domain.
 		return None
 
-	# 'via' is a hint to the user about which certificate is in use for the domain
-	if cert_info['certificate'] == os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'):
-		# Using the server certificate.
-		via = "Using same %s certificate as for %s." % (cert_type, env['PRIMARY_HOSTNAME'])
-	elif cert_info['primary-domain'] != domain and cert_info['primary-domain'] in ssl_certificates and cert_info == ssl_certificates[cert_info['primary-domain']]:
-		via = "Using same %s certificate as for %s." % (cert_type, cert_info['primary-domain'])
-	else:
-		via = None # don't show a hint - show expiration info instead
 
-	return cert_info['private-key'], cert_info['certificate'], via
+# PROVISIONING CERTIFICATES FROM LETSENCRYPT
+
+def get_certificates_to_provision(env, show_extended_problems=True, force_domains=None):
+	# Get a set of domain names that we should now provision certificates
+	# for. Provision if a domain name has no valid certificate or if any
+	# certificate is expiring in 14 days. If provisioning anything, also
+	# provision certificates expiring within 30 days. The period between
+	# 14 and 30 days allows us to consolidate domains into multi-domain
+	# certificates for domains expiring around the same time.
+
+	from web_update import get_web_domains
+
+	import datetime
+	now = datetime.datetime.utcnow()
+
+	# Get domains with missing & expiring certificates.
+	certs = get_ssl_certificates(env)
+	domains = set()
+	domains_if_any = set()
+	problems = { }
+	for domain in get_web_domains(env):
+		# If the user really wants a cert for certain domains, include it.
+		if force_domains:
+			if force_domains == "ALL" or (isinstance(force_domains, list) and domain in force_domains):
+				domains.add(domain)
+			continue
+
+		# Include this domain if its certificate is missing, self-signed, or expiring soon.
+		try:
+			cert = get_domain_ssl_files(domain, certs, env, allow_missing_cert=True)
+		except FileNotFoundError as e:
+			# system certificate is not present
+			problems[domain] = "Error: " + str(e)
+			continue
+		if cert is None:
+			# No valid certificate available.
+			domains.add(domain)
+		else:
+			cert = cert["certificate_object"]
+			if cert.issuer == cert.subject:
+				# This is self-signed. Get a real one.
+				domains.add(domain)
+			
+			# Valid certificate today, but is it expiring soon?
+			elif cert.not_valid_after-now < datetime.timedelta(days=14):
+				domains.add(domain)
+			elif cert.not_valid_after-now < datetime.timedelta(days=30):
+				domains_if_any.add(domain)
+
+			# It's valid. Should we report its validness?
+			elif show_extended_problems:
+				problems[domain] = "The certificate is valid for at least another 30 days --- no need to replace."
+
+	# Warn the user about domains hosted elsewhere.
+	if not force_domains and show_extended_problems:
+		for domain in set(get_web_domains(env, exclude_dns_elsewhere=False)) - set(get_web_domains(env)):
+			problems[domain] = "The domain's DNS is pointed elsewhere, so there is no point to installing a TLS certificate here and we could not automatically provision one anyway because provisioning requires access to the website (which isn't here)."
+
+	# Filter out domains that we can't provision a certificate for.
+	def can_provision_for_domain(domain):
+		# Let's Encrypt doesn't yet support IDNA domains.
+		# We store domains in IDNA (ASCII). To see if this domain is IDNA,
+		# we'll see if its IDNA-decoded form is different.
+		if idna.decode(domain.encode("ascii")) != domain:
+			problems[domain] = "Let's Encrypt does not yet support provisioning certificates for internationalized domains."
+			return False
+
+		# Does the domain resolve to this machine in public DNS? If not,
+		# we can't do domain control validation. For IPv6 is configured,
+		# make sure both IPv4 and IPv6 are correct because we don't know
+		# how Let's Encrypt will connect.
+		import dns.resolver
+		for rtype, value in [("A", env["PUBLIC_IP"]), ("AAAA", env.get("PUBLIC_IPV6"))]:
+			if not value: continue # IPv6 is not configured
+			try:
+				# Must make the qname absolute to prevent a fall-back lookup with a
+				# search domain appended, by adding a period to the end.
+				response = dns.resolver.query(domain + ".", rtype)
+			except (dns.resolver.NoNameservers, dns.resolver.NXDOMAIN, dns.resolver.NoAnswer) as e:
+				problems[domain] = "DNS isn't configured properly for this domain: DNS resolution failed (%s: %s)." % (rtype, str(e) or repr(e)) # NoAnswer's str is empty
+				return False
+			except Exception as e:
+				problems[domain] = "DNS isn't configured properly for this domain: DNS lookup had an error: %s." % str(e)
+				return False
+			if len(response) != 1 or str(response[0]) != value:
+				problems[domain] = "Domain control validation cannot be performed for this domain because DNS points the domain to another machine (%s %s)." % (rtype, ", ".join(str(r) for r in response))
+				return False
+
+		return True
+
+	domains = set(filter(can_provision_for_domain, domains))
+
+	# If there are any domains we definitely will provision for, add in
+	# additional domains to do at this time.
+	if len(domains) > 0:
+		domains |= set(filter(can_provision_for_domain, domains_if_any))
+
+	return (domains, problems)
+
+def provision_certificates(env, agree_to_tos_url=None, logger=None, show_extended_problems=True, force_domains=None, jsonable=False):
+	import requests.exceptions
+	import acme.messages
+
+	from free_tls_certificates import client
+
+	# What domains should we provision certificates for? And what
+	# errors prevent provisioning for other domains.
+	domains, problems = get_certificates_to_provision(env, force_domains=force_domains, show_extended_problems=show_extended_problems)
+
+	# Exit fast if there is nothing to do.
+	if len(domains) == 0:
+		return {
+			"requests": [],
+			"problems": problems,
+		}
+
+	# Break into groups of up to 100 certificates at a time, which is Let's Encrypt's
+	# limit for a single certificate. We'll sort to put related domains together.
+	domains = sort_domains(domains, env)
+	certs = []
+	while len(domains) > 0:
+		certs.append( domains[0:100] )
+		domains = domains[100:]
+
+	# Prepare to provision.
+
+	# Where should we put our Let's Encrypt account info and state cache.
+	account_path = os.path.join(env['STORAGE_ROOT'], 'ssl/lets_encrypt')
+	if not os.path.exists(account_path):
+		os.mkdir(account_path)
+
+	# Where should we put ACME challenge files. This is mapped to /.well-known/acme_challenge
+	# by the nginx configuration.
+	challenges_path = os.path.join(account_path, 'acme_challenges')
+	if not os.path.exists(challenges_path):
+		os.mkdir(challenges_path)
+
+	# Read in the private key that we use for all TLS certificates. We'll need that
+	# to generate a CSR (done by free_tls_certificates).
+	with open(os.path.join(env['STORAGE_ROOT'], 'ssl/ssl_private_key.pem'), 'rb') as f:
+		private_key = f.read()
+
+	# Provision certificates.
+
+	ret = []
+	for domain_list in certs:
+		# For return.
+		ret_item = {
+			"domains": domain_list,
+			"log": [],
+		}
+		ret.append(ret_item)
+
+		# Logging for free_tls_certificates.
+		def my_logger(message):
+			if logger: logger(message)
+			ret_item["log"].append(message)
+
+		# Attempt to provision a certificate.
+		try:
+			try:
+				cert = client.issue_certificate(
+					domain_list,
+					account_path,
+					agree_to_tos_url=agree_to_tos_url,
+					private_key=private_key,
+					logger=my_logger)
+
+			except client.NeedToTakeAction as e:
+				# Write out the ACME challenge files.
+				for action in e.actions:
+					if isinstance(action, client.NeedToInstallFile):
+						fn = os.path.join(challenges_path, action.file_name)
+						with open(fn, 'w') as f:
+							f.write(action.contents)
+					else:
+						raise ValueError(str(action))
+
+				# Try to provision now that the challenge files are installed.
+
+				cert = client.issue_certificate(
+					domain_list,
+					account_path,
+					private_key=private_key,
+					logger=my_logger)
+
+		except client.NeedToAgreeToTOS as e:
+			# The user must agree to the Let's Encrypt terms of service agreement
+			# before any further action can be taken.
+			ret_item.update({
+				"result": "agree-to-tos",
+				"url": e.url,
+			})
+
+		except client.WaitABit as e:
+			# We need to hold on for a bit before querying again to see if we can
+			# acquire a provisioned certificate.
+			import time, datetime
+			ret_item.update({
+				"result": "wait",
+				"until": e.until_when if not jsonable else e.until_when.isoformat(),
+				"seconds": (e.until_when - datetime.datetime.now()).total_seconds()
+			})
+
+		except client.AccountDataIsCorrupt as e:
+			# This is an extremely rare condition.
+			ret_item.update({
+				"result": "error",
+				"message": "Something unexpected went wrong. It looks like your local Let's Encrypt account data is corrupted. There was a problem with the file " + e.account_file_path + ".",
+			})
+
+		except (client.InvalidDomainName, client.NeedToTakeAction, client.ChallengeFailed, acme.messages.Error, requests.exceptions.RequestException) as e:
+			ret_item.update({
+				"result": "error",
+				"message": "Something unexpected went wrong: " + str(e),
+			})
+
+		else:
+			# A certificate was issued.
+
+			install_status = install_cert(domain_list[0], cert['cert'].decode("ascii"), b"\n".join(cert['chain']).decode("ascii"), env, raw=True)
+
+			# str indicates the certificate was not installed.
+			if isinstance(install_status, str):
+				ret_item.update({
+					"result": "error",
+					"message": "Something unexpected was wrong with the provisioned certificate: " + install_status,
+				})
+			else:
+				# A list indicates success and what happened next.
+				ret_item["log"].extend(install_status)
+				ret_item.update({
+					"result": "installed",
+				})
+
+	# Return what happened with each certificate request.
+	return {
+		"requests": ret,
+		"problems": problems,
+	}
+
+def provision_certificates_cmdline():
+	import sys
+	from utils import load_environment, exclusive_process
+
+	exclusive_process("update_tls_certificates")
+	env = load_environment()
+
+	verbose = False
+	headless = False
+	force_domains = None
+	show_extended_problems = True
+	
+	args = list(sys.argv)
+	args.pop(0) # program name
+	if args and args[0] == "-v":
+		verbose = True
+		args.pop(0)
+	if args and args[0] == "q":
+		show_extended_problems = False
+		args.pop(0)
+	if args and args[0] == "--headless":
+		headless = True
+		args.pop(0)
+	if args and args[0] == "--force":
+		force_domains = "ALL"
+		args.pop(0)
+	else:
+		force_domains = args
+
+	agree_to_tos_url = None
+	while True:
+		# Run the provisioning script. This installs certificates. If there are
+		# a very large number of domains on this box, it issues separate
+		# certificates for groups of domains. We have to check the result for
+		# each group.
+		def my_logger(message):
+			if verbose:
+				print(">", message)
+		status = provision_certificates(env, agree_to_tos_url=agree_to_tos_url, logger=my_logger, force_domains=force_domains, show_extended_problems=show_extended_problems)
+		agree_to_tos_url = None # reset to prevent infinite looping
+
+		if not status["requests"]:
+			# No domains need certificates.
+			if not headless or verbose:
+				if len(status["problems"]) == 0:
+					print("No domains hosted on this box need a new TLS certificate at this time.")
+				elif len(status["problems"]) > 0:
+					print("No TLS certificates could be provisoned at this time:")
+					print()
+					for domain in sort_domains(status["problems"], env):
+						print("%s: %s" % (domain, status["problems"][domain]))
+
+			sys.exit(0)
+
+		# What happened?
+		wait_until = None
+		wait_domains = []
+		for request in status["requests"]:
+			if request["result"] == "agree-to-tos":
+				# We may have asked already in a previous iteration.
+				if agree_to_tos_url is not None:
+					continue
+
+				# Can't ask the user a question in this mode. Warn the user that something
+				# needs to be done.
+				if headless:
+					print(", ".join(request["domains"]) + " need a new or renewed TLS certificate.")
+					print()
+					print("This box can't do that automatically for you until you agree to Let's Encrypt's")
+					print("Terms of Service agreement. Use the Mail-in-a-Box control panel to provision")
+					print("certificates for these domains.")
+					sys.exit(1)
+
+				print("""
+I'm going to provision a TLS certificate (formerly called a SSL certificate)
+for you from Let's Encrypt (letsencrypt.org).
+
+TLS certificates are cryptographic keys that ensure communication between
+you and this box are secure when getting and sending mail and visiting
+websites hosted on this box. Let's Encrypt is a free provider of TLS
+certificates.
+
+Please open this document in your web browser:
+
+%s
+
+It is Let's Encrypt's terms of service agreement. If you agree, I can
+provision that TLS certificate. If you don't agree, you will have an
+opportunity to install your own TLS certificate from the Mail-in-a-Box
+control panel.
+
+Do you agree to the agreement? Type Y or N and press <ENTER>: """
+				 % request["url"], end='', flush=True)
+			
+				if sys.stdin.readline().strip().upper() != "Y":
+					print("\nYou didn't agree. Quitting.")
+					sys.exit(1)
+
+				# Okay, indicate agreement on next iteration.
+				agree_to_tos_url = request["url"]
+
+			if request["result"] == "wait":
+				# Must wait. We'll record until when. The wait occurs below.
+				if wait_until is None:
+					wait_until = request["until"]
+				else:
+					wait_until = max(wait_until, request["until"])
+				wait_domains += request["domains"]
+
+			if request["result"] == "error":
+				print(", ".join(request["domains"]) + ":")
+				print(request["message"])
+
+			if request["result"] == "installed":
+				print("A TLS certificate was successfully installed for " + ", ".join(request["domains"]) + ".")
+
+		if wait_until:
+			# Wait, then loop.
+			import time, datetime
+			print()
+			print("A TLS certificate was requested for: " + ", ".join(wait_domains) + ".")
+			first = True
+			while wait_until > datetime.datetime.now():
+				if not headless or first:
+					print ("We have to wait", int(round((wait_until - datetime.datetime.now()).total_seconds())), "seconds for the certificate to be issued...")
+				time.sleep(10)
+				first = False
+
+			continue # Loop!
+
+		if agree_to_tos_url:
+			# The user agrees to the TOS. Loop to try again by agreeing.
+			continue # Loop!
+
+		# Unless we were instructed to wait, or we just agreed to the TOS,
+		# we're done for now.
+		break
+
+	# And finally show the domains with problems.
+	if len(status["problems"]) > 0:
+		print("TLS certificates could not be provisoned for:")
+		for domain in sort_domains(status["problems"], env):
+			print("%s: %s" % (domain, status["problems"][domain]))
+
+# INSTALLING A NEW CERTIFICATE FROM THE CONTROL PANEL
 
 def create_csr(domain, ssl_key, country_code, env):
 	return shell("check_output", [
@@ -144,7 +548,7 @@ def create_csr(domain, ssl_key, country_code, env):
                 "-sha256",
                 "-subj", "/C=%s/ST=/L=/O=/CN=%s" % (country_code, domain)])
 
-def install_cert(domain, ssl_cert, ssl_chain, env):
+def install_cert(domain, ssl_cert, ssl_chain, env, raw=False):
 	# Write the combined cert+chain to a temporary path and validate that it is OK.
 	# The certificate always goes above the chain.
 	import tempfile
@@ -203,8 +607,10 @@ def install_cert(domain, ssl_cert, ssl_chain, env):
 	# Update the web configuration so nginx picks up the new certificate file.
 	from web_update import do_web_update
 	ret.append( do_web_update(env) )
+	if raw: return ret
 	return "\n".join(ret)
 
+# VALIDATION OF CERTIFICATES
 
 def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring_soon=True, rounded_time=False, just_check_domain=False):
 	# Check that the ssl_certificate & ssl_private_key files are good
@@ -296,7 +702,7 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 			return ("The certificate is missing an intermediate chain or the intermediate chain is incorrect or incomplete. (%s)" % verifyoutput, None)
 
 		# There is some unknown problem. Return the `openssl verify` raw output.
-		return ("There is a problem with the SSL certificate.", verifyoutput.strip())
+		return ("There is a problem with the certificate.", verifyoutput.strip())
 
 	else:
 		# `openssl verify` returned a zero exit status so the cert is currently
@@ -305,16 +711,16 @@ def check_certificate(domain, ssl_certificate, ssl_private_key, warn_if_expiring
 		# But is it expiring soon?
 		cert_expiration_date = cert.not_valid_after
 		ndays = (cert_expiration_date-now).days
-		if not rounded_time or ndays < 7:
+		if not rounded_time or ndays <= 10:
+			# Yikes better renew soon!
 			expiry_info = "The certificate expires in %d days on %s." % (ndays, cert_expiration_date.strftime("%x"))
-		elif ndays <= 14:
-			expiry_info = "The certificate expires in less than two weeks, on %s." % cert_expiration_date.strftime("%x")
-		elif ndays <= 31:
-			expiry_info = "The certificate expires in less than a month, on %s." % cert_expiration_date.strftime("%x")
 		else:
+			# We'll renew it with Lets Encrypt.
 			expiry_info = "The certificate expires on %s." % cert_expiration_date.strftime("%x")
 
-		if ndays <= 31 and warn_if_expiring_soon:
+		if ndays <= 10 and warn_if_expiring_soon:
+			# Warn on day 10 to give 4 days for us to automatically renew the
+			# certificate, which occurs on day 14.
 			return ("The certificate is expiring soon: " + expiry_info, None)
 
 		# Return the special OK code.
@@ -381,3 +787,7 @@ def get_certificate_domains(cert):
 		pass
 
 	return names, cn
+
+if __name__  == "__main__":
+	# Provision certificates.
+	provision_certificates_cmdline()

management/status_checks.py

@@ -1,7 +1,7 @@
 #!/usr/bin/python3
 #
 # Checks that the upstream DNS has been set correctly and that
-# SSL certificates have been signed, etc., and if not tells the user
+# TLS certificates have been signed, etc., and if not tells the user
 # what to do next.
 
 import sys, os, os.path, re, subprocess, datetime, multiprocessing.pool
@@ -9,6 +9,7 @@ import sys, os, os.path, re, subprocess, datetime, multiprocessing.pool
 import dns.reversename, dns.resolver
 import dateutil.parser, dateutil.tz
 import idna
+import psutil
 
 from dns_update import get_dns_zones, build_tlsa_record, get_custom_dns_config, get_secondary_dns, get_custom_dns_record
 from web_update import get_web_domains, get_domains_with_a_records
@@ -71,7 +72,6 @@ def run_services_checks(env, output, pool):
 		{ "name": "OpenDKIM", "port": 8891, "public": False, },
 		{ "name": "OpenDMARC", "port": 8893, "public": False, },
 		{ "name": "Memcached", "port": 11211, "public": False, },
-		{ "name": "Sieve (dovecot)", "port": 4190, "public": False, },
 		{ "name": "Mail-in-a-Box Management Daemon", "port": 10222, "public": False, },
 
 		{ "name": "SSH Login (ssh)", "port": get_ssh_port(), "public": True, },
@@ -80,6 +80,7 @@ def run_services_checks(env, output, pool):
 		{ "name": "Outgoing Mail (SMTP 587/postfix)", "port": 587, "public": True, },
 		#{ "name": "Postfix/master", "port": 10587, "public": True, },
 		{ "name": "IMAPS (dovecot)", "port": 993, "public": True, },
+		{ "name": "Mail Filters (Sieve/dovecot)", "port": 4190, "public": True, },
 		{ "name": "HTTP Web (nginx)", "port": 80, "public": True, },
 		{ "name": "HTTPS Web (nginx)", "port": 443, "public": True, },
 	]
@@ -166,6 +167,7 @@ def run_system_checks(rounded_values, env, output):
 	check_miab_version(env, output)
 	check_system_aliases(env, output)
 	check_free_disk_space(rounded_values, env, output)
+	check_free_memory(rounded_values, env, output)
 
 def check_ssh_password(env, output):
 	# Check that SSH login with password is disabled. The openssh-server
@@ -216,6 +218,20 @@ def check_free_disk_space(rounded_values, env, output):
 	else:
 		output.print_error(disk_msg)
 
+def check_free_memory(rounded_values, env, output):
+	# Check free memory.
+	percent_free = 100 - psutil.virtual_memory().percent
+	memory_msg = "System memory is %s%% free." % str(round(percent_free))
+	if percent_free >= 20:
+		if rounded_values: memory_msg = "System free memory is at least 20%."
+		output.print_ok(memory_msg)
+	elif percent_free >= 10:
+		if rounded_values: memory_msg = "System free memory is below 20%."
+		output.print_warning(memory_msg)
+	else:
+		if rounded_values: memory_msg = "System free memory is below 10%."
+		output.print_error(memory_msg)
+
 def run_network_checks(env, output):
 	# Also see setup/network-checks.sh.
 
@@ -262,23 +278,24 @@ def run_domain_checks(rounded_time, env, output, pool):
 	# Get the list of domains that we don't serve web for because of a custom CNAME/A record.
 	domains_with_a_records = get_domains_with_a_records(env)
 
-	ssl_certificates = get_ssl_certificates(env)
-
 	# Serial version:
 	#for domain in sort_domains(domains_to_check, env):
 	#	run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains)
 
 	# Parallelize the checks across a worker pool.
-	args = ((domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains, domains_with_a_records, ssl_certificates)
+	args = ((domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains, domains_with_a_records)
 		for domain in domains_to_check)
 	ret = pool.starmap(run_domain_checks_on_domain, args, chunksize=1)
 	ret = dict(ret) # (domain, output) => { domain: output }
 	for domain in sort_domains(ret, env):
 		ret[domain].playback(output)
 
-def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains, domains_with_a_records, ssl_certificates):
+def run_domain_checks_on_domain(domain, rounded_time, env, dns_domains, dns_zonefiles, mail_domains, web_domains, domains_with_a_records):
 	output = BufferedOutput()
 
+	# we'd move this up, but this returns non-pickleable values
+	ssl_certificates = get_ssl_certificates(env)
+
 	# The domain is IDNA-encoded in the database, but for display use Unicode.
 	try:
 		domain_display = idna.decode(domain.encode('ascii'))
@@ -447,7 +464,7 @@ def check_dns_zone(domain, env, output, dns_zonefiles):
 			elif ip is None:
 				output.print_error("Secondary nameserver %s is not configured to resolve this domain." % ns)
 			else:
-				output.print_error("Secondary nameserver %s is not configured correctly. (It resolved this domain as %s. It should be %s.)" % (ns, ip, env['PUBLIC_IP']))
+				output.print_error("Secondary nameserver %s is not configured correctly. (It resolved this domain as %s. It should be %s.)" % (ns, ip, correct_ip))
 
 def check_dns_zone_suggestions(domain, env, output, dns_zonefiles, domains_with_a_records):
 	# Warn if a custom DNS record is preventing this or the automatic www redirect from
@@ -584,15 +601,23 @@ def check_web_domain(domain, rounded_time, ssl_certificates, env, output):
 	# for PRIMARY_HOSTNAME, for which it is required for mail specifically. For it and
 	# other domains, it is required to access its website.
 	if domain != env['PRIMARY_HOSTNAME']:
-		ip = query_dns(domain, "A")
-		if ip == env['PUBLIC_IP']:
-			output.print_ok("Domain resolves to this box's IP address. [%s ↦ %s]" % (domain, env['PUBLIC_IP']))
-		else:
-			output.print_error("""This domain should resolve to your box's IP address (%s) if you would like the box to serve
-				webmail or a website on this domain. The domain currently resolves to %s in public DNS. It may take several hours for
-				public DNS to update after a change. This problem may result from other issues listed here.""" % (env['PUBLIC_IP'], ip))
+		ok_values = []
+		for (rtype, expected) in (("A", env['PUBLIC_IP']), ("AAAA", env.get('PUBLIC_IPV6'))):
+			if not expected: continue # IPv6 is not configured
+			value = query_dns(domain, rtype)
+			if value == expected:
+				ok_values.append(value)
+			else:
+				output.print_error("""This domain should resolve to your box's IP address (%s %s) if you would like the box to serve
+					webmail or a website on this domain. The domain currently resolves to %s in public DNS. It may take several hours for
+					public DNS to update after a change. This problem may result from other issues listed here.""" % (rtype, expected, value))
+				return
 
-	# We need a SSL certificate for PRIMARY_HOSTNAME because that's where the
+		# If both A and AAAA are correct...
+		output.print_ok("Domain resolves to this box's IP address. [%s ↦ %s]" % (domain, '; '.join(ok_values)))
+
+
+	# We need a TLS certificate for PRIMARY_HOSTNAME because that's where the
 	# user will log in with IMAP or webmail. Any other domain we serve a
 	# website for also needs a signed certificate.
 	check_ssl_cert(domain, rounded_time, ssl_certificates, env, output)
@@ -634,56 +659,39 @@ def query_dns(qname, rtype, nxdomain='[Not Set]', at=None):
 	return "; ".join(sorted(str(r).rstrip('.') for r in response))
 
 def check_ssl_cert(domain, rounded_time, ssl_certificates, env, output):
-	# Check that SSL certificate is signed.
+	# Check that TLS certificate is signed.
 
 	# Skip the check if the A record is not pointed here.
 	if query_dns(domain, "A", None) not in (env['PUBLIC_IP'], None): return
 
-	# Where is the SSL stored?
-	x = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
-
-	if x is None:
-		output.print_warning("""No SSL certificate is installed for this domain. Visitors to a website on
+	# Where is the certificate file stored?
+	tls_cert = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
+	if tls_cert is None:
+		output.print_warning("""No TLS (SSL) certificate is installed for this domain. Visitors to a website on
 			this domain will get a security warning. If you are not serving a website on this domain, you do
-			not need to take any action. Use the SSL Certificates page in the control panel to install a
-			SSL certificate.""")
+			not need to take any action. Use the TLS Certificates page in the control panel to install a
+			TLS certificate.""")
 		return
 
-	ssl_key, ssl_certificate, ssl_via = x
-
 	# Check that the certificate is good.
 
-	cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key, rounded_time=rounded_time)
+	cert_status, cert_status_details = check_certificate(domain, tls_cert["certificate"], tls_cert["private-key"], rounded_time=rounded_time)
 
 	if cert_status == "OK":
 		# The certificate is ok. The details has expiry info.
-		output.print_ok("SSL certificate is signed & valid. %s %s" % (ssl_via if ssl_via else "", cert_status_details))
+		output.print_ok("TLS (SSL) certificate is signed & valid. " + cert_status_details)
 
 	elif cert_status == "SELF-SIGNED":
 		# Offer instructions for purchasing a signed certificate.
-
-		fingerprint = shell('check_output', [
-			"openssl",
-			"x509",
-			"-in", ssl_certificate,
-			"-noout",
-			"-fingerprint"
-			])
-		fingerprint = re.sub(".*Fingerprint=", "", fingerprint).strip()
-
 		if domain == env['PRIMARY_HOSTNAME']:
-			output.print_error("""The SSL certificate for this domain is currently self-signed. You will get a security
+			output.print_error("""The TLS (SSL) certificate for this domain is currently self-signed. You will get a security
 			warning when you check or send email and when visiting this domain in a web browser (for webmail or
-			static site hosting). Use the SSL Certificates page in the control panel to install a signed SSL certificate.
-			You may choose to leave the self-signed certificate in place and confirm the security exception, but check that
-			the certificate fingerprint matches the following:""")
-			output.print_line("")
-			output.print_line("   " + fingerprint, monospace=True)
+			static site hosting).""")
 		else:
-			output.print_error("""The SSL certificate for this domain is self-signed.""")
+			output.print_error("""The TLS (SSL) certificate for this domain is self-signed.""")
 
 	else:
-		output.print_error("The SSL certificate has a problem: " + cert_status)
+		output.print_error("The TLS (SSL) certificate has a problem: " + cert_status)
 		if cert_status_details:
 			output.print_line("")
 			output.print_line(cert_status_details)
@@ -732,10 +740,10 @@ def what_version_is_this(env):
 	return tag
 
 def get_latest_miab_version():
-	# This pings https://mailinabox.email/bootstrap.sh and extracts the tag named in
+	# This pings https://mailinabox.email/setup.sh and extracts the tag named in
 	# the script to determine the current product version.
 	import urllib.request
-	return re.search(b'TAG=(.*)', urllib.request.urlopen("https://mailinabox.email/bootstrap.sh?ping=1").read()).group(1).decode("utf8")
+	return re.search(b'TAG=(.*)', urllib.request.urlopen("https://mailinabox.email/setup.sh?ping=1").read()).group(1).decode("utf8")
 
 def check_miab_version(env, output):
 	config = load_settings(env)
@@ -743,23 +751,24 @@ def check_miab_version(env, output):
 	if config.get("privacy", True):
 		output.print_warning("Mail-in-a-Box version check disabled by privacy setting.")
 	else:
-		this_ver = what_version_is_this(env)
+		try:
+			this_ver = what_version_is_this(env)
+		except:
+			this_ver = "Unknown"
+
 		latest_ver = get_latest_miab_version()
+
 		if this_ver == latest_ver:
 			output.print_ok("Mail-in-a-Box is up to date. You are running version %s." % this_ver)
 		else:
 			output.print_error("A new version of Mail-in-a-Box is available. You are running version %s. The latest version is %s. For upgrade instructions, see https://mailinabox.email. "
 				% (this_ver, latest_ver))
 
-def run_and_output_changes(env, pool, send_via_email):
+def run_and_output_changes(env, pool):
 	import json
 	from difflib import SequenceMatcher
 
-	if not send_via_email:
-		out = ConsoleOutput()
-	else:
-		import io
-		out = FileOutput(io.StringIO(""), 70)
+	out = ConsoleOutput()
 
 	# Run status checks.
 	cur = BufferedOutput()
@@ -818,28 +827,6 @@ def run_and_output_changes(env, pool, send_via_email):
 				out.add_heading(category)
 				out.print_warning("This section was removed.")
 
-	if send_via_email:
-		# If there were changes, send off an email.
-		buf = out.buf.getvalue()
-		if len(buf) > 0:
-			# create MIME message
-			from email.message import Message
-			msg = Message()
-			msg['From'] = "\"%s\" <administrator@%s>" % (env['PRIMARY_HOSTNAME'], env['PRIMARY_HOSTNAME'])
-			msg['To'] = "administrator@%s" % env['PRIMARY_HOSTNAME']
-			msg['Subject'] = "[%s] Status Checks Change Notice" % env['PRIMARY_HOSTNAME']
-			msg.set_payload(buf, "UTF-8")
-
-			# send to administrator@
-			import smtplib
-			mailserver = smtplib.SMTP('localhost', 25)
-			mailserver.ehlo()
-			mailserver.sendmail(
-				"administrator@%s" % env['PRIMARY_HOSTNAME'], # MAIL FROM
-				"administrator@%s" % env['PRIMARY_HOSTNAME'], # RCPT TO
-				msg.as_string())
-			mailserver.quit()
-
 	# Store the current status checks output for next time.
 	os.makedirs(os.path.dirname(cache_fn), exist_ok=True)
 	with open(cache_fn, "w") as f:
@@ -870,7 +857,7 @@ class FileOutput:
 		words = re.split("(\s+)", message)
 		linelen = 0
 		for w in words:
-			if linelen + len(w) > self.width-1-len(first_line):
+			if self.width and (linelen + len(w) > self.width-1-len(first_line)):
 				print(file=self.buf)
 				print("   ", end="", file=self.buf)
 				linelen = 0
@@ -886,10 +873,22 @@ class FileOutput:
 class ConsoleOutput(FileOutput):
 	def __init__(self):
 		self.buf = sys.stdout
-		try:
-			self.width = int(shell('check_output', ['stty', 'size']).split()[1])
-		except:
-			self.width = 76
+		
+		# Do nice line-wrapping according to the size of the terminal.
+		# The 'stty' program queries standard input for terminal information.
+		if sys.stdin.isatty():
+			try:
+				self.width = int(shell('check_output', ['stty', 'size']).split()[1])
+			except:
+				self.width = 76
+
+		else:
+			# However if standard input is not a terminal, we would get
+			# "stty: standard input: Inappropriate ioctl for device". So
+			# we test with sys.stdin.isatty first, and if it is not a
+			# terminal don't do any line wrapping. When this script is
+			# run from cron, or if stdin has been redirected, this happens.
+			self.width = None
 
 class BufferedOutput:
 	# Record all of the instance method calls so we can play them back later.
@@ -917,7 +916,7 @@ if __name__ == "__main__":
 		run_checks(False, env, ConsoleOutput(), pool)
 
 	elif sys.argv[1] == "--show-changes":
-		run_and_output_changes(env, pool, sys.argv[-1] == "--smtp")
+		run_and_output_changes(env, pool)
 
 	elif sys.argv[1] == "--check-primary-hostname":
 		# See if the primary hostname appears resolvable and has a signed certificate.
@@ -925,10 +924,10 @@ if __name__ == "__main__":
 		if query_dns(domain, "A") != env['PUBLIC_IP']:
 			sys.exit(1)
 		ssl_certificates = get_ssl_certificates(env)
-		ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, ssl_certificates, env)
-		if not os.path.exists(ssl_certificate):
+		tls_cert = get_domain_ssl_files(domain, ssl_certificates, env)
+		if not os.path.exists(tls_cert["certificate"]):
 			sys.exit(1)
-		cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key, warn_if_expiring_soon=False)
+		cert_status, cert_status_details = check_certificate(domain, tls_cert["certificate"], tls_cert["private-key"], warn_if_expiring_soon=False)
 		if cert_status != "OK":
 			sys.exit(1)
 		sys.exit(0)

management/templates/index.html

@@ -70,7 +70,7 @@
     <!--[if lt IE 8]><p>Internet Explorer version 8 or any modern web browser is required to use this website, sorry.<![endif]-->
     <!--[if gt IE 7]><!-->
 
-    <div class="navbar navbar-inverse" role="navigation">
+    <div class="navbar navbar-inverse navbar-static-top" role="navigation">
       <div class="container">
         <div class="navbar-header">
           <button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target=".navbar-collapse">
@@ -87,7 +87,7 @@
               <a href="#" class="dropdown-toggle" data-toggle="dropdown">System <b class="caret"></b></a>
               <ul class="dropdown-menu">
                 <li><a href="#system_status" onclick="return show_panel(this);">Status Checks</a></li>
-                <li><a href="#ssl" onclick="return show_panel(this);">SSL Certificates</a></li>
+                <li><a href="#tls" onclick="return show_panel(this);">TLS (SSL) Certificates</a></li>
                 <li><a href="#system_backup" onclick="return show_panel(this);">Backup Status</a></li>
                 <li class="divider"></li>
                 <li class="dropdown-header">Advanced Pages</li>
@@ -155,7 +155,7 @@
       {% include "web.html" %}
       </div>
 
-      <div id="panel_ssl" class="admin_panel">
+      <div id="panel_tls" class="admin_panel">
       {% include "ssl.html" %}
       </div>
 

management/templates/login.html

@@ -117,7 +117,7 @@ function do_login() {
       // Open the next panel the user wants to go to. Do this after the XHR response
       // is over so that we don't start a new XHR request while this one is finishing,
       // which confuses the loading indicator.
-      setTimeout(function() { show_panel(!switch_back_to_panel ? 'system_status' : switch_back_to_panel) }, 300);
+      setTimeout(function() { show_panel(!switch_back_to_panel || switch_back_to_panel == "login" ? 'system_status' : switch_back_to_panel) }, 300);
     }
   })
 }

management/templates/ssl.html

@@ -1,12 +1,46 @@
 <style>
 </style>
 
-<h2>SSL Certificates</h2>
+<h2>TLS (SSL) Certificates</h2>
+
+<p>A TLS (formerly called SSL) certificate is a cryptographic file that proves to anyone connecting to a web address that the connection is secure between you and the owner of that address.</p>
+
+<p>You need a TLS certificate for this box&rsquo;s hostname ({{hostname}}) and every other domain name and subdomain that this box is hosting a website for (see the list below).</p>
+
+<div id="ssl_provision">
+<h3>Provision a Certificate</h3>
+
+<div id="ssl_provision_p" style="display: none; margin-top: 1.5em">
+  <button onclick='return provision_tls_cert();' class='btn btn-primary' style="float: left; margin: 0 1.5em 1em 0;">Provision</button>
+  <p>A TLS certificate can be automatically provisioned from <a href="https://letsencrypt.org/" target="_blank">Let&rsquo;s Encrypt</a>, a free TLS certificate provider, for:<br>
+  <span class="text-primary"></span></p>
+</div>
+
+<div class="clearfix"> </div>
+
+<div id="ssl_provision_result"></div>
+
+<div id="ssl_provision_problems_div" style="display: none;">
+<p style="margin-bottom: .5em;">Certificates cannot be automatically provisioned for:</p>
+<table id="ssl_provision_problems" style="margin-top: 0;" class="table">
+  <thead>
+    <tr>
+      <th>Domain</th>
+      <th>Problem</th>
+    </tr>
+  </thead>
+  <tbody>
+  </tbody>
+</table>
+<p>Use the <em>Install Certificate</em> button below for these domains.</p>
+</div>
+</div>
 
 <h3>Certificate Status</h3>
 
+<p style="margin-top: 1.5em">Certificates expire after a period of time. All certificates will be automatically renewed through <a href="https://letsencrypt.org/" target="_blank">Let&rsquo;s Encrypt</a> 14 days prior to expiration.</p>
 
-<table id="ssl_domains" class="table" style="margin-bottom: 2em; width: auto;">
+<table id="ssl_domains" class="table" style="margin-bottom: 2em; width: auto; display: none">
   <thead>
     <tr>
       <th>Domain</th>
@@ -18,17 +52,18 @@
   </tbody>
 </table>
 
-<p>A multi-domain or wildcard certificate will be automatically applied to any domains it is valid for.</p>
 
-<h3 id="ssl_install_header">Install SSL Certificate</h3>
+<h3 id="ssl_install_header">Install Certificate</h3>
 
-<p>There are many places where you can get a free or cheap SSL certificate. We recommend <a href="https://www.namecheap.com/security/ssl-certificates/domain-validation.aspx">Namecheap&rsquo;s $9 certificate</a>, <a href="https://www.startssl.com/">StartSSL&rsquo;s free express lane</a> or <a href="https://buy.wosign.com/free/">WoSign&rsquo;s free SSL</a></a>.</p>
+<p>There are many places where you can get a free or cheap certificate. We recommend <a href="https://www.namecheap.com/security/ssl-certificates/domain-validation.aspx">Namecheap&rsquo;s $9 certificate</a>, <a href="https://www.startssl.com/">StartSSL&rsquo;s free express lane</a> or <a href="https://buy.wosign.com/free/">WoSign&rsquo;s free TLS</a></a>.</p>
 
-<p>Which domain are you getting an SSL certificate for?</p>
+<p>Which domain are you getting a certificate for?</p>
 
 <p><select id="ssldomain" onchange="show_csr()" class="form-control" style="width: auto"></select></p>
 
-<p>What country are you in? This is required by some SSL certificate providers. You may leave this blank if you know your SSL certificate provider doesn't require it.</p>
+<p>(A multi-domain or wildcard certificate will be automatically applied to any domains it is valid for besides the one you choose above.)</p>
+
+<p>What country are you in? This is required by some TLS certificate providers. You may leave this blank if you know your TLS certificate provider doesn't require it.</p>
 
 <p><select id="sslcc" onchange="show_csr()" class="form-control" style="width: auto">
 	<option value="">(Select)</option>
@@ -38,18 +73,18 @@
 </select></p>
 
 <div id="csr_info" style="display: none">
-  <p>You will need to provide the SSL certificate provider this Certificate Signing Request (CSR):</p>
+  <p>You will need to provide the certificate provider this Certificate Signing Request (CSR):</p>
 
   <pre id="ssl_csr"></pre>
 
   <p><small>The CSR is safe to share. It can only be used in combination with a secret key stored on this machine.</small></p>
 
-  <p>The SSL certificate provider will then provide you with an SSL certificate. They may also provide you with an intermediate chain. Paste each separately into the boxes below:</p>
+  <p>The certificate provider will then provide you with a TLS/SSL certificate. They may also provide you with an intermediate chain. Paste each separately into the boxes below:</p>
 
-  <p style="margin-bottom: .5em">SSL certificate:</p>
+  <p style="margin-bottom: .5em">TLS/SSL certificate:</p>
   <p><textarea id="ssl_paste_cert" class="form-control" style="max-width: 40em; height: 8em" placeholder="-----BEGIN CERTIFICATE-----&#xA;stuff here&#xA;-----END CERTIFICATE-----"></textarea></p>
 
-  <p style="margin-bottom: .5em">SSL intermediate chain (if provided):</p>
+  <p style="margin-bottom: .5em">TLS/SSL intermediate chain (if provided):</p>
   <p><textarea id="ssl_paste_chain" class="form-control" style="max-width: 40em; height: 8em" placeholder="-----BEGIN CERTIFICATE-----&#xA;stuff here&#xA;-----END CERTIFICATE-----&#xA;-----BEGIN CERTIFICATE-----&#xA;more stuff here&#xA;-----END CERTIFICATE-----"></textarea></p>
 
   <p>After you paste in the information, click the install button.</p>
@@ -58,26 +93,53 @@
 </div>
 
 <script>
-function show_ssl() {
+function show_tls(keep_provisioning_shown) {
  api(
-    "/web/domains",
+    "/ssl/status",
     "GET",
     {
     },
-    function(domains) {
+    function(res) {
+      // provisioning status
+
+      if (!keep_provisioning_shown)
+        $('#ssl_provision').toggle(res.can_provision.length + res.cant_provision.length > 0)
+
+      $('#ssl_provision_p').toggle(res.can_provision.length > 0);
+      if (res.can_provision.length > 0)
+          $('#ssl_provision_p span').text(res.can_provision.join(", "));
+      
+      $('#ssl_provision_problems_div').toggle(res.cant_provision.length > 0);
+      $('#ssl_provision_problems tbody').text("");
+      for (var i = 0; i < res.cant_provision.length; i++) {
+        var domain = res.cant_provision[i];
+        var row = $("<tr><th class='domain'><a href=''></a></th><td class='status'></td></tr>");
+        $('#ssl_provision_problems tbody').append(row);
+        row.attr('data-domain', domain.domain);
+        row.find('.domain a').text(domain.domain);
+        row.find('.domain a').attr('href', 'https://' + domain.domain);
+        row.find('.status').text(domain.problem);
+      }
+
+      // certificate status
+      var domains = res.status;
       var tb = $('#ssl_domains tbody');
       tb.text('');
       $('#ssldomain').html('<option value="">(select)</option>');
-
+      $('#ssl_domains').show();
       for (var i = 0; i < domains.length; i++) {
         var row = $("<tr><th class='domain'><a href=''></a></th><td class='status'></td> <td class='actions'><a href='#' onclick='return ssl_install(this);' class='btn btn-xs'>Install Certificate</a></td></tr>");
         tb.append(row);
         row.attr('data-domain', domains[i].domain);
         row.find('.domain a').text(domains[i].domain);
         row.find('.domain a').attr('href', 'https://' + domains[i].domain);
-        row.addClass("text-" + domains[i].ssl_certificate[0]);
-        row.find('.status').text(domains[i].ssl_certificate[1]);
-        if (domains[i].ssl_certificate[0] == "success") {
+        if (domains[i].status == "not-applicable") {
+          domains[i].status = "muted"; // text-muted css class
+          row.find('.actions a').remove(); // no actions applicable
+        }
+        row.addClass("text-" + domains[i].status);
+        row.find('.status').text(domains[i].text);
+        if (domains[i].status == "success") {
           row.find('.actions a').addClass('btn-default').text('Replace Certificate');
         } else {
           row.find('.actions a').addClass('btn-primary').text('Install Certificate');
@@ -91,14 +153,15 @@ function show_ssl() {
 function ssl_install(elem) {
   var domain = $(elem).parents('tr').attr('data-domain');
   $('#ssldomain').val(domain);
-  $('#csr_info').slideDown();
-  $('#ssl_csr').text('Loading...');
   show_csr();
   $('html, body').animate({ scrollTop: $('#ssl_install_header').offset().top - $('.navbar-fixed-top').height() - 20 })
   return false;
 }
 
 function show_csr() {
+ if ($('#ssldomain').val() == "") return;
+ $('#csr_info').slideDown();
+ $('#ssl_csr').text('Loading...');
  api(
     "/ssl/csr/" + $('#ssldomain').val(),
     "POST",
@@ -122,10 +185,100 @@ function install_cert() {
     function(status) {
       if (/^OK($|\n)/.test(status)) {
         console.log(status)
-        show_modal_error("SSL Certificate Installation", "Certificate has been installed. Check that you have no connection problems to the domain.", function() { show_ssl(); $('#csr_info').slideUp(); });
+        show_modal_error("TLS Certificate Installation", "Certificate has been installed. Check that you have no connection problems to the domain.", function() { show_ssl(); $('#csr_info').slideUp(); });
       } else {
-        show_modal_error("SSL Certificate Installation", status);
+        show_modal_error("TLS Certificate Installation", status);
       }
     });
 }
+
+var agree_to_tos_url_prompt = null;
+var agree_to_tos_url = null;
+function provision_tls_cert() {
+  // Automatically provision any certs.
+  $('#ssl_provision_p .btn').attr('disabled', '1'); // prevent double-clicks
+  api(
+    "/ssl/provision",
+    "POST",
+    {
+      agree_to_tos_url: agree_to_tos_url
+    },
+    function(status) {
+      // Clear last attempt.
+      agree_to_tos_url = null;
+      $('#ssl_provision_result').text("");
+      may_reenable_provision_button = true;
+
+      // Nothing was done. There might also be problem domains, but we've already displayed those.
+      if (status.requests.length == 0) {
+        show_modal_error("TLS Certificate Provisioning", "There were no domain names to provision certificates for.");
+        // don't return - haven't re-enabled the provision button
+      }
+
+      // Each provisioning API call returns zero or more "requests" which represent
+      // a request to Let's Encrypt for a single certificate. Normally there is just
+      // one request (for a single multi-domain certificate).
+      for (var i = 0; i < status.requests.length; i++) {
+        var r = status.requests[i];
+
+        // create an HTML block to display the results of this request
+        var n = $("<div><h4/><p/></div>");
+        $('#ssl_provision_result').append(n);
+
+        // show a header only to disambiguate request blocks
+        if (status.requests.length > 0)
+          n.find("h4").text(r.domains.join(", "));
+
+        if (r.result == "agree-to-tos") {
+          // user needs to agree to Let's Encrypt's TOS
+          agree_to_tos_url_prompt = r.url;
+          $('#ssl_provision_p .btn').attr('disabled', '1');
+          n.find("p").html("Please open and review <a href='" + r.url + "' target='_blank'>Let's Encrypt's terms of service agreement</a>. You must agree to their terms for a certificate to be automatically provisioned from them.");
+          n.append($('<button onclick="agree_to_tos_url = agree_to_tos_url_prompt; return provision_tls_cert();" class="btn btn-success" style="margin-left: 2em">Agree &amp; Try Again</button>'));
+
+          // don't re-enable the Provision button -- user must use the Agree button
+          may_reenable_provision_button = false;
+
+        } else if (r.result == "error") {
+          n.find("p").addClass("text-danger").text(r.message);
+
+        } else if (r.result == "wait") {
+          // Show a button that counts down to zero, at which point it becomes enabled.
+          n.find("p").text("A certificate is now in the process of being provisioned, but it takes some time. Please wait until the Finish button is enabled, and then click it to acquire the certificate.");
+          var b = $('<button onclick="return provision_tls_cert();" class="btn btn-success" style="margin-left: 2em">Finish</button>');
+          b.attr("disabled", "1");
+          var now = new Date();
+          n.append(b);
+          function ready_to_finish() {
+            var remaining = Math.round(r.seconds - (new Date() - now)/1000);
+            if (remaining > 0) {
+              setTimeout(ready_to_finish, 1000);
+              b.text("Finish (" + remaining + "...)")
+            } else {
+              b.text("Finish (ready)")
+              b.removeAttr("disabled");
+            }
+          }
+          ready_to_finish();
+          
+          // don't re-enable the Provision button -- user must use the Retry button when it becomes enabled
+          may_reenable_provision_button = false;
+
+        } else if (r.result == "installed") {
+          n.find("p").addClass("text-success").text("The TLS certificate was provisioned and installed.");
+          setTimeout("show_tls(true)", 1); // update main table of certificate statuses, call with arg keep_provisioning_shown true so that we don't clear what we just outputted
+        }
+        
+        // display the detailed log info in case of problems
+        var trace = $("<div class='small text-muted' style='margin-top: 1.5em'>Log:</div>");
+        n.append(trace);
+        for (var j = 0; j < r.log.length; j++)
+          trace.append($("<div/>").text(r.log[j]));
+
+      }
+
+      if (may_reenable_provision_button)
+        $('#ssl_provision_p .btn').removeAttr("disabled");
+    });
+}
 </script>

management/templates/system-backup.html

@@ -121,6 +121,11 @@ function show_system_backup() {
     "GET",
     { },
     function(r) {
+      if (r.error) {
+          show_modal_error("Backup Error", $("<pre/>").text(r.error));
+          return;
+      }
+
       $('#backup-status tbody').html("");
       var total_disk_size = 0;
 
@@ -137,7 +142,7 @@ function show_system_backup() {
         var b = r.backups[i];
         var tr = $('<tr/>');
         if (b.full) tr.addClass("full-backup");
-        tr.append( $('<td/>').text(b.date_str + " " + r.tz) );
+        tr.append( $('<td/>').text(b.date_str) );
         tr.append( $('<td/>').text(b.date_delta + " ago") );
         tr.append( $('<td/>').text(b.full ? "full" : "increment") );
         tr.append( $('<td style="text-align: right"/>').text( nice_size(b.size)) );

management/web_update.py

@@ -9,7 +9,7 @@ from dns_update import get_custom_dns_config, get_dns_zones
 from ssl_certificates import get_ssl_certificates, get_domain_ssl_files, check_certificate
 from utils import shell, safe_domain_name, sort_domains
 
-def get_web_domains(env, include_www_redirects=True):
+def get_web_domains(env, include_www_redirects=True, exclude_dns_elsewhere=True):
 	# What domains should we serve HTTP(S) for?
 	domains = set()
 
@@ -24,9 +24,10 @@ def get_web_domains(env, include_www_redirects=True):
 		# the topmost of each domain we serve.
 		domains |= set('www.' + zone for zone, zonefile in get_dns_zones(env))
 	 
-	# ...Unless the domain has an A/AAAA record that maps it to a different
-	# IP address than this box. Remove those domains from our list.
-	domains -= get_domains_with_a_records(env)
+	if exclude_dns_elsewhere:
+		# ...Unless the domain has an A/AAAA record that maps it to a different
+		# IP address than this box. Remove those domains from our list.
+		domains -= get_domains_with_a_records(env)
 
 	# Ensure the PRIMARY_HOSTNAME is in the list so we can serve webmail
 	# as well as Z-Push for Exchange ActiveSync. This can't be removed
@@ -119,7 +120,7 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 	root = get_web_root(domain, env)
 
 	# What private key and SSL certificate will we use for this domain?
-	ssl_key, ssl_certificate, ssl_via = get_domain_ssl_files(domain, ssl_certificates, env)
+	tls_cert = get_domain_ssl_files(domain, ssl_certificates, env)
 
 	# ADDITIONAL DIRECTIVES.
 
@@ -136,7 +137,7 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 		finally:
 			f.close()
 		return sha1.hexdigest()
-	nginx_conf_extra += "# ssl files sha1: %s / %s\n" % (hashfile(ssl_key), hashfile(ssl_certificate))
+	nginx_conf_extra += "# ssl files sha1: %s / %s\n" % (hashfile(tls_cert["private-key"]), hashfile(tls_cert["certificate"]))
 
 	# Add in any user customizations in YAML format.
 	hsts = "yes"
@@ -177,8 +178,8 @@ def make_domain_config(domain, templates, ssl_certificates, env):
 	nginx_conf = nginx_conf.replace("$STORAGE_ROOT", env['STORAGE_ROOT'])
 	nginx_conf = nginx_conf.replace("$HOSTNAME", domain)
 	nginx_conf = nginx_conf.replace("$ROOT", root)
-	nginx_conf = nginx_conf.replace("$SSL_KEY", ssl_key)
-	nginx_conf = nginx_conf.replace("$SSL_CERTIFICATE", ssl_certificate)
+	nginx_conf = nginx_conf.replace("$SSL_KEY", tls_cert["private-key"])
+	nginx_conf = nginx_conf.replace("$SSL_CERTIFICATE", tls_cert["certificate"])
 	nginx_conf = nginx_conf.replace("$REDIRECT_DOMAIN", re.sub(r"^www\.", "", domain)) # for default www redirects to parent domain
 
 	return nginx_conf
@@ -193,20 +194,15 @@ def get_web_root(domain, env, test_exists=True):
 def get_web_domains_info(env):
 	www_redirects = set(get_web_domains(env)) - set(get_web_domains(env, include_www_redirects=False))
 	has_root_proxy_or_redirect = set(get_web_domains_with_root_overrides(env))
+	ssl_certificates = get_ssl_certificates(env)
 
 	# for the SSL config panel, get cert status
 	def check_cert(domain):
-		ssl_certificates = get_ssl_certificates(env)
-		x = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
-		if x is None: return ("danger", "No Certificate Installed")
-		ssl_key, ssl_certificate, ssl_via = x
-		cert_status, cert_status_details = check_certificate(domain, ssl_certificate, ssl_key)
+		tls_cert = get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=True)
+		if tls_cert is None: return ("danger", "No Certificate Installed")
+		cert_status, cert_status_details = check_certificate(domain, tls_cert["certificate"], tls_cert["private-key"])
 		if cert_status == "OK":
-			if not ssl_via:
-				return ("success", "Signed & valid. " + cert_status_details)
-			else:
-				# This is an alternate domain but using the same cert as the primary domain.
-				return ("success", "Signed & valid. " + ssl_via)
+			return ("success", "Signed & valid. " + cert_status_details)
 		elif cert_status == "SELF-SIGNED":
 			return ("warning", "Self-signed. Get a signed certificate to stop warnings.")
 		else:
@@ -221,4 +217,4 @@ def get_web_domains_info(env):
 			"static_enabled": domain not in (www_redirects | has_root_proxy_or_redirect),
 		}
 		for domain in get_web_domains(env)
-	]
+	]

security.md

@@ -38,7 +38,7 @@ These services are protected by [TLS](https://en.wikipedia.org/wiki/Transport_La
 
 The services all follow these rules:
 
-* SSL certificates are generated with 2048-bit RSA keys and SHA-256 fingerprints. The box provides a self-signed certificate by default. The [setup guide](https://mailinabox.email/guide.html) explains how to verify the certificate fingerprint on first login. Users are encouraged to replace the certificate with a proper CA-signed one. ([source](setup/ssl.sh))
+* TLS certificates are generated with 2048-bit RSA keys and SHA-256 fingerprints. The box provides a self-signed certificate by default. The [setup guide](https://mailinabox.email/guide.html) explains how to verify the certificate fingerprint on first login. Users are encouraged to replace the certificate with a proper CA-signed one. ([source](setup/ssl.sh))
 * Only TLSv1, TLSv1.1 and TLSv1.2 are offered (the older SSL protocols are not offered).
 * Export-grade ciphers, the anonymous DH/ECDH algorithms (aNULL), and clear-text ciphers (eNULL) are not offered.
 * The minimum cipher key length offered is 112 bits. The maximum is 256 bits. Diffie-Hellman ciphers use a 2048-bit key for forward secrecy.

setup/bootstrap.sh

@@ -2,12 +2,12 @@
 #########################################################
 # This script is intended to be run like this:
 #
-#   curl https://.../bootstrap.sh | sudo bash
+#   curl https://mailinabox.email/setup.sh | sudo bash
 #
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.14
+	TAG=v0.17b
 fi
 
 # Are we running as root?

setup/dkim.sh

@@ -39,7 +39,7 @@ fi
 # Create a new DKIM key. This creates mail.private and mail.txt
 # in $STORAGE_ROOT/mail/dkim. The former is the private key and
 # the latter is the suggested DNS TXT entry which we'll include
-# in our DNS setup. Note tha the files are named after the
+# in our DNS setup. Note that the files are named after the
 # 'selector' of the key, which we can change later on to support
 # key rotation.
 #

setup/mail-dovecot.sh

@@ -203,5 +203,8 @@ chown -R mail.mail $STORAGE_ROOT/mail/sieve
 ufw_allow imaps
 ufw_allow pop3s
 
+# Allow the Sieve port in the firewall.
+ufw_allow sieve
+
 # Restart services.
 restart_service dovecot

setup/mail-postfix.sh

@@ -57,15 +57,26 @@ apt_install postfix postfix-pcre postgrey ca-certificates
 # Set some basic settings...
 #
 # * Have postfix listen on all network interfaces.
+# * Make outgoing connections on a particular interface (if multihomed) so that SPF passes on the receiving side.
 # * Set our name (the Debian default seems to be "localhost" but make it our hostname).
 # * Set the name of the local machine to localhost, which means xxx@localhost is delivered locally, although we don't use it.
 # * Set the SMTP banner (which must have the hostname first, then anything).
 tools/editconf.py /etc/postfix/main.cf \
 	inet_interfaces=all \
+	smtp_bind_address=$PRIVATE_IP \
+	smtp_bind_address6=$PRIVATE_IPV6 \
 	myhostname=$PRIMARY_HOSTNAME\
 	smtpd_banner="\$myhostname ESMTP Hi, I'm a Mail-in-a-Box (Ubuntu/Postfix; see https://mailinabox.email/)" \
 	mydestination=localhost
 
+# Tweak some queue settings:
+# * Inform users when their e-mail delivery is delayed more than 3 hours (default is not to warn).
+# * Stop trying to send an undeliverable e-mail after 2 days (instead of 5), and for bounce messages just try for 1 day.
+tools/editconf.py /etc/postfix/main.cf \
+	delay_warning_time=3h \
+	maximal_queue_lifetime=2d \
+	bounce_queue_lifetime=1d
+
 # ### Outgoing Mail
 
 # Enable the 'submission' port 587 smtpd server and tweak its settings.

setup/management.sh

@@ -4,16 +4,25 @@ source setup/functions.sh
 
 echo "Installing Mail-in-a-Box system management daemon..."
 
-# Switching python 2 boto to package manager's, not pypi's.
-if [ -f /usr/local/lib/python2.7/dist-packages/boto/__init__.py ]; then hide_output pip uninstall -y boto; fi
+# Install packages.
+# flask, yaml, dnspython, and dateutil are all for our Python 3 management daemon itself.
+# duplicity does backups. python-pip is so we can 'pip install boto' for Python 2, for duplicity, so it can do backups to AWS S3.
+apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil python-pip
 
-# duplicity uses python 2 so we need to use the python 2 package of boto
-# build-essential libssl-dev libffi-dev python3-dev: Required to pip install cryptography.
-apt_install python3-flask links duplicity python-boto libyaml-dev python3-dnspython python3-dateutil \
-	build-essential libssl-dev libffi-dev python3-dev python-pip
-hide_output pip3 install --upgrade rtyaml "email_validator>=1.0.0" "idna>=2.0.0" "cryptography>=1.0.2" boto
+# These are required to pip install cryptography.
+apt_install build-essential libssl-dev libffi-dev python3-dev
 
-# email_validator is repeated in setup/questions.sh
+# Install other Python 3 packages used by the management daemon.
+# The first line is the packages that Josh maintains himself!
+# NOTE: email_validator is repeated in setup/questions.sh, so please keep the versions synced.
+hide_output pip3 install --upgrade \
+	rtyaml "email_validator>=1.0.0" "free_tls_certificates>=0.1.3" \
+	"idna>=2.0.0" "cryptography>=1.0.2" boto psutil
+
+# duplicity uses python 2 so we need to get the python 2 package of boto to have backups to S3.
+# boto from the Ubuntu package manager is too out-of-date -- it doesn't support the newer
+# S3 api used in some regions, which breaks backups to those regions.  See #627, #653.
+hide_output pip install --upgrade boto
 
 # Create a backup directory and a random key for encrypting backups.
 mkdir -p $STORAGE_ROOT/backup
@@ -44,5 +53,5 @@ cat > /etc/cron.d/mailinabox-nightly << EOF;
 0 3 * * *	root	(cd `pwd` && management/daily_tasks.sh)
 EOF
 
-# Start it.
+# Start the management server.
 restart_service mailinabox

setup/owncloud.sh

@@ -52,8 +52,8 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 	# The two apps we actually want are not in ownCloud core. Clone them from
 	# their github repositories.
 	mkdir -p /usr/local/lib/owncloud/apps
-	git_clone https://github.com/owncloud/contacts 4ff855e7c2075309041bead09fbb9eb7df678244 '' /usr/local/lib/owncloud/apps/contacts
-	git_clone https://github.com/owncloud/calendar ec53139b144c0f842c33813305612e8006c42ea5 '' /usr/local/lib/owncloud/apps/calendar
+	git_clone https://github.com/owncloudarchive/contacts 4ff855e7c2075309041bead09fbb9eb7df678244 '' /usr/local/lib/owncloud/apps/contacts
+	git_clone https://github.com/owncloudarchive/calendar ec53139b144c0f842c33813305612e8006c42ea5 '' /usr/local/lib/owncloud/apps/calendar
 
 	# Fix weird permissions.
 	chmod 750 /usr/local/lib/owncloud/{apps,config}

setup/questions.sh

@@ -3,9 +3,9 @@ if [ -z "$NONINTERACTIVE" ]; then
 	# this was being able to ask the user for input even if stdin has been redirected,
 	# e.g. if we piped a bootstrapping install script to bash to get started. In that
 	# case, the nifty '[ -t 0 ]' test won't work. But with Vagrant we must suppress so we
-	# use a shell flag instead. Really supress any output from installing dialog.
+	# use a shell flag instead. Really suppress any output from installing dialog.
 	#
-	# Also install depencies needed to validate the email address.
+	# Also install dependencies needed to validate the email address.
 	if [ ! -f /usr/bin/dialog ] || [ ! -f /usr/bin/python3 ] || [ ! -f /usr/bin/pip3 ]; then
 		echo Installing packages needed for setup...
 		apt-get -q -q update
@@ -18,7 +18,8 @@ if [ -z "$NONINTERACTIVE" ]; then
 	message_box "Mail-in-a-Box Installation" \
 		"Hello and thanks for deploying a Mail-in-a-Box!
 		\n\nI'm going to ask you a few questions.
-		\n\nTo change your answers later, just run 'sudo mailinabox' from the command line."
+		\n\nTo change your answers later, just run 'sudo mailinabox' from the command line.
+		\n\nNOTE: You should only install this on a brand new Ubuntu installation 100% dedicated to Mail-in-a-Box. Mail-in-a-Box will, for example, remove apache2."
 fi
 
 # The box needs a name.

setup/start.sh

@@ -10,8 +10,8 @@ source setup/preflight.sh
 
 # Ensure Python reads/writes files in UTF-8. If the machine
 # triggers some other locale in Python, like ASCII encoding,
-# Python may not be able to read/write files. Here and in
-# the management daemon startup script.
+# Python may not be able to read/write files. This is also
+# in the management daemon startup script and the cron script.
 
 if [ -z `locale -a | grep en_US.utf8` ]; then
     # Generate locale if not exists
@@ -116,6 +116,10 @@ done
 tools/dns_update
 tools/web_update
 
+# If DNS is already working, try to provision TLS certficates from Let's Encrypt.
+# Suppress extra reasons why domains aren't getting a new certificate.
+management/ssl_certificates.py -q
+
 # If there aren't any mail users yet, create one.
 source setup/firstuser.sh
 

setup/webmail.sh

@@ -34,12 +34,12 @@ apt-get purge -qq -y roundcube* #NODOC
 # Install Roundcube from source if it is not already present or if it is out of date.
 # Combine the Roundcube version number with the commit hash of vacation_sieve to track
 # whether we have the latest version.
-VERSION=1.1.3
-HASH=4513227bd64eb8564f056817341b1dfe478e215e
+VERSION=1.1.4
+HASH=4883c8bb39fadf8af94ffb09ee426cba9f8ef2e3
 VACATION_SIEVE_VERSION=91ea6f52216390073d1f5b70b5f6bea0bfaee7e5
-PERSISTENT_LOGIN_VERSION=117fbd8f93b56b2bf72ad055193464803ef3bc36
+PERSISTENT_LOGIN_VERSION=1e9d724476a370ce917a2fcd5b3217b0c306c24e
 HTML5_NOTIFIER_VERSION=046eb388dd63b1ec77a3ee485757fc25ae9e684d
-UPDATE_KEY=$VERSION:$VACATION_SIEVE_VERSION:$PERSISTENT_LOGIN_VERSION:$HTML5_NOTIFIER_VERSION
+UPDATE_KEY=$VERSION:$VACATION_SIEVE_VERSION:$PERSISTENT_LOGIN_VERSION:$HTML5_NOTIFIER_VERSION:a
 needs_update=0 #NODOC
 if [ ! -f /usr/local/lib/roundcubemail/version ]; then
 	# not installed yet #NODOC
@@ -54,7 +54,7 @@ if [ $needs_update == 1 ]; then
 		https://downloads.sourceforge.net/project/roundcubemail/roundcubemail/$VERSION/roundcubemail-$VERSION.tar.gz \
 		$HASH \
 		/tmp/roundcube.tgz
-	tar -C /usr/local/lib -zxf /tmp/roundcube.tgz
+	tar -C /usr/local/lib --no-same-owner -zxf /tmp/roundcube.tgz
 	rm -rf /usr/local/lib/roundcubemail
 	mv /usr/local/lib/roundcubemail-$VERSION/ /usr/local/lib/roundcubemail
 	rm -f /tmp/roundcube.tgz

tools/parse-nginx-log-bootstrap-accesses.py

@@ -2,7 +2,8 @@
 #
 # This is a tool Josh uses on his box serving mailinabox.email to parse the nginx
 # access log to see how many people are installing Mail-in-a-Box each day, by
-# looking at accesses to the bootstrap.sh script.
+# looking at accesses to the bootstrap.sh script (which is currently at the URL
+# .../setup.sh).
 
 import re, glob, gzip, os.path, json
 import dateutil.parser
@@ -24,9 +25,10 @@ for fn in glob.glob("/var/log/nginx/access.log*"):
 	# Loop through the lines in the access log.
 	with f:
 		for line in f:
-			# Find lines that are GETs on /bootstrap.sh by either curl or wget.
+			# Find lines that are GETs on the bootstrap script by either curl or wget.
 			# (Note that we purposely skip ...?ping=1 requests which is the admin panel querying us for updates.)
-			m = re.match(rb"(?P<ip>\S+) - - \[(?P<date>.*?)\] \"GET /bootstrap.sh HTTP/.*\" 200 \d+ .* \"(?:curl|wget)", line, re.I)
+			# (Also, the URL changed in January 2016, but we'll accept both.)
+			m = re.match(rb"(?P<ip>\S+) - - \[(?P<date>.*?)\] \"GET /(bootstrap.sh|setup.sh) HTTP/.*\" 200 \d+ .* \"(?:curl|wget)", line, re.I)
 			if m:
 				date, time = m.group("date").decode("ascii").split(":", 1)
 				date = dateutil.parser.parse(date).date().isoformat()