v0.21c

pip<6.1 + setuptools>=34 have a problem with packages that
try to update setuptools during installation, like cryptography.
See https://github.com/pypa/pip/issues/4253. The Ubuntu 14.04
package versions are pip 1.5.4 and setuptools 3.3. When we
install cryptography under those versions, it tries to update
setuptools to version 34, which became available about 10 days
ago, and then pip gets permanently broken with errors like
"ImportError: No module named 'packaging'".

The easiest work-around on systems that aren't already broken is
to upgrade pip and setuptools individually before we install any
package that tries to update setuptools.

Also try to detect a broken system and forcibly remove setuptools
first before trying to install/upgrade pip.

fixes #1080, fixes #1081, fixes #1086
see #1083
see https://discourse.mailinabox.email/t/error-with-pip-and-python/1880
see https://discourse.mailinabox.email/t/error-installing-mib/1875

CHANGELOG.md

@@ -1,6 +1,45 @@
 CHANGELOG
 =========
 
+v0.21c (February 1, 2017)
+-------------------------
+
+Installations and upgrades started failing about 10 days ago with the error "ImportError: No module named 'packaging'" after an upstream package (Python's setuptools) was updated by its maintainers. The updated package conflicted with Ubuntu 14.04's version of another package (Python's pip). This update upgrades both packages to remove the conflict.
+
+If you already encountered the error during installation or upgrade of Mail-in-a-Box, this update may not correct the problem on your existing system. See https://discourse.mailinabox.email/t/v0-21c-release-fixes-python-package-installation-issue/1881 for help if the problem persists after upgrading to this version of Mail-in-a-Box.
+
+v0.21b (December 4, 2016)
+-------------------------
+
+This update corrects a first-time installation issue introduced in v0.21 caused by the new Exchange/ActiveSync feature.
+
+v0.21 (November 30, 2016)
+-------------------------
+
+This version updates ownCloud, which may include security fixes, and makes some other smaller improvements.
+
+Mail:
+
+* Header privacy filters were improperly running on the contents of forwarded email --- that's fixed.
+* We have another go at fixing a long-standing issue with training the spam filter (because of a file permissions issue).
+* Exchange/ActiveSync will now use your display name set in Roundcube in the From: line of outgoing email.
+
+ownCloud:
+
+* Updated ownCloud to version 9.1.1.
+
+Control panel:
+
+* Backups can now be made using rsync-over-ssh!
+* Status checks failed if the system doesn't support iptables or doesn't have ufw installed.
+* Added support for SSHFP records when sshd listens on non-standard ports.
+* Recommendations for TLS certificate providers were removed now that everyone mostly uses Let's Encrypt.
+
+System:
+
+* Ubuntu's "Upgrade to 16.04" notice is suppressed since you should not do that.
+* Lowered memory requirements to 512MB, display a warning if system memory is below 768MB.
+
 v0.20 (September 23, 2016)
 --------------------------
 

README.md

@@ -59,7 +59,7 @@ by me:
 	$ curl -s https://keybase.io/joshdata/key.asc | gpg --import
 	gpg: key C10BDD81: public key "Joshua Tauberer <jt@occams.info>" imported
 
-	$ git verify-tag v0.20
+	$ git verify-tag v0.21c
 	gpg: Signature made ..... using RSA key ID C10BDD81
 	gpg: Good signature from "Joshua Tauberer <jt@occams.info>"
 	gpg: WARNING: This key is not certified with a trusted signature!
@@ -72,7 +72,7 @@ and on my [personal homepage](https://razor.occams.info/). (Of course, if this r
 
 Checkout the tag corresponding to the most recent release:
 
-	$ git checkout v0.20
+	$ git checkout v0.21c
 
 Begin the installation.
 
@@ -95,5 +95,5 @@ The History
 * In 2007 I wrote a relatively popular Mozilla Thunderbird extension that added client-side SPF and DKIM checks to mail to warn users about possible phishing: [add-on page](https://addons.mozilla.org/en-us/thunderbird/addon/sender-verification-anti-phish/), [source](https://github.com/JoshData/thunderbird-spf).
 * In August 2013 I began Mail-in-a-Box by combining my own mail server configuration with the setup in ["NSA-proof your email in 2 hours"](http://sealedabstract.com/code/nsa-proof-your-e-mail-in-2-hours/) and making the setup steps reproducible with bash scripts.
 * Mail-in-a-Box was a semifinalist in the 2014 [Knight News Challenge](https://www.newschallenge.org/challenge/2014/submissions/mail-in-a-box), but it was not selected as a winner.
-* Mail-in-a-Box hit the front page of Hacker News in [April](https://news.ycombinator.com/item?id=7634514) 2014, [September](https://news.ycombinator.com/item?id=8276171) 2014, and [May](https://news.ycombinator.com/item?id=9624267) 2015.
+* Mail-in-a-Box hit the front page of Hacker News in [April](https://news.ycombinator.com/item?id=7634514) 2014, [September](https://news.ycombinator.com/item?id=8276171) 2014, [May](https://news.ycombinator.com/item?id=9624267) 2015, and [November](https://news.ycombinator.com/item?id=13050500) 2016.
 * FastCompany mentioned Mail-in-a-Box a [roundup of privacy projects](http://www.fastcompany.com/3047645/your-own-private-cloud) on June 26, 2015.

conf/zpush/backend_imap.php

@@ -8,7 +8,7 @@
 define('IMAP_SERVER', '127.0.0.1');
 define('IMAP_PORT', 993);
 define('IMAP_OPTIONS', '/ssl/norsh/novalidate-cert');
-define('IMAP_DEFAULTFROM', '');
+define('IMAP_DEFAULTFROM', 'sql');
 
 define('SYSTEM_MIME_TYPES_MAPPING', '/etc/mime.types');
 define('IMAP_AUTOSEEN_ON_DELETE', false);
@@ -23,15 +23,16 @@ define('IMAP_FOLDER_TRASH', 'TRASH');
 define('IMAP_FOLDER_SPAM', 'SPAM');
 define('IMAP_FOLDER_ARCHIVE', 'ARCHIVE');
 
-
-// not used
-define('IMAP_FROM_SQL_DSN', '');
+define('IMAP_FROM_SQL_DSN', 'sqlite:STORAGE_ROOT/mail/roundcube/roundcube.sqlite');
 define('IMAP_FROM_SQL_USER', '');
 define('IMAP_FROM_SQL_PASSWORD', '');
 define('IMAP_FROM_SQL_OPTIONS', serialize(array(PDO::ATTR_PERSISTENT => true)));
-define('IMAP_FROM_SQL_QUERY', "select first_name, last_name, mail_address from users where mail_address = '#username@#domain'");
-define('IMAP_FROM_SQL_FIELDS', serialize(array('first_name', 'last_name', 'mail_address')));
-define('IMAP_FROM_SQL_FROM', '#first_name #last_name <#mail_address>');
+define('IMAP_FROM_SQL_QUERY', "SELECT name, email FROM identities i INNER JOIN users u ON i.user_id = u.user_id WHERE u.username = '#username' AND i.standard = 1 AND i.del = 0 AND i.name <> ''");
+define('IMAP_FROM_SQL_FIELDS', serialize(array('name', 'email')));
+define('IMAP_FROM_SQL_FROM', '#name <#email>');
+define('IMAP_FROM_SQL_FULLNAME', '#name');
+
+// not used
 define('IMAP_FROM_LDAP_SERVER', '');
 define('IMAP_FROM_LDAP_SERVER_PORT', '389');
 define('IMAP_FROM_LDAP_USER', 'cn=zpush,ou=servers,dc=zpush,dc=org');
@@ -40,6 +41,7 @@ define('IMAP_FROM_LDAP_BASE', 'dc=zpush,dc=org');
 define('IMAP_FROM_LDAP_QUERY', '(mail=#username@#domain)');
 define('IMAP_FROM_LDAP_FIELDS', serialize(array('givenname', 'sn', 'mail')));
 define('IMAP_FROM_LDAP_FROM', '#givenname #sn <#mail>');
+define('IMAP_FROM_LDAP_FULLNAME', '#givenname #sn');
 
 define('IMAP_SMTP_METHOD', 'sendmail');
 

management/backup.py

@@ -13,6 +13,11 @@ import rtyaml
 
 from utils import exclusive_process, load_environment, shell, wait_for_service, fix_boto
 
+rsync_ssh_options = [
+	"--ssh-options='-i /root/.ssh/id_rsa_miab'",
+	"--rsync-options=-e \"/usr/bin/ssh -oStrictHostKeyChecking=no -oBatchMode=yes -p 22 -i /root/.ssh/id_rsa_miab\"",
+]
+
 def backup_status(env):
 	# Root folder
 	backup_root = os.path.join(env["STORAGE_ROOT"], 'backup')
@@ -52,6 +57,7 @@ def backup_status(env):
 			"size": 0, # collection-status doesn't give us the size
 			"volumes": keys[2], # number of archive volumes for this backup (not really helpful)
 		}
+
 	code, collection_status = shell('check_output', [
 		"/usr/bin/duplicity",
 		"collection-status",
@@ -59,7 +65,7 @@ def backup_status(env):
 		"--gpg-options", "--cipher-algo=AES256",
 		"--log-fd", "1",
 		config["target"],
-		],
+		] + rsync_ssh_options,
 		get_env(env),
 		trap=True)
 	if code != 0:
@@ -204,7 +210,7 @@ def perform_backup(full_backup):
 	backup_cache_dir = os.path.join(backup_root, 'cache')
 	backup_dir = os.path.join(backup_root, 'encrypted')
 
-	# Are backups dissbled?
+	# Are backups disabled?
 	if config["target"] == "off":
 		return
 
@@ -283,7 +289,7 @@ def perform_backup(full_backup):
 			env["STORAGE_ROOT"],
 			config["target"],
 			"--allow-source-mismatch"
-			],
+			] + rsync_ssh_options,
 			get_env(env))
 	finally:
 		# Start services again.
@@ -305,7 +311,7 @@ def perform_backup(full_backup):
 		"--archive-dir", backup_cache_dir,
 		"--force",
 		config["target"]
-		],
+		] + rsync_ssh_options,
 		get_env(env))
 
 	# From duplicity's manual:
@@ -320,7 +326,7 @@ def perform_backup(full_backup):
 		"--archive-dir", backup_cache_dir,
 		"--force",
 		config["target"]
-		],
+		] + rsync_ssh_options,
 		get_env(env))
 
 	# Change ownership of backups to the user-data user, so that the after-bcakup
@@ -359,7 +365,7 @@ def run_duplicity_verification():
 		"--exclude", backup_root,
 		config["target"],
 		env["STORAGE_ROOT"],
-	], get_env(env))
+	] + rsync_ssh_options, get_env(env))
 
 def run_duplicity_restore(args):
 	env = load_environment()
@@ -370,7 +376,7 @@ def run_duplicity_restore(args):
 		"restore",
 		"--archive-dir", backup_cache_dir,
 		config["target"],
-		] + args,
+		] + rsync_ssh_options + args,
 	get_env(env))
 
 def list_target_files(config):
@@ -383,6 +389,36 @@ def list_target_files(config):
 	if p.scheme == "file":
 		return [(fn, os.path.getsize(os.path.join(p.path, fn))) for fn in os.listdir(p.path)]
 
+	elif p.scheme == "rsync":
+		rsync_fn_size_re = re.compile(r'.*    ([^ ]*) [^ ]* [^ ]* (.*)')
+		rsync_target = '{host}:{path}'
+
+		_, target_host, target_path = config['target'].split('//')
+		target_path = '/' + target_path
+		if not target_path.endswith('/'):
+			target_path += '/'
+
+		rsync_command = [ 'rsync',
+					'-e',
+					'/usr/bin/ssh -i /root/.ssh/id_rsa_miab -oStrictHostKeyChecking=no -oBatchMode=yes',
+					'--list-only',
+					'-r',
+					rsync_target.format(
+						host=target_host,
+						path=target_path)
+				]
+
+		code, listing = shell('check_output', rsync_command, trap=True)
+		if code == 0:
+			ret = []
+			for l in listing.split('\n'):
+				match = rsync_fn_size_re.match(l)
+				if match:
+					ret.append( (match.groups()[1], int(match.groups()[0].replace(',',''))) )
+			return ret
+		else:
+			raise ValueError("Connection to rsync host failed")
+
 	elif p.scheme == "s3":
 		# match to a Region
 		fix_boto() # must call prior to importing boto
@@ -482,6 +518,9 @@ def get_backup_config(env, for_save=False, for_ui=False):
 	if config["target"] == "local":
 		# Expand to the full URL.
 		config["target"] = "file://" + config["file_target_directory"]
+	ssh_pub_key = os.path.join('/root', '.ssh', 'id_rsa_miab.pub')
+	if os.path.exists(ssh_pub_key):
+		config["ssh_pub_key"] = open(ssh_pub_key, 'r').read()
 
 	return config
 

management/dns_update.py

@@ -348,7 +348,18 @@ def build_sshfp_records():
 	# like the known_hosts file: hostname, keytype, fingerprint. The order
 	# of the output is arbitrary, so sort it to prevent spurrious updates
 	# to the zone file (that trigger bumping the serial number).
-	keys = shell("check_output", ["ssh-keyscan", "localhost"])
+
+	# scan the sshd_config and find the ssh ports (port 22 may be closed)
+	with open('/etc/ssh/sshd_config', 'r') as f:
+		ports = []
+		t = f.readlines()
+		for line in t:
+			s = line.split()
+			if len(s) == 2 and s[0] == 'Port':
+				ports = ports + [s[1]]
+	# the keys are the same at each port, so we only need to get
+	# them at the first port found (may not be port 22)
+	keys = shell("check_output", ["ssh-keyscan", "-p", ports[0], "localhost"])
 	for key in sorted(keys.split("\n")):
 		if key.strip() == "" or key[0] == "#": continue
 		try:

management/status_checks.py

@@ -169,8 +169,19 @@ def run_system_checks(rounded_values, env, output):
 	check_free_memory(rounded_values, env, output)
 
 def check_ufw(env, output):
-	ufw = shell('check_output', ['ufw', 'status']).splitlines()
+	if not os.path.isfile('/usr/sbin/ufw'):
+		output.print_warning("""The ufw program was not installed. If your system is able to run iptables, rerun the setup.""")
+		return
 
+	code, ufw = shell('check_output', ['ufw', 'status'], trap=True)
+
+	if code != 0:
+		# The command failed, it's safe to say the firewall is disabled
+		output.print_warning("""The firewall is not working on this machine. An error was received
+					while trying to check the firewall. To investigate run 'sudo ufw status'.""")
+		return
+
+	ufw = ufw.splitlines()
 	if ufw[0] == "Status: active":
 		not_allowed_ports = 0
 		for service in get_services():

management/templates/aliases.html

@@ -123,7 +123,7 @@
 <table class="table" style="margin-top: .5em">
 <thead><th>Verb</th> <th>Action</th><th></th></thead>
 <tr><td>GET</td><td><i>(none)</i></td> <td>Returns a list of existing mail aliases. Adding <code>?format=json</code> to the URL will give JSON-encoded results.</td></tr>
-<tr><td>POST</td><td>/add</td> <td>Adds a new mail alias. Required POST-body parameters are <code>address</code> and <code>forward_to</code>.</td></tr>
+<tr><td>POST</td><td>/add</td> <td>Adds a new mail alias. Required POST-body parameters are <code>address</code> and <code>forwards_to</code>.</td></tr>
 <tr><td>POST</td><td>/remove</td> <td>Removes a mail alias. Required POST-body parameter is <code>address</code>.</td></tr>
 </table>
 
@@ -135,7 +135,7 @@
 curl -X GET https://{{hostname}}/admin/mail/aliases?format=json
 
 # Adds a new alias
-curl -X POST -d "address=new_alias@mydomail.com" -d "forward_to=my_email@mydomain.com" https://{{hostname}}/admin/mail/aliases/add
+curl -X POST -d "address=new_alias@mydomail.com" -d "forwards_to=my_email@mydomain.com" https://{{hostname}}/admin/mail/aliases/add
 
 # Removes an alias
 curl -X POST -d "address=new_alias@mydomail.com" https://{{hostname}}/admin/mail/aliases/remove

management/templates/custom-dns.html

@@ -35,8 +35,8 @@
         <option value="AAAA" data-hint="Enter an IPv6 address.">AAAA (IPv6 address)</option>
         <option value="CNAME" data-hint="Enter another domain name followed by a period at the end (e.g. mypage.github.io.).">CNAME (DNS forwarding)</option>
         <option value="TXT" data-hint="Enter arbitrary text.">TXT (text record)</option>
-        <option value="MX" data-hint="Enter record in the form of PRIORIY DOMAIN., including trailing period (e.g. 20 mx.example.com.).">MX (mail exchanger)</option>
-        <option value="SRV" data-hint="Enter record in the form of PRIORIY WEIGHT PORT TARGET., including trailing period (e.g. 10 10 5060 sip.example.com.).">SRV (service record)</option>
+        <option value="MX" data-hint="Enter record in the form of PRIORITY DOMAIN., including trailing period (e.g. 20 mx.example.com.).">MX (mail exchanger)</option>
+        <option value="SRV" data-hint="Enter record in the form of PRIORITY WEIGHT PORT TARGET., including trailing period (e.g. 10 10 5060 sip.example.com.).">SRV (service record)</option>
       </select>
     </div>
   </div>

management/templates/ssl.html

@@ -55,7 +55,7 @@
 
 <h3 id="ssl_install_header">Install certificate</h3>
 
-<p>There are many other places where you can get a free or cheap certificate. If you don't want to use our automatic Let's Encrypt integration, you can give <a href="https://www.namecheap.com/security/ssl-certificates/domain-validation.aspx">Namecheap&rsquo;s $9 certificate</a>, <a href="https://www.startssl.com/">StartSSL&rsquo;s free express lane</a>, <a href="https://buy.wosign.com/free/">WoSign&rsquo;s free TLS</a></a> or any other certificate provider a try.</p>
+<p>If you don't want to use our automatic Let's Encrypt integration, you can give any other certificate provider a try. You can generate the needed CSR below.</p>
 
 <p>Which domain are you getting a certificate for?</p>
 

management/templates/system-backup.html

@@ -16,16 +16,60 @@
       <select class="form-control" rows="1" id="backup-target-type" onchange="toggle_form()">
         <option value="off">Nowhere (Disable Backups)</option>
         <option value="local">{{hostname}}</option>
+        <option value="rsync">rsync</option>
         <option value="s3">Amazon S3</option>
       </select>
     </div>
   </div>
+  <!-- LOCAL BACKUP -->
   <div class="form-group backup-target-local">
     <div class="col-sm-10 col-sm-offset-2">
-      <p>Backups are stored on this machine&rsquo;s own hard disk. You are responsible for periodically using SFTP (FTP over SSH) to copy the backup files from <tt id="backup-location"></tt> to a safe location. These files are encrypted, so they are safe to store anywhere.</p>
+      <p>Backups are stored on this machine&rsquo;s own hard disk. You are responsible for periodically using SFTP (FTP over SSH) to copy the backup files from <tt class="backup-location"></tt> to a safe location. These files are encrypted, so they are safe to store anywhere.</p>
       <p>Separately copy the encryption password from <tt class="backup-encpassword-file"></tt> to a safe and secure location. You will need this file to decrypt backup files.</p>
     </div>
   </div>
+  <!-- RSYNC BACKUP -->
+  <div class="form-group backup-target-rsync">
+    <div class="col-sm-10 col-sm-offset-2">
+
+      <p>Backups synced to a remote machine using rsync over SSH, with local
+      copies in <tt class="backup-location"></tt>. These files are encrypted, so
+      they are safe to store anywhere.</p> <p>Separately copy the encryption
+      password from <tt class="backup-encpassword-file"></tt> to a safe and
+      secure location. You will need this file to decrypt backup files.</p>
+
+    </div>
+  </div>
+  <div class="form-group backup-target-rsync">
+    <label for="backup-target-rsync-host" class="col-sm-2 control-label">Hostname</label>
+    <div class="col-sm-8">
+      <input type="text" placeholder="hostname.local" class="form-control" rows="1" id="backup-target-rsync-host">
+    </div>
+  </div>
+  <div class="form-group backup-target-rsync">
+    <label for="backup-target-rsync-path" class="col-sm-2 control-label">Path</label>
+    <div class="col-sm-8">
+        <input type="text" placeholder="/backups/{{hostname}}" class="form-control" rows="1" id="backup-target-rsync-path">
+    </div>
+  </div>
+  <div class="form-group backup-target-rsync">
+    <label for="backup-target-rsync-user" class="col-sm-2 control-label">Username</label>
+    <div class="col-sm-8">
+      <input type="text" class="form-control" rows="1" id="backup-target-rsync-user">
+    </div>
+  </div>
+  <div class="form-group backup-target-rsync">
+    <label for="ssh-pub-key" class="col-sm-2 control-label">Public SSH Key</label>
+    <div class="col-sm-8">
+      <input type="text" class="form-control" rows="1" id="ssh-pub-key" readonly>
+      <div class="small" style="margin-top: 2px">
+        Copy the Public SSH Key above, and paste it within the <tt>~/.ssh/authorized_keys</tt>
+        of target user on the backup server specified above. That way you'll enable secure and
+        passwordless authentication from your mail-in-a-box server and your backup server.
+      </div>
+    </div>
+  </div>
+  <!-- S3 BACKUP -->
   <div class="form-group backup-target-s3">
     <div class="col-sm-10 col-sm-offset-2">
       <p>Backups are stored in an Amazon Web Services S3 bucket. You must have an AWS account already.</p>
@@ -60,7 +104,8 @@
       <input type="text" class="form-control" rows="1" id="backup-target-pass">
     </div>
   </div>
-  <div class="form-group backup-target-local backup-target-s3">
+  <!-- Common -->
+  <div class="form-group backup-target-local backup-target-rsync backup-target-s3">
     <label for="min-age" class="col-sm-2 control-label">Days:</label>
     <div class="col-sm-8">
       <input type="number" class="form-control" rows="1" id="min-age">
@@ -92,7 +137,7 @@
 
 function toggle_form() {
   var target_type = $("#backup-target-type").val();
-  $(".backup-target-local, .backup-target-s3").hide();
+  $(".backup-target-local, .backup-target-rsync, .backup-target-s3").hide();
   $(".backup-target-" + target_type).show();
 }
 
@@ -160,16 +205,30 @@ function show_system_backup() {
 }
 
 function show_custom_backup() {
-    $(".backup-target-local, .backup-target-s3").hide();
+    $(".backup-target-local, .backup-target-rsync, .backup-target-s3").hide();
     api(
       "/system/backup/config",
       "GET",
       { },
       function(r) {
+        $("#backup-target-user").val(r.target_user);
+        $("#backup-target-pass").val(r.target_pass);
+        $("#min-age").val(r.min_age_in_days);
+        $(".backup-location").text(r.file_target_directory);
+        $(".backup-encpassword-file").text(r.enc_pw_file);
+        $("#ssh-pub-key").val(r.ssh_pub_key);
+
         if (r.target == "file://" + r.file_target_directory) {
           $("#backup-target-type").val("local");
         } else if (r.target == "off") {
           $("#backup-target-type").val("off");
+        } else if (r.target.substring(0, 8) == "rsync://") {
+          $("#backup-target-type").val("rsync");
+          var path = r.target.substring(8).split('//');
+          var host_parts = path.shift().split('@');
+          $("#backup-target-rsync-user").val(host_parts[0]);
+          $("#backup-target-rsync-host").val(host_parts[1]);
+          $("#backup-target-rsync-path").val('/'+path[0]);
         } else if (r.target.substring(0, 5) == "s3://") {
           $("#backup-target-type").val("s3");
           var hostpath = r.target.substring(5).split('/');
@@ -177,11 +236,6 @@ function show_custom_backup() {
           $("#backup-target-s3-host").val(host);
           $("#backup-target-s3-path").val(hostpath.join('/'));
         }
-        $("#backup-target-user").val(r.target_user);
-        $("#backup-target-pass").val(r.target_pass);
-        $("#min-age").val(r.min_age_in_days);
-        $('#backup-location').text(r.file_target_directory);
-        $('.backup-encpassword-file').text(r.enc_pw_file);
         toggle_form()
       })
 }
@@ -196,6 +250,12 @@ function set_custom_backup() {
     target = target_type;
   else if (target_type == "s3")
     target = "s3://" + $("#backup-target-s3-host").val() + "/" + $("#backup-target-s3-path").val();
+  else if (target_type == "rsync") {
+    target = "rsync://" + $("#backup-target-rsync-user").val() + "@" + $("#backup-target-rsync-host").val()
+                                                                + "/" + $("#backup-target-rsync-path").val();
+    target_user = '';
+  }
+
 
   var min_age = $("#min-age").val();
   api(

setup/bootstrap.sh

@@ -7,7 +7,7 @@
 #########################################################
 
 if [ -z "$TAG" ]; then
-	TAG=v0.20
+	TAG=v0.21c
 fi
 
 # Are we running as root?

setup/mail-postfix.sh

@@ -91,7 +91,8 @@ tools/editconf.py /etc/postfix/main.cf \
 # * Give it a different name in syslog to distinguish it from the port 25 smtpd server.
 # * Add a new cleanup service specific to the submission service ('authclean')
 #   that filters out privacy-sensitive headers on mail being sent out by
-#   authenticated users.
+#   authenticated users.  By default Postfix also applies this to attached
+#   emails but we turn this off by setting nested_header_checks empty.
 tools/editconf.py /etc/postfix/master.cf -s -w \
 	"submission=inet n       -       -       -       -       smtpd
 	  -o syslog_name=postfix/submission
@@ -100,7 +101,8 @@ tools/editconf.py /etc/postfix/master.cf -s -w \
 	  -o smtpd_tls_ciphers=high -o smtpd_tls_exclude_ciphers=aNULL,DES,3DES,MD5,DES+MD5,RC4 -o smtpd_tls_mandatory_protocols=!SSLv2,!SSLv3
 	  -o cleanup_service_name=authclean" \
 	"authclean=unix  n       -       -       -       0       cleanup
-	  -o header_checks=pcre:/etc/postfix/outgoing_mail_header_filters"
+	  -o header_checks=pcre:/etc/postfix/outgoing_mail_header_filters
+	  -o nested_header_checks="
 
 # Install the `outgoing_mail_header_filters` file required by the new 'authclean' service.
 cp conf/postfix_outgoing_mail_header_filters /etc/postfix/outgoing_mail_header_filters

setup/management.sh

@@ -4,7 +4,10 @@ source setup/functions.sh
 
 echo "Installing Mail-in-a-Box system management daemon..."
 
-# Install packages.
+# DEPENDENCIES
+
+# Install Python packages that are available from the Ubuntu
+# apt repository:
 # flask, yaml, dnspython, and dateutil are all for our Python 3 management daemon itself.
 # duplicity does backups. python-pip is so we can 'pip install boto' for Python 2, for duplicity, so it can do backups to AWS S3.
 apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-dateutil python-pip
@@ -12,18 +15,46 @@ apt_install python3-flask links duplicity libyaml-dev python3-dnspython python3-
 # These are required to pip install cryptography.
 apt_install build-essential libssl-dev libffi-dev python3-dev
 
+# pip<6.1 + setuptools>=34 have a problem with packages that
+# try to update setuptools during installation, like cryptography.
+# See https://github.com/pypa/pip/issues/4253. The Ubuntu 14.04
+# package versions are pip 1.5.4 and setuptools 3.3. When we
+# install cryptography under those versions, it tries to update
+# setuptools to version 34, which now creates the conflict, and
+# then pip gets permanently broken with errors like
+# "ImportError: No module named 'packaging'".
+#
+# Let's test for the error:
+if ! python3 -c "from pkg_resources import load_entry_point" 2&> /dev/null; then
+	# This system seems to be broken already.
+	echo "Fixing broken pip and setuptools..."
+	rm -rf /usr/local/lib/python3.4/dist-packages/{pkg_resources,setuptools}*
+	apt-get install --reinstall python3-setuptools python3-pip python3-pkg-resources
+fi
+#
+# The easiest work-around on systems that aren't already broken is
+# to upgrade pip (to >=9.0.1) and setuptools (to >=34.1) individually
+# before we install any package that tries to update setuptools.
+hide_output pip3 install --upgrade pip
+hide_output pip3 install --upgrade setuptools
+
 # Install other Python 3 packages used by the management daemon.
 # The first line is the packages that Josh maintains himself!
 # NOTE: email_validator is repeated in setup/questions.sh, so please keep the versions synced.
+# Force acme to be updated because it seems to need it after the
+# pip/setuptools breakage (see above) and the ACME protocol may
+# have changed (I got an error on one of my systems).
 hide_output pip3 install --upgrade \
 	rtyaml "email_validator>=1.0.0" "free_tls_certificates>=0.1.3" \
-	"idna>=2.0.0" "cryptography>=1.0.2" boto psutil
+	"idna>=2.0.0" "cryptography>=1.0.2" acme boto psutil
 
 # duplicity uses python 2 so we need to get the python 2 package of boto to have backups to S3.
 # boto from the Ubuntu package manager is too out-of-date -- it doesn't support the newer
 # S3 api used in some regions, which breaks backups to those regions.  See #627, #653.
 hide_output pip install --upgrade boto
 
+# CONFIGURATION
+
 # Create a backup directory and a random key for encrypting backups.
 mkdir -p $STORAGE_ROOT/backup
 if [ ! -f $STORAGE_ROOT/backup/secret_key.txt ]; then

setup/owncloud.sh

@@ -16,10 +16,6 @@ apt_install \
 
 apt-get purge -qq -y owncloud*
 
-# Install ownCloud from source of this version:
-owncloud_ver=8.2.7
-owncloud_hash=723ba3f46dad219109cdf28dcc016fcd8a6bc434
-
 # Migrate <= v0.10 setups that stored the ownCloud config.php in /usr/local rather than
 # in STORAGE_ROOT. Move the file to STORAGE_ROOT.
 if [ ! -f $STORAGE_ROOT/owncloud/config.php ] \
@@ -32,28 +28,34 @@ if [ ! -f $STORAGE_ROOT/owncloud/config.php ] \
 	ln -sf $STORAGE_ROOT/owncloud/config.php /usr/local/lib/owncloud/config/config.php
 fi
 
-# Check if ownCloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
-if [ ! -d /usr/local/lib/owncloud/ ] \
-	|| ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then
+InstallOwncloud() {
+	echo
+	echo "Upgrading to ownCloud version $1"
+	echo
+
+        version=$1
+        hash=$2
+
+	# Remove the current owncloud
+	rm -rf /usr/local/lib/owncloud
 
 	# Download and verify
-	wget_verify https://download.owncloud.org/community/owncloud-$owncloud_ver.zip $owncloud_hash /tmp/owncloud.zip
-
-	# Clear out the existing ownCloud.
-	if [ -d /usr/local/lib/owncloud/ ]; then
-		echo "upgrading ownCloud to $owncloud_ver (backing up existing ownCloud directory to /tmp/owncloud-backup-$$)..."
-		mv /usr/local/lib/owncloud /tmp/owncloud-backup-$$
-	fi
+	wget_verify https://download.owncloud.org/community/owncloud-$version.zip $hash /tmp/owncloud.zip
 
 	# Extract ownCloud
-	unzip -u -o -q /tmp/owncloud.zip -d /usr/local/lib #either extracts new or replaces current files
+	unzip -q /tmp/owncloud.zip -d /usr/local/lib
 	rm -f /tmp/owncloud.zip
 
-	# The two apps we actually want are not in ownCloud core. Clone them from
+	# The two apps we actually want are not in ownCloud core. Download the releases from
 	# their github repositories.
 	mkdir -p /usr/local/lib/owncloud/apps
-	git_clone https://github.com/owncloudarchive/contacts 9ba2e667ae8c7ea36d8c4a4c3413c374beb24b1b '' /usr/local/lib/owncloud/apps/contacts
-	git_clone https://github.com/owncloudarchive/calendar 2086e738a3b7b868ec59cd61f0f88b49c3f21dd1 '' /usr/local/lib/owncloud/apps/calendar
+	wget_verify https://github.com/owncloud/contacts/releases/download/v1.4.0.0/contacts.tar.gz c1c22d29699456a45db447281682e8bc3f10e3e7 /tmp/contacts.tgz
+	tar xf /tmp/contacts.tgz -C /usr/local/lib/owncloud/apps/
+	rm /tmp/contacts.tgz
+
+        wget_verify https://github.com/nextcloud/calendar/releases/download/v1.4.0/calendar.tar.gz c84f3170efca2a99ea6254de34b0af3cb0b3a821 /tmp/calendar.tgz
+	tar xf /tmp/calendar.tgz -C /usr/local/lib/owncloud/apps/
+	rm /tmp/calendar.tgz
 
 	# Fix weird permissions.
 	chmod 750 /usr/local/lib/owncloud/{apps,config}
@@ -69,7 +71,7 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 
 	# If this isn't a new installation, immediately run the upgrade script.
 	# Then check for success (0=ok and 3=no upgrade needed, both are success).
-	if [ -f $STORAGE_ROOT/owncloud/owncloud.db ]; then
+	if [ -e $STORAGE_ROOT/owncloud/owncloud.db ]; then
 		# ownCloud 8.1.1 broke upgrades. It may fail on the first attempt, but
 		# that can be OK.
 		sudo -u www-data php /usr/local/lib/owncloud/occ upgrade
@@ -81,6 +83,76 @@ if [ ! -d /usr/local/lib/owncloud/ ] \
 			echo "...which seemed to work."
 		fi
 	fi
+}
+
+owncloud_ver=9.1.1
+
+# Check if ownCloud dir exist, and check if version matches owncloud_ver (if either doesn't - install/upgrade)
+if [ ! -d /usr/local/lib/owncloud/ ] \
+        || ! grep -q $owncloud_ver /usr/local/lib/owncloud/version.php; then
+
+	# Stop php-fpm
+	hide_output service php5-fpm stop
+
+        # Backup the existing ownCloud.
+	# Create a backup directory to store the current installation and database to
+	BACKUP_DIRECTORY=$STORAGE_ROOT/owncloud-backup/`date +"%Y-%m-%d-%T"`
+	mkdir -p "$BACKUP_DIRECTORY"
+	if [ -d /usr/local/lib/owncloud/ ]; then
+		echo "upgrading ownCloud to $owncloud_ver (backing up existing ownCloud installation, configuration and database to directory to $BACKUP_DIRECTORY..."
+		cp -r /usr/local/lib/owncloud "$BACKUP_DIRECTORY/owncloud-install"
+	fi
+	if [ -e /home/user-data/owncloud/owncloud.db ]; then
+		cp /home/user-data/owncloud/owncloud.db $BACKUP_DIRECTORY
+        fi
+        if [ -e /home/user-data/owncloud/config.php ]; then
+                cp /home/user-data/owncloud/config.php $BACKUP_DIRECTORY
+        fi
+
+	# We only need to check if we do upgrades when owncloud was previously installed
+	if [ -e /usr/local/lib/owncloud/version.php ]; then
+		if grep -q "8.1.[0-9]" /usr/local/lib/owncloud/version.php; then
+			echo "We are running 8.1.x, upgrading to 8.2.3 first"
+			InstallOwncloud 8.2.3 bfdf6166fbf6fc5438dc358600e7239d1c970613
+		fi
+
+		# If we are upgrading from 8.2.x we should go to 9.0 first. Owncloud doesn't support skipping minor versions
+		if grep -q "8.2.[0-9]" /usr/local/lib/owncloud/version.php; then
+			echo "We are running version 8.2.x, upgrading to 9.0.2 first"
+
+			# We need to disable memcached. The upgrade and install fails
+			# with memcached
+			CONFIG_TEMP=$(/bin/mktemp)
+			php <<EOF > $CONFIG_TEMP && mv $CONFIG_TEMP $STORAGE_ROOT/owncloud/config.php;
+			<?php
+				include("$STORAGE_ROOT/owncloud/config.php");
+
+				\$CONFIG['memcache.local'] = '\OC\Memcache\APC';
+
+				echo "<?php\n\\\$CONFIG = ";
+				var_export(\$CONFIG);
+				echo ";";
+			?>
+EOF
+			chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
+
+			# We can now install owncloud 9.0.2
+			InstallOwncloud 9.0.2 72a3d15d09f58c06fa8bee48b9e60c9cd356f9c5
+
+			# The owncloud 9 migration doesn't migrate calendars and contacts
+			# The option to migrate these are removed in 9.1
+			# So the migrations should be done when we have 9.0 installed
+			sudo -u www-data php /usr/local/lib/owncloud/occ dav:migrate-addressbooks
+			# The following migration has to be done for each owncloud user
+			for directory in $STORAGE_ROOT/owncloud/*@*/ ; do
+				username=$(basename "${directory}")
+				sudo -u www-data php /usr/local/lib/owncloud/occ dav:migrate-calendar $username
+			done
+			sudo -u www-data php /usr/local/lib/owncloud/occ dav:sync-birthday-calendar
+		fi
+	fi
+
+	InstallOwncloud $owncloud_ver 72ed9812432f01b3a459c4afc33f5c76b71eec09
 fi
 
 # ### Configuring ownCloud
@@ -110,10 +182,7 @@ if [ ! -f $STORAGE_ROOT/owncloud/owncloud.db ]; then
       'arguments'=>array('{127.0.0.1:993/imap/ssl/novalidate-cert}')
     )
   ),
-  'memcache.local' => '\\OC\\Memcache\\Memcached',
-  "memcached_servers" => array (
-    array('127.0.0.1', 11211),
-  ),
+  'memcache.local' => '\OC\Memcache\APC',
   'mail_smtpmode' => 'sendmail',
   'mail_smtpsecure' => '',
   'mail_smtpauthtype' => 'LOGIN',
@@ -173,7 +242,7 @@ include("$STORAGE_ROOT/owncloud/config.php");
 
 \$CONFIG['trusted_domains'] = array('$PRIMARY_HOSTNAME');
 
-\$CONFIG['memcache.local'] = '\\OC\\Memcache\\Memcached';
+\$CONFIG['memcache.local'] = '\OC\Memcache\APC';
 \$CONFIG['overwrite.cli.url'] = '/cloud';
 \$CONFIG['mail_from_address'] = 'administrator'; # just the local part, matches our master administrator address
 
@@ -212,6 +281,12 @@ tools/editconf.py /etc/php5/fpm/php.ini -c ';' \
 	max_execution_time=600 \
 	short_open_tag=On
 
+# If apc is explicitly disabled we need to enable it
+if grep -q apc.enabled=0 /etc/php5/mods-available/apcu.ini; then
+	tools/editconf.py /etc/php5/mods-available/apcu.ini -c ';' \
+		apc.enabled=1
+fi
+
 # Set up a cron job for owncloud.
 cat > /etc/cron.hourly/mailinabox-owncloud << EOF;
 #!/bin/bash

setup/preflight.sh

@@ -19,20 +19,26 @@ fi
 
 # Check that we have enough memory.
 #
-# /proc/meminfo reports free memory in kibibytes. Our baseline will be 768 MB,
-# which is 750000 kibibytes.
+# /proc/meminfo reports free memory in kibibytes. Our baseline will be 512 MB,
+# which is 500000 kibibytes.
+#
+# We will display a warning if the memory is below 768 MB which is 750000 kibibytes
 #
 # Skip the check if we appear to be running inside of Vagrant, because that's really just for testing.
 TOTAL_PHYSICAL_MEM=$(head -n 1 /proc/meminfo | awk '{print $2}')
-if [ $TOTAL_PHYSICAL_MEM -lt 750000 ]; then
+if [ $TOTAL_PHYSICAL_MEM -lt 500000 ]; then
 if [ ! -d /vagrant ]; then
 	TOTAL_PHYSICAL_MEM=$(expr \( \( $TOTAL_PHYSICAL_MEM \* 1024 \) / 1000 \) / 1000)
 	echo "Your Mail-in-a-Box needs more memory (RAM) to function properly."
-	echo "Please provision a machine with at least 768 MB, 1 GB recommended."
+	echo "Please provision a machine with at least 512 MB, 1 GB recommended."
 	echo "This machine has $TOTAL_PHYSICAL_MEM MB memory."
 	exit
 fi
 fi
+if [ $TOTAL_PHYSICAL_MEM -lt 750000 ]; then
+	echo "WARNING: Your Mail-in-a-Box has less than 768 MB of memory."
+	echo "         It might run unreliably when under heavy load."
+fi
 
 # Check that tempfs is mounted with exec
 MOUNTED_TMP_AS_NO_EXEC=$(grep "/tmp.*noexec" /proc/mounts)

setup/spamassassin.sh

@@ -84,7 +84,7 @@ tools/editconf.py /etc/spamassassin/local.cf -s \
 
 tools/editconf.py /etc/spamassassin/local.cf -s \
 	bayes_path=$STORAGE_ROOT/mail/spamassassin/bayes \
-	bayes_file_mode=0660
+	bayes_file_mode=0666
 
 mkdir -p $STORAGE_ROOT/mail/spamassassin
 chown -R spampd:spampd $STORAGE_ROOT/mail/spamassassin

setup/system.sh

@@ -119,6 +119,14 @@ apt_install python3 python3-dev python3-pip \
 	haveged pollinate \
 	unattended-upgrades cron ntp fail2ban
 
+# ### Suppress Upgrade Prompts
+# Since Mail-in-a-Box might jump straight to 18.04 LTS, there's no need
+# to be reminded about 16.04 on every login.
+if [ -f /etc/update-manager/release-upgrades ]; then
+	tools/editconf.py /etc/update-manager/release-upgrades Prompt=never
+	rm -f /var/lib/ubuntu-release-upgrader/release-upgrade-available
+fi
+
 # ### Set the system timezone
 #
 # Some systems are missing /etc/timezone, which we cat into the configs for
@@ -208,6 +216,12 @@ pollinate  -q -r
 
 # Between these two, we really ought to be all set.
 
+# We need an ssh key to store backups via rsync, if it doesn't exist create one
+if [ ! -f /root/.ssh/id_rsa_miab ]; then
+	echo 'Creating SSH key for backup…'
+	ssh-keygen -t rsa -b 2048 -a 100 -f /root/.ssh/id_rsa_miab -N '' -q
+fi
+
 # ### Package maintenance
 #
 # Allow apt to install system updates automatically every day.

setup/webmail.sh

@@ -160,11 +160,10 @@ chmod 775 $STORAGE_ROOT/mail
 chown root.www-data $STORAGE_ROOT/mail/users.sqlite 
 chmod 664 $STORAGE_ROOT/mail/users.sqlite 
 
-# Run Roundcube database migration script, if the database exists (it's created by
-# Roundcube on first use).
-if [ -f $STORAGE_ROOT/mail/roundcube/roundcube.sqlite ]; then
+# Run Roundcube database migration script (database is created if it does not exist)
 /usr/local/lib/roundcubemail/bin/updatedb.sh --dir /usr/local/lib/roundcubemail/SQL --package roundcube
-fi
+chown www-data:www-data $STORAGE_ROOT/mail/roundcube/roundcube.sqlite
+chmod 664 $STORAGE_ROOT/mail/roundcube/roundcube.sqlite
 
 # Enable PHP modules.
 php5enmod mcrypt

setup/zpush.sh

@@ -53,6 +53,7 @@ cp conf/zpush/backend_combined.php /usr/local/lib/z-push/backend/combined/config
 # Configure IMAP
 rm -f /usr/local/lib/z-push/backend/imap/config.php
 cp conf/zpush/backend_imap.php /usr/local/lib/z-push/backend/imap/config.php
+sed -i "s%STORAGE_ROOT%$STORAGE_ROOT%" /usr/local/lib/z-push/backend/imap/config.php
 
 # Configure CardDav
 rm -f /usr/local/lib/z-push/backend/carddav/config.php

tests/fail2ban.py

@@ -10,11 +10,11 @@ import sys, os, time, functools
 
 # parse command line
 
-if len(sys.argv) != 3:
-	print("Usage: tests/fail2ban.py \"ssh user@hostname\" hostname")
+if len(sys.argv) != 4:
+	print("Usage: tests/fail2ban.py \"ssh user@hostname\" hostname owncloud_user")
 	sys.exit(1)
 
-ssh_command, hostname = sys.argv[1:3]
+ssh_command, hostname, owncloud_user = sys.argv[1:4]
 
 # define some test types
 
@@ -215,7 +215,7 @@ if __name__ == "__main__":
 	run_test(http_test, ["/admin/munin/", 401], 20, 30, 1)
 
 	# ownCloud
-	run_test(http_test, ["/cloud/remote.php/webdav", 401, None, None, ["aa", "aa"]], 20, 120, 1)
+	run_test(http_test, ["/cloud/remote.php/webdav", 401, None, None, [owncloud_user, "aa"]], 20, 120, 1)
 
 	# restart fail2ban so that this client machine is no longer blocked
 	restart_fail2ban_service(final=True)

tools/owncloud-restore.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+#
+# This script will restore the backup made during an installation
+source /etc/mailinabox.conf # load global vars
+
+if [ -z "$1" ]; then
+	echo "Usage: owncloud-restore.sh <backup directory>"
+	echo
+	echo "WARNING: This will restore the database to the point of the installation!"
+	echo "         This means that you will lose all changes made by users after that point"
+	echo
+	echo
+	echo "Backups are stored here: $STORAGE_ROOT/owncloud-backup/"
+	echo
+	echo "Available backups:"
+	echo
+	find $STORAGE_ROOT/owncloud-backup/* -maxdepth 0 -type d
+	echo
+	echo "Supply the directory that was created during the last installation as the only commandline argument"
+	exit
+fi
+
+if [ ! -f $1/config.php ]; then
+	echo "This isn't a valid backup location"
+	exit
+fi
+
+echo "Restoring backup from $1"
+service php5-fpm stop
+
+# remove the current owncloud installation
+rm -rf /usr/local/lib/owncloud/
+# restore the current owncloud application
+cp -r  "$1/owncloud-install" /usr/local/lib/owncloud
+
+# restore access rights
+chmod 750 /usr/local/lib/owncloud/{apps,config}
+
+cp "$1/owncloud.db" $STORAGE_ROOT/owncloud/
+cp "$1/config.php" $STORAGE_ROOT/owncloud/
+
+ln -sf $STORAGE_ROOT/owncloud/config.php /usr/local/lib/owncloud/config/config.php
+chown -f -R www-data.www-data $STORAGE_ROOT/owncloud /usr/local/lib/owncloud
+chown www-data.www-data $STORAGE_ROOT/owncloud/config.php
+
+sudo -u www-data php /usr/local/lib/owncloud/occ maintenance:mode --off
+
+service php5-fpm start
+echo "Done"