diff --git a/CHANGELOG.md b/CHANGELOG.md index bd1745a8..a7d548ff 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,35 @@ CHANGELOG ========= +v0.52 (January 31, 2021) +------------------------ + +Software updates: + +* Upgraded Roundcube to version 1.4.10. +* Upgraded zpush to 2.6.1. + +Mail: + +* Incoming emails with SPF/DKIM/DMARC failures now get a higher spam score, and these messages are more likely to appear in the junk folder, since they are often spam/phishing. +* Fixed the MTA-STS policy file's line endings. + +Control panel: + +* A new Download button in the control panel's External DNS page can be used to download the required DNS records in zonefile format. +* Fixed the problem when the control panel would report DNS entries as Not Set by increasing a bind query limit. +* Fixed a control panel startup bug on some systems. +* Improved an error message on a DNS lookup timeout. +* A typo was fixed. + +DNS: + +* The TTL for NS records has been increased to 1 day to comply with some registrar requirements. + +System: + +* Nextcloud's photos, dashboard, and activity apps are disabled since we only support contacts and calendar. + v0.51 (November 14, 2020) ------------------------- @@ -13,7 +42,7 @@ Mail: * The MTA-STA max_age value was increased to the normal one week. -Control Panel: +Control panel: * Two-factor authentication can now be enabled for logins to the control panel. However, keep in mind that many online services (including domain name registrars, cloud server providers, and TLS certificate providers) may allow an attacker to take over your account or issue a fraudulent TLS certificate with only access to your email address, and this new two-factor authentication does not protect access to your inbox. It therefore remains very important that user accounts with administrative email addresses have strong passwords. * TLS certificate expiry dates are now shown in ISO8601 format for clarity. @@ -39,7 +68,7 @@ TLS: * TLS certificates are now provisioned in groups by parent domain to limit easy domain enumeration and make provisioning more resilient to errors for particular domains. -Control Panel: +Control panel: * The control panel API is now fully documented at https://mailinabox.email/api-docs.html. * User passwords can now have spaces. diff --git a/README.md b/README.md index 07b9cf58..d907ba95 100644 --- a/README.md +++ b/README.md @@ -135,36 +135,18 @@ See the [setup guide](https://mailinabox.email/guide.html) for detailed, user-fr For experts, start with a completely fresh (really, I mean it) Ubuntu 18.04 LTS 64-bit machine. On the machine... -Clone this repository: +Clone this repository and checkout the tag corresponding to the most recent release: $ git clone https://github.com/mail-in-a-box/mailinabox $ cd mailinabox - -_Optional:_ Download Josh's PGP key and then verify that the sources were signed -by him: - - $ curl -s https://keybase.io/joshdata/key.asc | gpg --import - gpg: key C10BDD81: public key "Joshua Tauberer " imported - - $ git verify-tag v0.51 - gpg: Signature made ..... using RSA key ID C10BDD81 - gpg: Good signature from "Joshua Tauberer " - gpg: WARNING: This key is not certified with a trusted signature! - gpg: There is no indication that the signature belongs to the owner. - Primary key fingerprint: 5F4C 0E73 13CC D744 693B 2AEA B920 41F4 C10B DD81 - -You'll get a lot of warnings, but that's OK. Check that the primary key fingerprint matches the -fingerprint in the key details at [https://keybase.io/joshdata](https://keybase.io/joshdata) -and on his [personal homepage](https://razor.occams.info/). (Of course, if this repository has been compromised you can't trust these instructions.) - -Checkout the tag corresponding to the most recent release: - - $ git checkout v0.51 + $ git checkout v0.52 Begin the installation. $ sudo setup/start.sh +The installation will install, uninstall, and configure packages to turn the machine into a working, good mail server. + For help, DO NOT contact Josh directly --- I don't do tech support by email or tweet (no exceptions). Post your question on the [discussion forum](https://discourse.mailinabox.email/) instead, where maintainers and Mail-in-a-Box users may be able to help you. @@ -172,6 +154,7 @@ Post your question on the [discussion forum](https://discourse.mailinabox.email/ Note that while we want everything to "just work," we can't control the rest of the Internet. Other mail services might block or spam-filter email sent from your Mail-in-a-Box. This is a challenge faced by everyone who runs their own mail server, with or without Mail-in-a-Box. See our discussion forum for tips about that. + Contributing and Development ---------------------------- diff --git a/api/mailinabox.yml b/api/mailinabox.yml index 7bdadea1..f5453f6f 100644 --- a/api/mailinabox.yml +++ b/api/mailinabox.yml @@ -15,7 +15,7 @@ info: license: name: CC0 1.0 Universal url: https://creativecommons.org/publicdomain/zero/1.0/legalcode - version: 0.47.0 + version: 0.51.0 x-logo: url: https://mailinabox.email/static/logo.png altText: Mail-in-a-Box logo @@ -860,6 +860,38 @@ paths: text/html: schema: type: string + /dns/zonefile/{zone}: + parameters: + - in: path + name: zone + schema: + $ref: '#/components/schemas/Hostname' + required: true + description: Hostname + get: + tags: + - DNS + summary: Get DNS zonefile + description: Returns a DNS zone file for a hostname. + operationId: getDnsZonefile + x-codeSamples: + - lang: curl + source: | + curl -X GET "https://{host}/admin/dns/zonefile/" \ + -u ":" + responses: + 200: + description: Successful operation + content: + application/json: + schema: + $ref: '#/components/schemas/DNSZonefileResponse' + 403: + description: Forbidden + content: + text/html: + schema: + type: string /dns/update: post: tags: @@ -1898,7 +1930,7 @@ components: text/plain: schema: type: string - example: 1.2.3.4 + example: '1.2.3.4' description: The value of the DNS record. example: '1.2.3.4' schemas: @@ -2167,6 +2199,8 @@ components: items: $ref: '#/components/schemas/Hostname' description: DNS zones response. + DNSZonefileResponse: + type: string DNSSecondaryNameserverResponse: type: object required: @@ -2839,13 +2873,6 @@ components: type: string MfaEnableSuccessResponse: type: string - MfaEnableBadRequestResponse: - type: object - required: - - error - properties: - error: - type: string MfaDisableRequest: type: object properties: diff --git a/conf/mta-sts.txt b/conf/mta-sts.txt index 26acc015..e7bdc4c4 100644 --- a/conf/mta-sts.txt +++ b/conf/mta-sts.txt @@ -1,4 +1,4 @@ -version: STSv1 -mode: MODE -mx: PRIMARY_HOSTNAME -max_age: 604800 +version: STSv1 +mode: MODE +mx: PRIMARY_HOSTNAME +max_age: 604800 diff --git a/management/backup.py b/management/backup.py index ce53353c..f5773cab 100755 --- a/management/backup.py +++ b/management/backup.py @@ -470,6 +470,23 @@ def list_target_files(config): raise ValueError(e.reason) return [(key.name[len(path):], key.size) for key in bucket.list(prefix=path)] + elif target.scheme == 'b2': + from b2sdk.v1 import InMemoryAccountInfo, B2Api + from b2sdk.v1.exception import NonExistentBucket + info = InMemoryAccountInfo() + b2_api = B2Api(info) + + # Extract information from target + b2_application_keyid = target.netloc[:target.netloc.index(':')] + b2_application_key = target.netloc[target.netloc.index(':')+1:target.netloc.index('@')] + b2_bucket = target.netloc[target.netloc.index('@')+1:] + + try: + b2_api.authorize_account("production", b2_application_keyid, b2_application_key) + bucket = b2_api.get_bucket_by_name(b2_bucket) + except NonExistentBucket as e: + raise ValueError("B2 Bucket does not exist. Please double check your information!") + return [(key.file_name, key.size) for key, _ in bucket.ls()] else: raise ValueError(config["target"]) diff --git a/management/daemon.py b/management/daemon.py index 73fd8f14..d08dfdb3 100755 --- a/management/daemon.py +++ b/management/daemon.py @@ -1,3 +1,12 @@ +#!/usr/local/lib/mailinabox/env/bin/python3 +# +# During development, you can start the Mail-in-a-Box control panel +# by running this script, e.g.: +# +# service mailinabox stop # stop the system process +# DEBUG=1 management/daemon.py +# service mailinabox start # when done debugging, start it up again + import os, os.path, re, json, time import multiprocessing.pool, subprocess @@ -341,6 +350,12 @@ def dns_get_dump(): from dns_update import build_recommended_dns return json_response(build_recommended_dns(env)) +@app.route('/dns/zonefile/') +@authorized_personnel_only +def dns_get_zonefile(zone): + from dns_update import get_dns_zonefile + return Response(get_dns_zonefile(zone, env), status=200, mimetype='text/plain') + # SSL @app.route('/ssl/status') @@ -734,7 +749,22 @@ def log_failed_login(request): # APP if __name__ == '__main__': - if "DEBUG" in os.environ: app.debug = True + if "DEBUG" in os.environ: + # Turn on Flask debugging. + app.debug = True + + # Use a stable-ish master API key so that login sessions don't restart on each run. + # Use /etc/machine-id to seed the key with a stable secret, but add something + # and hash it to prevent possibly exposing the machine id, using the time so that + # the key is not valid indefinitely. + import hashlib + with open("/etc/machine-id") as f: + api_key = f.read() + api_key += "|" + str(int(time.time() / (60*60*2))) + hasher = hashlib.sha1() + hasher.update(api_key.encode("ascii")) + auth_service.key = hasher.hexdigest() + if "APIKEY" in os.environ: auth_service.key = os.environ["APIKEY"] if not app.debug: diff --git a/management/dns_update.py b/management/dns_update.py index 46fb4edf..6280414f 100755 --- a/management/dns_update.py +++ b/management/dns_update.py @@ -470,14 +470,14 @@ def write_nsd_zone(domain, zonefile, records, env, force): zone = """ $ORIGIN {domain}. -$TTL 1800 ; default time to live +$TTL 86400 ; default time to live @ IN SOA ns1.{primary_domain}. hostmaster.{primary_domain}. ( __SERIAL__ ; serial number 7200 ; Refresh (secondary nameserver update interval) - 1800 ; Retry (when refresh fails, how often to try again) + 86400 ; Retry (when refresh fails, how often to try again) 1209600 ; Expire (when refresh fails, how long secondary nameserver will keep records around anyway) - 1800 ; Negative TTL (how long negative responses are cached) + 86400 ; Negative TTL (how long negative responses are cached) ) """ @@ -564,6 +564,17 @@ $TTL 1800 ; default time to live return True # file is updated +def get_dns_zonefile(zone, env): + for domain, fn in get_dns_zones(env): + if zone == domain: + break + else: + raise ValueError("%s is not a domain name that corresponds to a zone." % zone) + + nsd_zonefile = "/etc/nsd/zones/" + fn + with open(nsd_zonefile, "r") as f: + return f.read() + ######################################################################## def write_nsd_conf(zonefiles, additional_records, env): diff --git a/management/status_checks.py b/management/status_checks.py index 6e0cdb44..d3fb7ae5 100755 --- a/management/status_checks.py +++ b/management/status_checks.py @@ -293,6 +293,8 @@ def run_network_checks(env, output): zen = query_dns(rev_ip4+'.zen.spamhaus.org', 'A', nxdomain=None) if zen is None: output.print_ok("IP address is not blacklisted by zen.spamhaus.org.") + elif zen == "[timeout]": + output.print_warning("Connection to zen.spamhaus.org timed out. We could not determine whether your server's IP address is blacklisted. Please try again later.") else: output.print_error("""The IP address of this machine %s is listed in the Spamhaus Block List (code %s), which may prevent recipients from receiving your email. See http://www.spamhaus.org/query/ip/%s.""" @@ -691,6 +693,8 @@ def check_mail_domain(domain, env, output): dbl = query_dns(domain+'.dbl.spamhaus.org', "A", nxdomain=None) if dbl is None: output.print_ok("Domain is not blacklisted by dbl.spamhaus.org.") + elif dbl == "[timeout]": + output.print_warning("Connection to dbl.spamhaus.org timed out. We could not determine whether the domain {} is blacklisted. Please try again later.".format(domain)) else: output.print_error("""This domain is listed in the Spamhaus Domain Block List (code %s), which may prevent recipients from receiving your mail. diff --git a/management/templates/custom-dns.html b/management/templates/custom-dns.html index 6984b081..b1b98b9b 100644 --- a/management/templates/custom-dns.html +++ b/management/templates/custom-dns.html @@ -89,7 +89,7 @@

- Multiple secondary servers can be separated with commas or spaces (i.e., ns2.hostingcompany.com ns3.hostingcompany.com). + Multiple secondary servers can be separated with commas or spaces (i.e., ns2.hostingcompany.com ns3.hostingcompany.com). To enable zone transfers to additional servers without listing them as secondary nameservers, add an IP address or subnet using xfr:10.20.30.40 or xfr:10.0.0.0/8.

+

Download zonefile

+

You can download your zonefiles here or use the table of records below.

+
+
+
+ + +
+ +
+
+ +

Records

@@ -57,6 +70,18 @@ diff --git a/management/templates/system-backup.html b/management/templates/system-backup.html index d05d8c9c..57e06d13 100644 --- a/management/templates/system-backup.html +++ b/management/templates/system-backup.html @@ -111,6 +111,31 @@ + +
+
+

Backups are stored in a Backblaze B2 bucket. You must have a Backblaze account already.

+

You MUST manually copy the encryption password from to a safe and secure location. You will need this file to decrypt backup files. It is NOT stored in your Backblaze B2 bucket.

+
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
@@ -149,7 +174,7 @@ function toggle_form() { var target_type = $("#backup-target-type").val(); - $(".backup-target-local, .backup-target-rsync, .backup-target-s3").hide(); + $(".backup-target-local, .backup-target-rsync, .backup-target-s3, .backup-target-b2").hide(); $(".backup-target-" + target_type).show(); init_inputs(target_type); @@ -225,7 +250,7 @@ function show_system_backup() { } function show_custom_backup() { - $(".backup-target-local, .backup-target-rsync, .backup-target-s3").hide(); + $(".backup-target-local, .backup-target-rsync, .backup-target-s3, .backup-target-b2").hide(); api( "/system/backup/config", "GET", @@ -255,6 +280,15 @@ function show_custom_backup() { var host = hostpath.shift(); $("#backup-target-s3-host").val(host); $("#backup-target-s3-path").val(hostpath.join('/')); + } else if (r.target.substring(0, 5) == "b2://") { + $("#backup-target-type").val("b2"); + var targetPath = r.target.substring(5); + var b2_application_keyid = targetPath.split(':')[0]; + var b2_applicationkey = targetPath.split(':')[1].split('@')[0]; + var b2_bucket = targetPath.split('@')[1]; + $("#backup-target-b2-user").val(b2_application_keyid); + $("#backup-target-b2-pass").val(b2_applicationkey); + $("#backup-target-b2-bucket").val(b2_bucket); } toggle_form() }) @@ -274,6 +308,11 @@ function set_custom_backup() { target = "rsync://" + $("#backup-target-rsync-user").val() + "@" + $("#backup-target-rsync-host").val() + "/" + $("#backup-target-rsync-path").val(); target_user = ''; + } else if (target_type == "b2") { + target = 'b2://' + $('#backup-target-b2-user').val() + ':' + $('#backup-target-b2-pass').val() + + '@' + $('#backup-target-b2-bucket').val() + target_user = ''; + target_pass = ''; } diff --git a/management/templates/users.html b/management/templates/users.html index 78fef61a..4b349875 100644 --- a/management/templates/users.html +++ b/management/templates/users.html @@ -31,7 +31,7 @@
    -
  • Passwords must be at least eight characters consisting of English lettters and numbers only. For best results, generate a random password.
    • +
  • Passwords must be at least eight characters consisting of English letters and numbers only. For best results, generate a random password.
  • Use aliases to create email addresses that forward to existing accounts.
  • Administrators get access to this control panel.
  • User accounts cannot contain any international (non-ASCII) characters, but aliases can.
    • diff --git a/setup/dkim.sh b/setup/dkim.sh index a2c67857..62d242c3 100755 --- a/setup/dkim.sh +++ b/setup/dkim.sh @@ -64,6 +64,32 @@ management/editconf.py /etc/opendmarc.conf -s \ "Syslog=true" \ "Socket=inet:8893@[127.0.0.1]" +# SPFIgnoreResults causes the filter to ignore any SPF results in the header +# of the message. This is useful if you want the filter to perfrom SPF checks +# itself, or because you don't trust the arriving header. This added header is +# used by spamassassin to evaluate the mail for spamminess. + +management/editconf.py /etc/opendmarc.conf -s \ + "SPFIgnoreResults=true" + +# SPFSelfValidate causes the filter to perform a fallback SPF check itself +# when it can find no SPF results in the message header. If SPFIgnoreResults +# is also set, it never looks for SPF results in headers and always performs +# the SPF check itself when this is set. This added header is used by +# spamassassin to evaluate the mail for spamminess. + +management/editconf.py /etc/opendmarc.conf -s \ + "SPFSelfValidate=true" + +# AlwaysAddARHeader Adds an "Authentication-Results:" header field even to +# unsigned messages from domains with no "signs all" policy. The reported DKIM +# result will be "none" in such cases. Normally unsigned mail from non-strict +# domains does not cause the results header field to be added. This added header +# is used by spamassassin to evaluate the mail for spamminess. + +management/editconf.py /etc/opendkim.conf -s \ + "AlwaysAddARHeader=true" + # Add OpenDKIM and OpenDMARC as milters to postfix, which is how OpenDKIM # intercepts outgoing mail to perform the signing (by adding a mail header) # and how they both intercept incoming mail to add Authentication-Results diff --git a/setup/management.sh b/setup/management.sh index 63590a96..0ec77579 100755 --- a/setup/management.sh +++ b/setup/management.sh @@ -18,11 +18,7 @@ while [ -d /usr/local/lib/python3.4/dist-packages/acme ]; do pip3 uninstall -y acme; done -# duplicity is used to make backups of user data. It uses boto -# (via Python 2) to do backups to AWS S3. boto from the Ubuntu -# package manager is too out-of-date -- it doesn't support the newer -# S3 api used in some regions, which breaks backups to those regions. -# See #627, #653. +# duplicity is used to make backups of user data. # # virtualenv is used to isolate the Python 3 packages we # install via pip from the system-installed packages. @@ -30,6 +26,9 @@ done # certbot installs EFF's certbot which we use to # provision free TLS certificates. apt_install duplicity python3-pip virtualenv certbot + +# boto is used for amazon aws backups. +# Both are installed outside the pipenv, so they can be used by duplicity hide_output pip3 install --upgrade boto # Create a virtualenv for the installation of Python 3 packages @@ -50,7 +49,7 @@ hide_output $venv/bin/pip install --upgrade pip hide_output $venv/bin/pip install --upgrade \ rtyaml "email_validator>=1.0.0" "exclusiveprocess" \ flask dnspython python-dateutil \ - qrcode[pil] pyotp \ + qrcode[pil] pyotp \ "idna>=2.0.0" "cryptography==2.2.2" boto psutil postfix-mta-sts-resolver # CONFIGURATION @@ -100,6 +99,12 @@ rm -f /tmp/fontawesome.zip # running after a reboot. cat > $inst_dir/start < /etc/spamassassin/miab_spf_dmarc.cf << EOF +# Evaluate DMARC Authentication-Results +header DMARC_PASS Authentication-Results =~ /$escapedprimaryhostname; dmarc=pass/ +describe DMARC_PASS DMARC check passed +score DMARC_PASS -0.1 + +header DMARC_NONE Authentication-Results =~ /$escapedprimaryhostname; dmarc=none/ +describe DMARC_NONE DMARC record not found +score DMARC_NONE 0.1 + +header DMARC_FAIL_NONE Authentication-Results =~ /$escapedprimaryhostname; dmarc=fail \(p=none/ +describe DMARC_FAIL_NONE DMARC check failed (p=none) +score DMARC_FAIL_NONE 2.0 + +header DMARC_FAIL_QUARANTINE Authentication-Results =~ /$escapedprimaryhostname; dmarc=fail \(p=quarantine/ +describe DMARC_FAIL_QUARANTINE DMARC check failed (p=quarantine) +score DMARC_FAIL_QUARANTINE 5.0 + +header DMARC_FAIL_REJECT Authentication-Results =~ /$escapedprimaryhostname; dmarc=fail \(p=reject/ +describe DMARC_FAIL_REJECT DMARC check failed (p=reject) +score DMARC_FAIL_REJECT 10.0 + +# Evaluate SPF Authentication-Results +header SPF_PASS Authentication-Results =~ /$escapedprimaryhostname; spf=pass/ +describe SPF_PASS SPF check passed +score SPF_PASS -0.1 + +header SPF_NONE Authentication-Results =~ /$escapedprimaryhostname; spf=none/ +describe SPF_NONE SPF record not found +score SPF_NONE 2.0 + +header SPF_FAIL Authentication-Results =~ /$escapedprimaryhostname; spf=fail/ +describe SPF_FAIL SPF check failed +score SPF_FAIL 5.0 +EOF + # Bayesean learning # ----------------- # diff --git a/setup/system.sh b/setup/system.sh index 34775059..637c8a51 100755 --- a/setup/system.sh +++ b/setup/system.sh @@ -301,6 +301,9 @@ fi #NODOC # name server, on IPV6. # * The listen-on directive in named.conf.options restricts `bind9` to # binding to the loopback interface instead of all interfaces. +# * The max-recursion-queries directive increases the maximum number of iterative queries. +# If more queries than specified are sent, bind9 returns SERVFAIL. After flushing the cache during system checks, +# we ran into the limit thus we are increasing it from 75 (default value) to 100. apt_install bind9 touch /etc/default/bind9 management/editconf.py /etc/default/bind9 \ @@ -309,6 +312,10 @@ if ! grep -q "listen-on " /etc/bind/named.conf.options; then # Add a listen-on directive if it doesn't exist inside the options block. sed -i "s/^}/\n\tlisten-on { 127.0.0.1; };\n}/" /etc/bind/named.conf.options fi +if ! grep -q "max-recursion-queries " /etc/bind/named.conf.options; then + # Add a max-recursion-queries directive if it doesn't exist inside the options block. + sed -i "s/^}/\n\tmax-recursion-queries 100;\n}/" /etc/bind/named.conf.options +fi # First we'll disable systemd-resolved's management of resolv.conf and its stub server. # Breaking the symlink to /run/systemd/resolve/stub-resolv.conf means diff --git a/setup/webmail.sh b/setup/webmail.sh index 332e1d3c..bf603c56 100755 --- a/setup/webmail.sh +++ b/setup/webmail.sh @@ -28,8 +28,8 @@ apt_install \ # Install Roundcube from source if it is not already present or if it is out of date. # Combine the Roundcube version number with the commit hash of plugins to track # whether we have the latest version of everything. -VERSION=1.4.9 -HASH=df650f4d3eae9eaae2d5a5f06d68665691daf57d +VERSION=1.4.10 +HASH=36b2351030e1ebddb8e39190d7b0ba82b1bbec1b PERSISTENT_LOGIN_VERSION=6b3fc450cae23ccb2f393d0ef67aa319e877e435 HTML5_NOTIFIER_VERSION=4b370e3cd60dabd2f428a26f45b677ad1b7118d5 CARDDAV_VERSION=3.0.3 diff --git a/setup/zpush.sh b/setup/zpush.sh index 51460115..cec1c196 100755 --- a/setup/zpush.sh +++ b/setup/zpush.sh @@ -22,8 +22,8 @@ apt_install \ phpenmod -v php imap # Copy Z-Push into place. -VERSION=2.5.2 -TARGETHASH=2dc3dbd791b96b0ba2638df0d3d1e03c7e1cbab2 +VERSION=2.6.1 +TARGETHASH=a4415f0dc0ed884acc8ad5c506944fc7e6d68eeb needs_update=0 #NODOC if [ ! -f /usr/local/lib/z-push/version ]; then needs_update=1 #NODOC