Merge branch 'master' into management-memory-usage

This commit is contained in:
yodax 2015-12-26 20:28:22 +01:00
commit 03606a27d4
15 changed files with 149 additions and 102 deletions

View File

@ -1,13 +1,14 @@
CHANGELOG
=========
Still In Development
--------------------
v0.15 Release Candidate
-----------------------
Mail:
* Updated Roundcube to version 1.1.3.
* Auto-create RFC2142 aliases for abuse@.
* Auto-create aliases for abuse@, as required by RFC2142.
* The DANE TLSA record is changed to use the certificate subject public key rather than the whole certificate, which means the record remains valid after certificate changes (so long as the private key remains the same, which it does for us).
Control panel:
@ -16,16 +17,22 @@ Control panel:
* DNS checks now have a timeout in case a DNS server is not responding, so the checks don't stall indefinitely.
* Better messages if external DNS is used and, weirdly, custom secondary nameservers are set.
* Add POP to the mail client settings documentation.
* The box's IP address is added to the fail2ban whitelist so that the status checks don't trigger the machine banning itself, which results in the status checks showing services down even though they are running.
* For SSL certificates, rather than asking you what country you are in during setup, ask at the time a CSR is generated. The default system self-signed certificate now omits a country in the subject (it was never needed). The CSR_COUNTRY Mail-in-a-Box setting is dropped entirely.
System:
* Nightly backups and system status checks are now moved to 3am in the system's timezone.
* fail2ban's recidive jail is now active, which guards against persistent brute force login attacks over long periods of time.
* Setup (first run only) now asks for your timezone to set the system time.
* The Exchange/ActiveSync server is now taken offline during nightly backups (along with SMTP and IMAP).
* The machine's random number generator (/dev/urandom) is now seeded with Ubuntu Pollinate and a blocking read on /dev/random.
* DNSSEC key generation during install now uses /dev/urandom (instead of /dev/random), which is faster.
* The $STORAGE_ROOT/ssl directory is flattened by a migration script and the system SSL certificate path is now a symlink to the actual certificate.
* If ownCloud sends out email, it will use the box's administrative address now (admin@yourboxname).
* Z-Push (Exchange/ActiveSync) logs now exclude warnings and are now rotated to save disk space.
* Fix pip command that might have not installed all necessary Python packages.
* The control panel and backup would not work on Google Compute Engine because they install a conflicting boto package.
* The control panel and backup would not work on Google Compute Engine because GCE installs a conflicting boto package.
* Added a new command `management/backup.py --restore` to restore files from a backup to a target directory (command line arguments are passed to `duplicity restore`).
v0.14 (November 4, 2015)

1
Vagrantfile vendored
View File

@ -22,7 +22,6 @@ Vagrant.configure("2") do |config|
export PUBLIC_IP=auto
export PUBLIC_IPV6=auto
export PRIMARY_HOSTNAME=auto-easy
export CSR_COUNTRY=US
#export SKIP_NETWORK_CHECKS=1
# Start the setup script.

View File

@ -1,27 +1,28 @@
# This list is derived from https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2.
# The columns are ISO_3166-1_alpha-2 code, display name, Wikipedia page name.
# The top 20 countries by number of Internet users are grouped first, see
# The top 21 countries by number of Internet users are grouped first, see
# https://en.wikipedia.org/wiki/List_of_countries_by_number_of_Internet_users.
BR Brazil
CA Canada
CN China
EG Egypt
FR France
DE Germany
IN India
ID Indonesia
IT Italy
JP Japan
MX Mexico
NG Nigeria
PH Philippines
RU Russian Federation Russia
ES Spain
KR South Korea
TR Turkey
GB United Kingdom
US United States
JP Japan
BR Brazil
RU Russian Federation Russia
DE Germany
NG Nigeria
GB United Kingdom
FR France
MX Mexico
EG Egypt
KR South Korea
VN Vietnam
ID Indonesia
PH Philippines
TR Turkey
IT Italy
PK Pakistan
ES Spain
CA Canada
AD Andorra
AE United Arab Emirates
AF Afghanistan
@ -183,7 +184,6 @@ PA Panama
PE Peru
PF French Polynesia
PG Papua New Guinea
PK Pakistan
PL Poland
PM Saint Pierre and Miquelon
PN Pitcairn Pitcairn Islands
Can't render this file because it has a wrong number of fields in line 5.

View File

@ -28,6 +28,14 @@ try:
except OSError:
pass
# for generating CSRs we need a list of country codes
csr_country_codes = []
with open(os.path.join(os.path.dirname(me), "csr_country_codes.tsv")) as f:
for line in f:
if line.strip() == "" or line.startswith("#"): continue
code, name = line.strip().split("\t")[0:2]
csr_country_codes.append((code, name))
app = Flask(__name__, template_folder=os.path.abspath(os.path.join(os.path.dirname(me), "templates")))
# Decorator to protect views that require a user with 'admin' privileges.
@ -101,9 +109,12 @@ def index():
return render_template('index.html',
hostname=env['PRIMARY_HOSTNAME'],
storage_root=env['STORAGE_ROOT'],
no_users_exist=no_users_exist,
no_admins_exist=no_admins_exist,
backup_s3_hosts=backup_s3_hosts,
csr_country_codes=csr_country_codes,
)
@app.route('/me')
@ -321,7 +332,7 @@ def dns_get_dump():
def ssl_get_csr(domain):
from ssl_certificates import create_csr
ssl_private_key = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_private_key.pem'))
return create_csr(domain, ssl_private_key, env)
return create_csr(domain, ssl_private_key, request.form.get('countrycode', ''), env)
@app.route('/ssl/install', methods=['POST'])
@authorized_personnel_only

8
management/daily_tasks.sh Executable file
View File

@ -0,0 +1,8 @@
#!/bin/bash
# This script is run daily (at 3am each night).
# Take a backup.
management/backup.py
# Run status checks and email the administrator if anything changed.
management/status_checks.py --show-changes --smtp

View File

@ -283,26 +283,40 @@ def build_zone(domain, all_domains, additional_records, www_redirect_domains, en
def build_tlsa_record(env):
# A DANE TLSA record in DNS specifies that connections on a port
# must use TLS and the certificate must match a particular certificate.
# must use TLS and the certificate must match a particular criteria.
#
# Thanks to http://blog.huque.com/2012/10/dnssec-and-certificates.html
# for explaining all of this!
# and https://community.letsencrypt.org/t/please-avoid-3-0-1-and-3-0-2-dane-tlsa-records-with-le-certificates/7022
# for explaining all of this! Also see https://tools.ietf.org/html/rfc6698#section-2.1
# and https://github.com/mail-in-a-box/mailinabox/issues/268#issuecomment-167160243.
#
# There are several criteria. We used to use "3 0 1" criteria, which
# meant to pin a leaf (3) certificate (0) with SHA256 hash (1). But
# certificates change, and especially as we move to short-lived certs
# they change often. The TLSA record handily supports the criteria of
# a leaf certificate (3)'s subject public key (1) with SHA256 hash (1).
# The subject public key is the public key portion of the private key
# that generated the CSR that generated the certificate. Since we
# generate a private key once the first time Mail-in-a-Box is set up
# and reuse it for all subsequent certificates, the TLSA record will
# remain valid indefinitely.
# Get the hex SHA256 of the DER-encoded server certificate:
certder = shell("check_output", [
"/usr/bin/openssl",
"x509",
"-in", os.path.join(env["STORAGE_ROOT"], "ssl", "ssl_certificate.pem"),
"-outform", "DER"
],
return_bytes=True)
certhash = hashlib.sha256(certder).hexdigest()
from ssl_certificates import load_cert_chain, load_pem
from cryptography.hazmat.primitives.serialization import Encoding, PublicFormat
fn = os.path.join(env["STORAGE_ROOT"], "ssl", "ssl_certificate.pem")
cert = load_pem(load_cert_chain(fn)[0])
subject_public_key = cert.public_key().public_bytes(Encoding.DER, PublicFormat.SubjectPublicKeyInfo)
# We could have also loaded ssl_private_key.pem and called priv_key.public_key().public_bytes(...)
pk_hash = hashlib.sha256(subject_public_key).hexdigest()
# Specify the TLSA parameters:
# 3: This is the certificate that the client should trust. No CA is needed.
# 0: The whole certificate is matched.
# 1: The certificate is SHA256'd here.
return "3 0 1 " + certhash
# 3: Match the (leaf) certificate. (No CA, no trust path needed.)
# 1: Match its subject public key.
# 1: Use SHA256.
return "3 1 1 " + pk_hash
def build_sshfp_records():
# The SSHFP record is a way for us to embed this server's SSH public

View File

@ -137,12 +137,12 @@ def get_domain_ssl_files(domain, ssl_certificates, env, allow_missing_cert=False
return cert_info['private-key'], cert_info['certificate'], via
def create_csr(domain, ssl_key, env):
def create_csr(domain, ssl_key, country_code, env):
return shell("check_output", [
"openssl", "req", "-new",
"-key", ssl_key,
"-sha256",
"-subj", "/C=%s/ST=/L=/O=/CN=%s" % (env["CSR_COUNTRY"], domain)])
"-subj", "/C=%s/ST=/L=/O=/CN=%s" % (country_code, domain)])
def install_cert(domain, ssl_cert, ssl_chain, env):
# Write the combined cert+chain to a temporary path and validate that it is OK.
@ -184,21 +184,22 @@ def install_cert(domain, ssl_cert, ssl_chain, env):
# When updating the cert for PRIMARY_HOSTNAME, symlink it from the system
# certificate path, which is hard-coded for various purposes, and then
# update DNS (because of the DANE TLSA record), postfix, and dovecot,
# which all use the file.
# restart postfix and dovecot.
if domain == env['PRIMARY_HOSTNAME']:
# Update symlink.
system_ssl_certificate = os.path.join(os.path.join(env["STORAGE_ROOT"], 'ssl', 'ssl_certificate.pem'))
os.unlink(system_ssl_certificate)
os.symlink(ssl_certificate, system_ssl_certificate)
# Update DNS & restart postfix and dovecot so they pick up the new file.
from dns_update import do_dns_update
ret.append( do_dns_update(env) )
# Restart postfix and dovecot so they pick up the new file.
shell('check_call', ["/usr/sbin/service", "postfix", "restart"])
shell('check_call', ["/usr/sbin/service", "dovecot", "restart"])
ret.append("mail services restarted")
# The DANE TLSA record will remain valid so long as the private key
# hasn't changed. We don't ever change the private key automatically.
# If the user does it, they must manually update DNS.
# Update the web configuration so nginx picks up the new certificate file.
from web_update import do_web_update
ret.append( do_web_update(env) )

View File

@ -28,6 +28,15 @@
<p><select id="ssldomain" onchange="show_csr()" class="form-control" style="width: auto"></select></p>
<p>What country are you in? This is required by some SSL certificate providers. You may leave this blank if you know your SSL certificate provider doesn't require it.</p>
<p><select id="sslcc" onchange="show_csr()" class="form-control" style="width: auto">
<option value="">(Select)</option>
{% for code, name in csr_country_codes %}
<option value="{{code}}">{{name}}</option>
{% endfor %}
</select></p>
<div id="csr_info" style="display: none">
<p>You will need to provide the SSL certificate provider this Certificate Signing Request (CSR):</p>
@ -94,6 +103,7 @@ function show_csr() {
"/ssl/csr/" + $('#ssldomain').val(),
"POST",
{
countrycode: $('#sslcc').val()
},
function(data) {
$('#ssl_csr').text(data);

View File

@ -32,12 +32,6 @@
<p>You MUST manually copy the encryption password from <tt class="backup-encpassword-file"></tt> to a safe and secure location. You will need this file to decrypt backup files. It is NOT stored in your Amazon S3 bucket.</p>
</div>
</div>
<div class="form-group backup-target-local backup-target-s3">
<label for="min-age" class="col-sm-2 control-label">How many days should backups be kept?</label>
<div class="col-sm-8">
<input type="number" class="form-control" rows="1" id="min-age">
</div>
</div>
<div class="form-group backup-target-s3">
<label for="backup-target-s3-host" class="col-sm-2 control-label">S3 Region</label>
<div class="col-sm-8">
@ -66,6 +60,13 @@
<input type="text" class="form-control" rows="1" id="backup-target-pass">
</div>
</div>
<div class="form-group backup-target-local backup-target-s3">
<label for="min-age" class="col-sm-2 control-label">Days:</label>
<div class="col-sm-8">
<input type="number" class="form-control" rows="1" id="min-age">
<div class="small" style="margin-top: 2px">This is the <i>minimum</i> number of days backup data is kept for. The box makes an incremental backup, so backup data is often kept much longer. An incremental backup file that is less than this number of days old requires that all previous increments back to the most recent full backup, plus that full backup, remain available.</div>
</div>
</div>
<div class="form-group">
<div class="col-sm-offset-2 col-sm-10">
<button id="set-s3-backup-button" type="submit" class="btn btn-primary">Save</button>

View File

@ -31,25 +31,18 @@ rm -f /etc/init.d/mailinabox
ln -s $(pwd)/conf/management-initscript /etc/init.d/mailinabox
hide_output update-rc.d mailinabox defaults
# Perform a daily backup.
cat > /etc/cron.daily/mailinabox-backup << EOF;
#!/bin/bash
# Mail-in-a-Box --- Do not edit / will be overwritten on update.
# Perform a backup.
$(pwd)/management/backup.py
EOF
chmod +x /etc/cron.daily/mailinabox-backup
# Remove old files we no longer use.
rm -f /etc/cron.daily/mailinabox-backup
rm -f /etc/cron.daily/mailinabox-statuschecks
# Perform daily status checks. Compare each day to the previous
# for changes and mail the changes to the administrator.
cat > /etc/cron.daily/mailinabox-statuschecks << EOF;
#!/bin/bash
# Mail-in-a-Box --- Do not edit / will be overwritten on update.
# Run status checks.
$(pwd)/management/status_checks.py --show-changes --smtp
EOF
chmod +x /etc/cron.daily/mailinabox-statuschecks
# Perform nightly tasks at 3am in system time: take a backup, run
# status checks and email the administrator any changes.
cat > /etc/cron.d/mailinabox-nightly << EOF;
# Mail-in-a-Box --- Do not edit / will be overwritten on update.
# Run nightly tasks: backup, status checks.
0 3 * * * root (cd `pwd` && management/daily_tasks.sh)
EOF
# Start it.
restart_service mailinabox

View File

@ -168,35 +168,6 @@ if [[ -z "$PRIVATE_IP" && -z "$PRIVATE_IPV6" ]]; then
exit
fi
# We need a country code to generate a certificate signing request. However
# if a CSR already exists then we won't be generating a new one and there's
# no reason to ask for the country code now. $STORAGE_ROOT has not yet been
# set so we'll check if $DEFAULT_STORAGE_ROOT and $DEFAULT_CSR_COUNTRY are
# set (the values from the current mailinabox.conf) and if the CSR exists
# in the expected location.
if [ ! -z "$DEFAULT_STORAGE_ROOT" ] && [ ! -z "$DEFAULT_CSR_COUNTRY" ] && [ -f $DEFAULT_STORAGE_ROOT/ssl/ssl_cert_sign_req.csr ]; then
CSR_COUNTRY=$DEFAULT_CSR_COUNTRY
fi
if [ -z "$CSR_COUNTRY" ]; then
# Get a list of country codes. Separate codes from country names with a ^.
# The input_menu function modifies shell word expansion to ignore spaces
# (since country names can have spaces) and use ^ instead.
country_code_list=$(grep -v "^#" setup/csr_country_codes.tsv | sed "s/\(..\)\t\([^\t]*\).*/\1^\2/")
input_menu "Country Code" \
"Choose the country where you live or where your organization is based.
\n\n(This is used to create an SSL certificate.)
\n\nCountry Code:" \
"$country_code_list" \
CSR_COUNTRY
if [ -z "$CSR_COUNTRY" ]; then
# user hit ESC/cancel
exit
fi
fi
# Automatic configuration, e.g. as used in our Vagrant configuration.
if [ "$PUBLIC_IP" = "auto" ]; then
# Use a public API to get our public IP address, or fall back to local network configuration.

View File

@ -74,7 +74,7 @@ if [ ! -f $STORAGE_ROOT/ssl/ssl_certificate.pem ]; then
CSR=/tmp/ssl_cert_sign_req-$$.csr
hide_output \
openssl req -new -key $STORAGE_ROOT/ssl/ssl_private_key.pem -out $CSR \
-sha256 -subj "/C=$CSR_COUNTRY/ST=/L=/O=/CN=$PRIMARY_HOSTNAME"
-sha256 -subj "/C=/ST=/L=/O=/CN=$PRIMARY_HOSTNAME"
# Generate the self-signed certificate.
CERT=$STORAGE_ROOT/ssl/$PRIMARY_HOSTNAME-selfsigned-$(date --rfc-3339=date | sed s/-//g).pem

View File

@ -34,6 +34,8 @@ if [ -f /etc/mailinabox.conf ]; then
cat /etc/mailinabox.conf | sed s/^/DEFAULT_/ > /tmp/mailinabox.prev.conf
source /tmp/mailinabox.prev.conf
rm -f /tmp/mailinabox.prev.conf
else
FIRST_TIME_SETUP=1
fi
# Put a start script in a global location. We tell the user to run 'mailinabox'
@ -45,7 +47,7 @@ source setup/start.sh
EOF
chmod +x /usr/local/bin/mailinabox
# Ask the user for the PRIMARY_HOSTNAME, PUBLIC_IP, PUBLIC_IPV6, and CSR_COUNTRY
# Ask the user for the PRIMARY_HOSTNAME, PUBLIC_IP, and PUBLIC_IPV6,
# if values have not already been set in environment variables. When running
# non-interactively, be sure to set values for all! Also sets STORAGE_USER and
# STORAGE_ROOT.
@ -87,7 +89,6 @@ PUBLIC_IP=$PUBLIC_IP
PUBLIC_IPV6=$PUBLIC_IPV6
PRIVATE_IP=$PRIVATE_IP
PRIVATE_IPV6=$PRIVATE_IPV6
CSR_COUNTRY=$CSR_COUNTRY
EOF
# Start service configuration.

View File

@ -55,6 +55,38 @@ apt_install python3 python3-dev python3-pip \
haveged pollinate \
unattended-upgrades cron ntp fail2ban
# ### Set the system timezone
#
# Some systems are missing /etc/timezone, which we cat into the configs for
# Z-Push and ownCloud, so we need to set it to something. Daily cron tasks
# like the system backup are run at a time tied to the system timezone, so
# letting the user choose will help us identify the right time to do those
# things (i.e. late at night in whatever timezone the user actually lives
# in).
#
# However, changing the timezone once it is set seems to confuse fail2ban
# and requires restarting fail2ban (done below in the fail2ban
# section) and syslog (see #328). There might be other issues, and it's
# not likely the user will want to change this, so we only ask on first
# setup.
if [ -z "$NONINTERACTIVE" ]; then
if [ ! -f /etc/timezone ] || [ ! -z $FIRST_TIME_SETUP ]; then
# If the file is missing or this is the user's first time running
# Mail-in-a-Box setup, run the interactive timezone configuration
# tool.
dpkg-reconfigure tzdata
restart_service rsyslog
fi
else
# This is a non-interactive setup so we can't ask the user.
# If /etc/timezone is missing, set it to UTC.
if [ ! -f /etc/timezone ]; then
echo "Setting timezone to UTC."
echo "Etc/UTC" > /etc/timezone
restart_service rsyslog
fi
fi
# ### Seed /dev/urandom
#
# /dev/urandom is used by various components for generating random bytes for

View File

@ -458,7 +458,6 @@ class BashScript(Grammar):
v = re.sub(r"(\$?)PRIMARY_HOSTNAME", r"<b>box.yourdomain.com</b>", v)
v = re.sub(r"\$STORAGE_ROOT", r"<b>$STORE</b>", v)
v = re.sub(r"\$CSR_COUNTRY", r"<b>US</b>", v)
v = v.replace("`pwd`", "<code><b>/path/to/mailinabox</b></code>")
return v