A minimal Ubuntu server installation might not have universe enabled by
default. By adding it, we ensure we can install packages only available
in universe, such as python3-pip
Merges #1650.
* Download and verify Nextcloud download before deleting old install directory
* Changed install logic to look at config.php and not version.php for database version number. When restoring from a backup, config.php in STORAGE_ROOT will hold the Nextcloud version that corresponds to the user's database and version.php in /usr/local won't even exist, so we were missing Nextcloud migration steps. In other cases they should be the same.
* Update to Nextcloud 15.0.7, Contacts to 3.1.1, and Calendar to 1.6.5
* Enabled localhost-only insecure IMAP login for localhost Nextcloud auth
* Add package php-imagick and BigInt conversion
* added support for /cloud/oc[sm]-provider/ endpoint
* Upgraded Nextcloud from 13.0.6 to 14.0.6.
* Upgraded Contacts from 2.1.5 to 2.1.8.
* Upgraded Calendar from 1.6.1 to 1.6.4.
* Cleanup unsupported version upgrades: Since an upgrade to v0.30 is mandatory before moving upward, I removed the checks for Nextcloud prior version 12.
* Fix the storage root path.
* Add missing indices. Thx @yodax for your feedback.
* drop the ondrej/php PPA since PHP 7.x is available directly from Ubuntu 18.04
* intall PHP 7.2 which is just the "php" package in Ubuntu 18.04
* some package names changed, some unnecessary packages are no longer provided
* update paths
@joshdata squashed pull request #1398, removed some comments, and added these notes:
* The old init.d script for the management daemon is replaced with a systemd service.
* A systemd service configuration is added to configure permissions for munin on startup.
* nginx SSL settings are updated because nginx's options and defaults have changed, and we now enable http2.
* Automatic SSHFP record generation is updated to know that 22 is the default SSH daemon port, since it is no longer explicit in sshd_config.
* The dovecot-lucene package is dropped because the Mail-in-a-Box PPA where we built the package has not been updated for Ubuntu 18.04.
* The stock postgrey package is installed instead of the one from our PPA (which we no longer support), which loses the automatic whitelisting of DNSWL.org-whitelisted senders.
* Drop memcached and the status check for memcached, which we used to use with ownCloud long ago but are no longer installing.
* Other minor changes.
fixes#1409
This reverts commit 82844ca651 ("make certbot auto-agree to TOS if NONINTERACTIVE=1 env var is set (#1399)") and instead *always* auto-agree. If we don't auto-agree, certbot asks the user interactively, but our "curl | bash" setup line does not permit interactive prompts, so certbot failed to register and all certificate things were broken until the command was re-run interactively.
Starting with 2.4, Z-Push no longer provides tarballs on their download server. The only options are getting the code from their git repository or using one of their distribution packages. Their Ubuntu 18.04 packaes don't seem to actually work in Ubuntu 18.04, so thinking ahead that's currently a bad choice. In 78d1c9be6e we switched from doing a git clone to using wget on their downloads server because of a problem with something related to stash.z-hub.io's SSL certificate. But wget also seems to work on their source code repository, so we can use that.
git clone (which uses curl) underneath was failing. Curiously, the same
git clone command would work on my macos host machine.
From the screenshot it looks like curl was somehow not able to negotiate
the connection. Might have been a missing CA certificate for Comodo, but
I was not able to determine if that was the issue.
fixes#1393closes#1387closes#1400
Our wget_verify function uses wget to download a file and then check
the file's hash. If wget fails, i.e. because of a 404 or other HTTP
or network error, we exited setup without displaying any output because
normally there are no errors and -q keeps the setup output clean.
Wrapping wget with our hide_output function, and dropping -q, captures
wget's output and shows it and exits setup just if wget fails.
see #1297
In 0088fb4553 I changed the management daemon's startup
script from a symlink to a Python script to a bash script that activated the new virtualenv
and then launched Python. As a result, the init.d script that starts the daemon would
write the pid of bash to the pidfile, and when trying to kill it, it would kill bash but
not the Python process.
Using exec to start Python fixes this problem by making the Python process have the pid
that the init.d script knows about.
fixes#1339
The cryptography package has created all sorts of installation trouble over the last few years, probably because of mismatches between OS-installed packages and pip-installed packages. Using a virtualenv for all Python packages used by the management daemon should make sure everything is consistent.
See #1298, see #1264.
The PHP5 packages have a dependency on (apache2 or php5-cgi or php5-fpm), and since removing php5-fpm apache2 started getting installed during setup, which caused a conflict with nginx of course.
These packages don't seem to be needed by Roundcube or Nextcloud --- Roundcube includes the ones it needs.
see #1264, #1298
To enable users to do custom spam filtering based on score, it's helpful to render the actual spam score as a float in a separate header rather than as part of X-Spam-Status where it only appears in a comma separated list.
Nextcloud 12 adds a new OC_VersionCanBeUpgradedFrom field to /usr/local/lib/owncloud/version.php which lists
prior NC/OC version numbers, which confuses our check for what the installed version is. Make our regex more strict.
merges #1238
* Install PHP7 via a PPA, enable unattended upgrades for the PPA, and switch all of our PHP configuration to the PHP7 install.
* Keep installing PHP5 for ownCloud/Nextcloud packages because we need it to possibly run transitional updates to ownCloud/Nextcloud versions less than 12. But replace PHP5 packages with PHP7 packages elsewhere.
* Update to Nextcloud 12 which requires PHP7, with a transitional upgrade to Nextcloud 11.0.3.
* Disable TLS cert validation by Roundcube when connecting to localhost IMAP and SMTP. Validation became the default in PHP7 but we don't necessarily have a (non-self-)signed certificate and it definitely isn't valid for the IP address 127.0.0.1.
Merges #1140
Since it says "RCMCardDAV requires at least PHP 5.6.18. Older versions might work", let's hope for the best.
Also hiding its preferences panel in settings since if it doesn't work, we don't want folks using it for anything but connecting to ownCloud contacts.
* Move variable assignment up and do not use call arguments directly
* Upgrade ownCloud to latest patch release 9.1.4
also move owncloud hash to its own variable
pip<6.1 + setuptools>=34 have a problem with packages that
try to update setuptools during installation, like cryptography.
See https://github.com/pypa/pip/issues/4253. The Ubuntu 14.04
package versions are pip 1.5.4 and setuptools 3.3. When we
install cryptography under those versions, it tries to update
setuptools to version 34, which became available about 10 days
ago, and then pip gets permanently broken with errors like
"ImportError: No module named 'packaging'".
The easiest work-around on systems that aren't already broken is
to upgrade pip and setuptools individually before we install any
package that tries to update setuptools.
Also try to detect a broken system and forcibly remove setuptools
first before trying to install/upgrade pip.
fixes#1080, fixes#1081, fixes#1086
see #1083
see https://discourse.mailinabox.email/t/error-with-pip-and-python/1880
see https://discourse.mailinabox.email/t/error-installing-mib/1875
* Added support for backup to a remote server using rsync
* updated web interface to get data from user
* added way to list files from server
It’s not using the “username” field of the yaml configuration
file to minimise the amount of patches needed. So the username
is actually sorted within the rsync URL.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Added ssh key generation upon installation for root user.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Removed stale blank lines, and fixed typo
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* fix backup-location lines, by switching it from id to class
* Various web UI fixes
- fixed user field being shadowed ;
- fixed settings reading comparaison ;
- fixed forgotten min-age field.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Added SSH Public Key shown on the web interface UI
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* trailing spaces.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* fixed the extraneous environment
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Updated key setup
- made key lower in bits, but stronger (using -a option),
- made ssh-keygen run in background using nohup,
- added independent key file, as id_rsa_miab,
- added ssh-options to all duplicity calls to use the id_rsa_miab keyfile,
- changed path to the public key display
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* added rsync options for ssh identity support
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* removed strict host checking for all backup operations
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Remove nohup from ssh-keygen so errors aren't hidden. Also only generate a key if none exists yet
* Add trailing slash when checking a remote backup. Also check if we actually can read the remote size
* Factorisation of the repeated rsync/ssh options
cf https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478919
* Updated message SSH key creation
https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478886
On every login we're notified:
New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Disable this so that an eager yet inattentive admin
doesn't accidentally follow these instructions.
[this is a squashed merge from-]
* Install owncoud 9.1 and provide an upgrade path from 8.2. This also disables memcached and goes with apc. The upgrade fails with memcached.
* Remove php apc setting
* Add dav migrations for each user
* Add some comments to the code
* When upgrading owncloud from 8.2.3 to 9.1.0 the backup of 8.2.3 was overwritten when going from 9.0 to 9.1
* Add upgrade path from 8.1.1. Only do an upgrade check if owncloud was previously installed.
* Stop php5-fpm before owncloud upgrade to prevent database locks
* Fix fail2ban tests for owncloud 9
* When upgrading owncloud copy the database to the user-data/owncloud-backup directory
* Remove not need unzip directives during owncloud extraction. Directory is removed beforehand so a normal extraction is fine
* Improve backup of owncloud installation and provide a post installation restore script. Update the owncloud version number to 9.1.1. Update the calendar and contacts apps to the latest versions
* Separate the ownCloud upgrades visually in the console output.
In the earlier commit, I added a Dovecot userdb lookup. Without a userdb lookup, Dovecot would use the password db for user lookups. With a userdb lookup we can support iterating over users.
But I forgot the WHERE clause in the query, resulting in every incoming message being accepted if the user database contained any users at all. Since the mailbox path template is the same for all users, mail was delivered correctly except that mail that should have been rejected was delivered too.
On some machines localhost is defined as something other than 127.0.0.1, and if we mix "127.0.0.1" and "localhost" then some connections won't be to to the address a service is actually running on.
This was the case with DKIM: It was running on "localhost" but Postfix was connecting to it at 127.0.0.1. (https://discourse.mailinabox.email/t/opendkim-is-not-running-port-8891/1188/12.)
I suppose "localhost" could be an alias to an IPv6 address? We don't really want local services binding on IPv6, so use "127.0.0.1" to be explicit and don't use "localhost" to be sure we get an IPv4 address.
Fixes#797
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJW/mJqAAoJELkgQfTBC92B/F8H/2s6wKhzzeoqkhLU2nvYJh0B
Q1d0SbtdQWIWrTQbcjIR3aGYwJzJ+HC7rylrwS4lB2ugpJBA0MnfD+ktwbe/EyDa
pN6WLlmnXyAw28//ubq0FQqC8Gawsj4WMfmSEw/XuDShik8XJmU7QUEnewClJ7So
ko4eVp9KL8MU3Rj/DebhyoW0EjpB/qrJvLSqtj4KCxKYES9J8nUVBFVRDL48yNx4
2KTIjqreGZmtW0/wxPnganMeV6DZn3B6vBmqOYYvw7bf6r/cY0ZkNK/ENlo+ntJD
3jFKki4TJChhGVWH5T4Tw2bys4Cua1+SA3cleNRH1rYSvRWyOCwK+LS4YBJHYp4=
=umMp
-----END PGP SIGNATURE-----
merge hotfix release tag 'v0.17c' into master
The hotfixes were all already applied to master in original PRs. This merge merely brings over the CHANGELOG and the updated install instructions (v0.17b=>v0.17c), including to bootstrap.sh which is what triggers v0.17c being the latest release.
----------------------
ownCloud moved their source code to a new location, breaking our installation script.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJW1YnLAAoJELkgQfTBC92BZg8H/RVfJRrhkM6B0UocYOmp2LQB
u6b/KxSM5JbBq9BTyQTAkQ0Dj6caWpBwAATGgxvyAYt0Du/QTx/ycfgenkP2DWcI
UYm+AkV2Eg9BiXpFqG33vJ/aWjATB+S3XM+77jMbeMb4kK+PfOHguvmxnsS4x8HY
h+unkXPy8fr1rATaOGg32fTZrKkVWAriHFfJqy3kkoTHS+9fj7MVGbTBPntPaC46
/42IsCFiQsNdz6QeHmXzFnP6AuqmkbcN9dpsFe36qfSfwVhNqC776lxxP+r1nK3p
AWs/q5VQ4V1ght3MzDch9dOlmGuqBVB1WkGdQVV7RemYrdNi5S1h+dXXUDiNXY8=
=C7LW
-----END PGP SIGNATURE-----
merge point release v0.17b
ownCloud moved their source code to a new location, breaking our installation script.
https://discourse.mailinabox.email/t/status-check-emails-empty-after-upgrading-to-v0-16/1082/3
A user on that thread suggests an alternate solution, adding `PYTHONIOENCODING=utf-8` to `/etc/environment`. Python docs say that affects stdin/out/err. But we also use these environment variables elsewhere to ensure that config files we read/write are opened with UTF8 too. Maybe all that can be simplified too.
The default timeout for Postfix's maximal_queue_lifetime and bounce_queue_lifetime is 5 days. This is way too long if you expect someone to have an answer and after 5 days you'll get the message that it's not delivered. This disrupts communication. It would be more responsive if the user got the 'can't deliver' error after 24 hours.
Sending mail through Exchange/ActiveSync (Z-Push) had been broken since v0.14. This is now fixed.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJWkQ9rAAoJELkgQfTBC92BPjQH/ibFBZHgma7C53Q4X9iPfUk2
dPK75rNBx06d6yW4LNAYuWVnNO1Mb0khb2k2UPwg4noImYWqsNS9hXih7C6oPHiK
Szz4ubCc5MEqnmhxiNkzLdIBvsyKmz8IfmCl+LCXu8uk0Fb+pB6zbSdAGxjtaSPL
itCwz8+ApTC4bl1CoNYPn2zudHmHNeC7L6INYdb+xbtc/Tz6mO/xMaBVPDiKeq9P
LqLTOXiJNENz7vKSZytlWGOTdtSTZqwd7JBXuBg0QFz5C9yg8EV4LWB9wOm5aTIf
Fol3WK5ZHA5YeihZKmZSjz9+p4iwv5hqR5osKL2n46LeVHyafESl+QnZSXRlDjE=
=i3ei
-----END PGP SIGNATURE-----
Merge release branch v0.15a
v0.15a
Sending mail through Exchange/ActiveSync (Z-Push) had been broken since v0.14. This is now fixed.
-----------------------
Mail:
* Updated Roundcube to version 1.1.3.
* Auto-create aliases for abuse@, as required by RFC2142.
* The DANE TLSA record is changed to use the certificate subject public key rather than the whole certificate, which means the record remains valid after certificate changes (so long as the private key remains the same, which it does for us).
Control panel:
* When IPv6 is enabled, check that system services are accessible over IPv6 too, that the box's hostname resolves over IPv6, and that reverse DNS is setup correctly for IPv6.
* Explanatory text for setting up secondary nameserver is added/fixed.
* DNS checks now have a timeout in case a DNS server is not responding, so the checks don't stall indefinitely.
* Better messages if external DNS is used and, weirdly, custom secondary nameservers are set.
* Add POP to the mail client settings documentation.
* The box's IP address is added to the fail2ban whitelist so that the status checks don't trigger the machine banning itself, which results in the status checks showing services down even though they are running.
* For SSL certificates, rather than asking you what country you are in during setup, ask at the time a CSR is generated. The default system self-signed certificate now omits a country in the subject (it was never needed). The CSR_COUNTRY Mail-in-a-Box setting is dropped entirely.
System:
* Nightly backups and system status checks are now moved to 3am in the system's timezone.
* fail2ban's recidive jail is now active, which guards against persistent brute force login attacks over long periods of time.
* Setup (first run only) now asks for your timezone to set the system time.
* The Exchange/ActiveSync server is now taken offline during nightly backups (along with SMTP and IMAP).
* The machine's random number generator (/dev/urandom) is now seeded with Ubuntu Pollinate and a blocking read on /dev/random.
* DNSSEC key generation during install now uses /dev/urandom (instead of /dev/random), which is faster.
* The $STORAGE_ROOT/ssl directory is flattened by a migration script and the system SSL certificate path is now a symlink to the actual certificate.
* If ownCloud sends out email, it will use the box's administrative address now (admin@yourboxname).
* Z-Push (Exchange/ActiveSync) logs now exclude warnings and are now rotated to save disk space.
* Fix pip command that might have not installed all necessary Python packages.
* The control panel and backup would not work on Google Compute Engine because GCE installs a conflicting boto package.
* Added a new command `management/backup.py --restore` to restore files from a backup to a target directory (command line arguments are passed to `duplicity restore`).
