Our wget_verify function uses wget to download a file and then check
the file's hash. If wget fails, i.e. because of a 404 or other HTTP
or network error, we exited setup without displaying any output because
normally there are no errors and -q keeps the setup output clean.
Wrapping wget with our hide_output function, and dropping -q, captures
wget's output and shows it and exits setup just if wget fails.
see #1297
In 0088fb4553 I changed the management daemon's startup
script from a symlink to a Python script to a bash script that activated the new virtualenv
and then launched Python. As a result, the init.d script that starts the daemon would
write the pid of bash to the pidfile, and when trying to kill it, it would kill bash but
not the Python process.
Using exec to start Python fixes this problem by making the Python process have the pid
that the init.d script knows about.
fixes#1339
The cryptography package has created all sorts of installation trouble over the last few years, probably because of mismatches between OS-installed packages and pip-installed packages. Using a virtualenv for all Python packages used by the management daemon should make sure everything is consistent.
See #1298, see #1264.
The PHP5 packages have a dependency on (apache2 or php5-cgi or php5-fpm), and since removing php5-fpm apache2 started getting installed during setup, which caused a conflict with nginx of course.
These packages don't seem to be needed by Roundcube or Nextcloud --- Roundcube includes the ones it needs.
see #1264, #1298
To enable users to do custom spam filtering based on score, it's helpful to render the actual spam score as a float in a separate header rather than as part of X-Spam-Status where it only appears in a comma separated list.
Nextcloud 12 adds a new OC_VersionCanBeUpgradedFrom field to /usr/local/lib/owncloud/version.php which lists
prior NC/OC version numbers, which confuses our check for what the installed version is. Make our regex more strict.
merges #1238
* Install PHP7 via a PPA, enable unattended upgrades for the PPA, and switch all of our PHP configuration to the PHP7 install.
* Keep installing PHP5 for ownCloud/Nextcloud packages because we need it to possibly run transitional updates to ownCloud/Nextcloud versions less than 12. But replace PHP5 packages with PHP7 packages elsewhere.
* Update to Nextcloud 12 which requires PHP7, with a transitional upgrade to Nextcloud 11.0.3.
* Disable TLS cert validation by Roundcube when connecting to localhost IMAP and SMTP. Validation became the default in PHP7 but we don't necessarily have a (non-self-)signed certificate and it definitely isn't valid for the IP address 127.0.0.1.
Merges #1140
Since it says "RCMCardDAV requires at least PHP 5.6.18. Older versions might work", let's hope for the best.
Also hiding its preferences panel in settings since if it doesn't work, we don't want folks using it for anything but connecting to ownCloud contacts.
* Move variable assignment up and do not use call arguments directly
* Upgrade ownCloud to latest patch release 9.1.4
also move owncloud hash to its own variable
pip<6.1 + setuptools>=34 have a problem with packages that
try to update setuptools during installation, like cryptography.
See https://github.com/pypa/pip/issues/4253. The Ubuntu 14.04
package versions are pip 1.5.4 and setuptools 3.3. When we
install cryptography under those versions, it tries to update
setuptools to version 34, which became available about 10 days
ago, and then pip gets permanently broken with errors like
"ImportError: No module named 'packaging'".
The easiest work-around on systems that aren't already broken is
to upgrade pip and setuptools individually before we install any
package that tries to update setuptools.
Also try to detect a broken system and forcibly remove setuptools
first before trying to install/upgrade pip.
fixes#1080, fixes#1081, fixes#1086
see #1083
see https://discourse.mailinabox.email/t/error-with-pip-and-python/1880
see https://discourse.mailinabox.email/t/error-installing-mib/1875
* Added support for backup to a remote server using rsync
* updated web interface to get data from user
* added way to list files from server
It’s not using the “username” field of the yaml configuration
file to minimise the amount of patches needed. So the username
is actually sorted within the rsync URL.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Added ssh key generation upon installation for root user.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Removed stale blank lines, and fixed typo
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* fix backup-location lines, by switching it from id to class
* Various web UI fixes
- fixed user field being shadowed ;
- fixed settings reading comparaison ;
- fixed forgotten min-age field.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Added SSH Public Key shown on the web interface UI
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* trailing spaces.
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* fixed the extraneous environment
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Updated key setup
- made key lower in bits, but stronger (using -a option),
- made ssh-keygen run in background using nohup,
- added independent key file, as id_rsa_miab,
- added ssh-options to all duplicity calls to use the id_rsa_miab keyfile,
- changed path to the public key display
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* added rsync options for ssh identity support
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* removed strict host checking for all backup operations
Signed-off-by: Bernard `Guyzmo` Pratz <guyzmo+github@m0g.net>
* Remove nohup from ssh-keygen so errors aren't hidden. Also only generate a key if none exists yet
* Add trailing slash when checking a remote backup. Also check if we actually can read the remote size
* Factorisation of the repeated rsync/ssh options
cf https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478919
* Updated message SSH key creation
https://github.com/mail-in-a-box/mailinabox/pull/678#discussion_r81478886
On every login we're notified:
New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.
Disable this so that an eager yet inattentive admin
doesn't accidentally follow these instructions.
[this is a squashed merge from-]
* Install owncoud 9.1 and provide an upgrade path from 8.2. This also disables memcached and goes with apc. The upgrade fails with memcached.
* Remove php apc setting
* Add dav migrations for each user
* Add some comments to the code
* When upgrading owncloud from 8.2.3 to 9.1.0 the backup of 8.2.3 was overwritten when going from 9.0 to 9.1
* Add upgrade path from 8.1.1. Only do an upgrade check if owncloud was previously installed.
* Stop php5-fpm before owncloud upgrade to prevent database locks
* Fix fail2ban tests for owncloud 9
* When upgrading owncloud copy the database to the user-data/owncloud-backup directory
* Remove not need unzip directives during owncloud extraction. Directory is removed beforehand so a normal extraction is fine
* Improve backup of owncloud installation and provide a post installation restore script. Update the owncloud version number to 9.1.1. Update the calendar and contacts apps to the latest versions
* Separate the ownCloud upgrades visually in the console output.
In the earlier commit, I added a Dovecot userdb lookup. Without a userdb lookup, Dovecot would use the password db for user lookups. With a userdb lookup we can support iterating over users.
But I forgot the WHERE clause in the query, resulting in every incoming message being accepted if the user database contained any users at all. Since the mailbox path template is the same for all users, mail was delivered correctly except that mail that should have been rejected was delivered too.
On some machines localhost is defined as something other than 127.0.0.1, and if we mix "127.0.0.1" and "localhost" then some connections won't be to to the address a service is actually running on.
This was the case with DKIM: It was running on "localhost" but Postfix was connecting to it at 127.0.0.1. (https://discourse.mailinabox.email/t/opendkim-is-not-running-port-8891/1188/12.)
I suppose "localhost" could be an alias to an IPv6 address? We don't really want local services binding on IPv6, so use "127.0.0.1" to be explicit and don't use "localhost" to be sure we get an IPv4 address.
Fixes#797
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJW/mJqAAoJELkgQfTBC92B/F8H/2s6wKhzzeoqkhLU2nvYJh0B
Q1d0SbtdQWIWrTQbcjIR3aGYwJzJ+HC7rylrwS4lB2ugpJBA0MnfD+ktwbe/EyDa
pN6WLlmnXyAw28//ubq0FQqC8Gawsj4WMfmSEw/XuDShik8XJmU7QUEnewClJ7So
ko4eVp9KL8MU3Rj/DebhyoW0EjpB/qrJvLSqtj4KCxKYES9J8nUVBFVRDL48yNx4
2KTIjqreGZmtW0/wxPnganMeV6DZn3B6vBmqOYYvw7bf6r/cY0ZkNK/ENlo+ntJD
3jFKki4TJChhGVWH5T4Tw2bys4Cua1+SA3cleNRH1rYSvRWyOCwK+LS4YBJHYp4=
=umMp
-----END PGP SIGNATURE-----
merge hotfix release tag 'v0.17c' into master
The hotfixes were all already applied to master in original PRs. This merge merely brings over the CHANGELOG and the updated install instructions (v0.17b=>v0.17c), including to bootstrap.sh which is what triggers v0.17c being the latest release.
----------------------
ownCloud moved their source code to a new location, breaking our installation script.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJW1YnLAAoJELkgQfTBC92BZg8H/RVfJRrhkM6B0UocYOmp2LQB
u6b/KxSM5JbBq9BTyQTAkQ0Dj6caWpBwAATGgxvyAYt0Du/QTx/ycfgenkP2DWcI
UYm+AkV2Eg9BiXpFqG33vJ/aWjATB+S3XM+77jMbeMb4kK+PfOHguvmxnsS4x8HY
h+unkXPy8fr1rATaOGg32fTZrKkVWAriHFfJqy3kkoTHS+9fj7MVGbTBPntPaC46
/42IsCFiQsNdz6QeHmXzFnP6AuqmkbcN9dpsFe36qfSfwVhNqC776lxxP+r1nK3p
AWs/q5VQ4V1ght3MzDch9dOlmGuqBVB1WkGdQVV7RemYrdNi5S1h+dXXUDiNXY8=
=C7LW
-----END PGP SIGNATURE-----
merge point release v0.17b
ownCloud moved their source code to a new location, breaking our installation script.
https://discourse.mailinabox.email/t/status-check-emails-empty-after-upgrading-to-v0-16/1082/3
A user on that thread suggests an alternate solution, adding `PYTHONIOENCODING=utf-8` to `/etc/environment`. Python docs say that affects stdin/out/err. But we also use these environment variables elsewhere to ensure that config files we read/write are opened with UTF8 too. Maybe all that can be simplified too.
The default timeout for Postfix's maximal_queue_lifetime and bounce_queue_lifetime is 5 days. This is way too long if you expect someone to have an answer and after 5 days you'll get the message that it's not delivered. This disrupts communication. It would be more responsive if the user got the 'can't deliver' error after 24 hours.
Sending mail through Exchange/ActiveSync (Z-Push) had been broken since v0.14. This is now fixed.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAABAgAGBQJWkQ9rAAoJELkgQfTBC92BPjQH/ibFBZHgma7C53Q4X9iPfUk2
dPK75rNBx06d6yW4LNAYuWVnNO1Mb0khb2k2UPwg4noImYWqsNS9hXih7C6oPHiK
Szz4ubCc5MEqnmhxiNkzLdIBvsyKmz8IfmCl+LCXu8uk0Fb+pB6zbSdAGxjtaSPL
itCwz8+ApTC4bl1CoNYPn2zudHmHNeC7L6INYdb+xbtc/Tz6mO/xMaBVPDiKeq9P
LqLTOXiJNENz7vKSZytlWGOTdtSTZqwd7JBXuBg0QFz5C9yg8EV4LWB9wOm5aTIf
Fol3WK5ZHA5YeihZKmZSjz9+p4iwv5hqR5osKL2n46LeVHyafESl+QnZSXRlDjE=
=i3ei
-----END PGP SIGNATURE-----
Merge release branch v0.15a
v0.15a
Sending mail through Exchange/ActiveSync (Z-Push) had been broken since v0.14. This is now fixed.
-----------------------
Mail:
* Updated Roundcube to version 1.1.3.
* Auto-create aliases for abuse@, as required by RFC2142.
* The DANE TLSA record is changed to use the certificate subject public key rather than the whole certificate, which means the record remains valid after certificate changes (so long as the private key remains the same, which it does for us).
Control panel:
* When IPv6 is enabled, check that system services are accessible over IPv6 too, that the box's hostname resolves over IPv6, and that reverse DNS is setup correctly for IPv6.
* Explanatory text for setting up secondary nameserver is added/fixed.
* DNS checks now have a timeout in case a DNS server is not responding, so the checks don't stall indefinitely.
* Better messages if external DNS is used and, weirdly, custom secondary nameservers are set.
* Add POP to the mail client settings documentation.
* The box's IP address is added to the fail2ban whitelist so that the status checks don't trigger the machine banning itself, which results in the status checks showing services down even though they are running.
* For SSL certificates, rather than asking you what country you are in during setup, ask at the time a CSR is generated. The default system self-signed certificate now omits a country in the subject (it was never needed). The CSR_COUNTRY Mail-in-a-Box setting is dropped entirely.
System:
* Nightly backups and system status checks are now moved to 3am in the system's timezone.
* fail2ban's recidive jail is now active, which guards against persistent brute force login attacks over long periods of time.
* Setup (first run only) now asks for your timezone to set the system time.
* The Exchange/ActiveSync server is now taken offline during nightly backups (along with SMTP and IMAP).
* The machine's random number generator (/dev/urandom) is now seeded with Ubuntu Pollinate and a blocking read on /dev/random.
* DNSSEC key generation during install now uses /dev/urandom (instead of /dev/random), which is faster.
* The $STORAGE_ROOT/ssl directory is flattened by a migration script and the system SSL certificate path is now a symlink to the actual certificate.
* If ownCloud sends out email, it will use the box's administrative address now (admin@yourboxname).
* Z-Push (Exchange/ActiveSync) logs now exclude warnings and are now rotated to save disk space.
* Fix pip command that might have not installed all necessary Python packages.
* The control panel and backup would not work on Google Compute Engine because GCE installs a conflicting boto package.
* Added a new command `management/backup.py --restore` to restore files from a backup to a target directory (command line arguments are passed to `duplicity restore`).
v0.14 (November 4, 2015)
------------------------
Mail:
* Spamassassin's network-based tests (Pyzor, others) and DKIM tests are now enabled. (Pyzor had always been installed but was not active due to a misconfiguration.)
* Moving spam out of the Spam folder and into Trash would incorrectly train Spamassassin that those messages were not spam.
* Automatically create the Sent and Archive folders for new users.
* The HTML5_Notifier plugin for Roundcube is now included, which when turned on in Roundcube settings provides desktop notifications for new mail.
* The Exchange/ActiveSync backend Z-Push has been updated to fix a problem with CC'd emails not being sent to the CC recipients.
Calender/Contacts:
* CalDAV/CardDAV and Exchange/ActiveSync for calendar/contacts wasn't working in some network configurations.
Web:
* When a new domain is added to the box, rather than applying a new self-signed certificate for that domain, the SSL certificate for the box's primary hostname will be used instead.
* If a custom DNS record is set on a domain or 'www'+domain, web would not be served for that domain. If the custom DNS record is just the box's IP address, that's a configuration mistake, but allow it and let web continue to be served.
* Accommodate really long domain names by increasing an nginx setting.
Control panel:
* Added an option to check for new Mail-in-a-Box versions within status checks. It is off by default so that boxes don't "phone home" without permission.
* Added a random password generator on the users page to simplify creating new accounts.
* When S3 backup credentials are set, the credentials are now no longer ever sent back from the box to the client, for better security.
* Fixed the jumpiness when a modal is displayed.
* Focus is put into the login form fields when the login form is displayed.
* Status checks now include a warning if a custom DNS record has been set on a domain that would normally serve web and as a result that domain no longer is serving web.
* Status checks now check that secondary nameservers, if specified, are actually serving the domains.
* Some errors in the control panel when there is invalid data in the database or an improperly named archived user account have been suppressed.
* Added subresource integrity attributes to all remotely-sourced resources (i.e. via CDNs) to guard against CDNs being used as an attack vector.
System:
* Tweaks to fail2ban settings.
* Fixed a spurrious warning while installing munin.
Seems like ownCloud 8.1.1 now doesn't play nice with trusted_domains. Whatever is put in ahead of time gets reset to an array containing 'localhost' only, probably because we invoke autoconfiguration from the command line where it doesn't know the hostname it's being accessed from. We now set this value after running autoconfig.
This has the added benefit of also fixing the problem that if PRIMARY_HOSTNAME changes, trusted_domains wasn't updated. Now it is. Fixes#503.
See #514.
Work-around for ownCloud 8.1.1 upgrade bug and tweaking munin's setup.
v0.13a (August 23, 2015)
------------------------
Note: v0.13 (no 'a', August 19, 2015) was pulled immediately due to an ownCloud bug that prevented upgrades. v0.13a works around that problem.
Mail:
* Outbound mail headers (the Recieved: header) are tweaked to possibly improve deliverability.
* Some MIME messages would hang Roundcube due to a missing package.
* The users permitted to send as an alias can now be different from where an alias forwards to.
DNS:
* The secondary nameservers option in the control panel now accepts more than one nameserver and a special xfr:IP format to specify zone-transfer-only IP addresses.
* A TLSA record is added for HTTPS for DNSSEC-aware clients that support it.
System:
* Backups can now be turned off, or stored in Amazon S3, through new control panel options.
* Munin was not working on machines confused about their hostname and had lots of errors related to PANGO, NTP peers and network interfaces that were not up.
* ownCloud updated to version 8.1.1 (with upgrade work-around), its memcached caching enabled.
* When upgrading, network checks like blocked port 25 are now skipped.
* Tweaks to the intrusion detection rules for IMAP.
* Mail-in-a-Box's setup is a lot quieter, hiding lots of irrelevant messages.
Control panel:
* SSL certificate checks were failing on OVH/OpenVZ servers due to missing /dev/stdin.
* Improve the sort order of the domains in the status checks.
* Some links in the control panel were only working in Chrome.
* let munin choose what it wants to monitor, it may need an update periodically
* deactivate monitoring of NTP peers (this makes no sense)
* deactivate monitoring of network interfaces that aren't up (also makes no sense)
* we get errors in munin-node.log if we don't create /var/lib/munin-node/plugin-state/
The antispam-plugin configuration is not ignoring deleted email (mail sent to a trash folder). This causes a big problem because if someone decides to "clean up" their Spam folder by deleting all of the emails, then sa-learn --ham is run on what is usually correctly caught spam messages. This causes big problems with the accuracy of the bayes scoring! It should really only be learning ham if someone drags the email to a non-Trash folder.
I ran "sa-learn --dbpath /home/user-data/mail/spamassassin --dump magic" before these changes and saw the nham counter increment with every message deleted from Spam. With this new config sa-learn --ham is not run when email is deleted from Spam so the bayes database is not incorrectly trained.
v0.13 (August 19, 2015)
-----------------------
Mail:
* Outbound mail headers (the Recieved: header) are tweaked to possibly improve deliverability.
* Some MIME messages would hang Roundcube due to a missing package.
* The users permitted to send as an alias can now be different from where an alias forwards to.
DNS:
* The secondary nameservers option in the control panel now accepts more than one nameserver and a special xfr:IP format to specify zone-transfer-only IP addresses.
* A TLSA record is added for HTTPS for DNSSEC-aware clients that support it.
System:
* Backups can now be turned off, or stored in Amazon S3, through new control panel options.
* Munin was not working on machines confused about their hostname.
* ownCloud updated to version 8.1.1, its memcached caching enabled.
* When upgrading, network checks like blocked port 25 are now skipped.
* Tweaks to the intrusion detection rules for IMAP.
* Mail-in-a-Box's setup is a lot quieter, hiding lots of irrelevant messages.
Control panel:
* SSL certificate checks were failing on OVH/OpenVZ servers due to missing /dev/stdin.
* Improve the sort order of the domains in the status checks.
* Some links in the control panel were only working in Chrome.
