diff --git a/CHANGELOG.md b/CHANGELOG.md index ef39eeca..12f07561 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,21 +2,52 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased](https://github.com/docker-mailserver/docker-mailserver/compare/v13.3.0...HEAD) +## [Unreleased](https://github.com/docker-mailserver/docker-mailserver/compare/v13.3.1...HEAD) > **Note**: Changes and additions listed here are contained in the `:edge` image tag. These changes may not be as stable as released changes. +The most noteworthy change of this release is the update of the container's base image from Debian 11 ("Bullseye") to Debian 12 ("Bookworm"). This update alone involves breaking changes and requires a careful update! + +### Breaking + +- **Updated base image to Debian 12** ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403)) + - Changed the default of `DOVECOT_COMMUNITY_REPO` to `0` (disabled) - the Dovecot community repo will (for now) not be the default when building the DMS. + - While Debian 12 (Bookworm) was released in June 2023 and the latest Dovecot `2.3.21` in Sep 2023, as of Jan 2024 there is no [Dovecot community repo available for Debian 12](https://repo.dovecot.org). + - This results in the Dovecot version being downgraded from `2.3.21` (DMS v13.3) to `2.3.19`, which [may affect functionality when you've explicitly configured for these features](https://github.com/dovecot/core/blob/30cde20f63650d8dcc4c7ad45418986f03159946/NEWS#L1-L158): + - OAuth2 (_mostly regarding JWT usage, or POST requests (`introspection_mode = post`) with `client_id` + `client_secret`_). + - Lua HTTP client (_DNS related_). + - Updated packages. For an overview, [we have a review comment on the PR that introduces Debian 12](https://github.com/docker-mailserver/docker-mailserver/pull/3403#issuecomment-1694563615) + - Notable major version bump: `openssl 3`, `clamav 1`, `spamassassin 4`, `redis-server 7`. + - Notable minor version bump: `postfix 3.5.23 => 3.7.9` + - Notable minor version bump + downgrade: `dovecot 2.3.13 => 2.3.19` (_Previous release provided `2.3.21` via community repo, `2.3.19` is now the default_) + - Updates to `packages.sh`: + - The script now uses `/etc/os-release` to determine the release name of Debian + - Removed custom installations of Fail2Ban, getmail6 and Rspamd + - Updated packages lists and added comments for maintainability +- **Postfix:** + - Postfix upgrade from 3.5 to 3.7 ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403)) + - `compatibility_level` was raised from `2` to `3.6` + - Postfix has deprecated the usage of `whitelist` / `blacklist` in config parameters and logging in favor of `allowlist` / `denylist` and similar variations. ([#3403](https://github.com/docker-mailserver/docker-mailserver/pull/3403/files#r1306356328)) + - This [may affect monitoring / analysis of logs output from Postfix](https://www.postfix.org/COMPATIBILITY_README.html#respectful_logging) that expects to match patterns on the prior terminology used. + - DMS `main.cf` has renamed `postscreen_dnsbl_whitelist_threshold` to `postscreen_dnsbl_allowlist_threshold` as part of this change. + - `smtpd_relay_restrictions` (relay policy) is now evaluated after `smtpd_recipient_restrictions` (spam policy). Previously it was evaluated before `smtpd_recipient_restrictions`. Mail to be relayed via DMS must now pass through the spam policy first. + - The TLS fingerprint policy has changed the default from MD5 to SHA256 (_DMS does not modify this Postfix parameter, but may affect any user customizations that do_). + ### Updates - **Tests:** - - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. + - Refactored helper methods for sending e-mails with specific `Message-ID` headers and the helpers for retrieving + filtering logs, which together help isolate logs relevant to specific mail when multiple mails have been processed within a single test. ([#3786](https://github.com/docker-mailserver/docker-mailserver/pull/3786)) + +## [v13.3.1](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.1) ### Fixes -**Dovecot:** +- **Dovecot:** - Restrict the auth mechanisms for PassDB configs we manage (oauth2, passwd-file, ldap) ([#3812](https://github.com/docker-mailserver/docker-mailserver/pull/3812)) - Prevents misleading auth failures from attempting to authenticate against a PassDB with incompatible auth mechanisms. - When the new OAuth2 feature was enabled, it introduced false-positives with logged auth failures which triggered Fail2Ban to ban the IP. +- **Rspamd:** + - Ensure correct ownership (`_rspamd:_rspamd`) for the Rspamd DKIM directory + files `/tmp/docker-mailserver/rspamd/dkim/` ([#3813](https://github.com/docker-mailserver/docker-mailserver/pull/3813)) ## [v13.3.0](https://github.com/docker-mailserver/docker-mailserver/releases/tag/v13.3.0) @@ -158,6 +189,7 @@ This patch release fixes two bugs that Rspamd users encountered with the `v13.0. - **Postfix:** - `/etc/postfix/master.cf` has renamed the "smtps" service to "submissions" ([#3235](https://github.com/docker-mailserver/docker-mailserver/pull/3235)) - This is the modern `/etc/services` name for port 465, aligning with the similar "submission" port 587. + - If you have configured Proxy Protocol support with a reverse proxy via `postfix-master.cf` (_as [per our docs guide](https://docker-mailserver.github.io/docker-mailserver/v13.0/examples/tutorials/mailserver-behind-proxy/)_), you will want to update `smtps` to `submissions` there. - Postfix now defaults to supporting DSNs (_[Delivery Status Notifications](https://github.com/docker-mailserver/docker-mailserver/pull/3572#issuecomment-1751880574)_) only for authenticated users (_via ports 465 + 587_). This is a security measure to reduce spammer abuse of your DMS instance as a backscatter source. ([#3572](https://github.com/docker-mailserver/docker-mailserver/pull/3572)) - If you need to modify this change, please let us know by opening an issue / discussion. - You can [opt out (_enable DSNs_) via the `postfix-main.cf` override support](https://docker-mailserver.github.io/docker-mailserver/v12.1/config/advanced/override-defaults/postfix/) using the contents: `smtpd_discard_ehlo_keywords =`. diff --git a/Dockerfile b/Dockerfile index e822632a..b8bf89bc 100644 --- a/Dockerfile +++ b/Dockerfile @@ -4,10 +4,10 @@ # This is in preparation for more granular stages (eg ClamAV and Fail2Ban split into their own) ARG DEBIAN_FRONTEND=noninteractive -ARG DOVECOT_COMMUNITY_REPO=1 +ARG DOVECOT_COMMUNITY_REPO=0 ARG LOG_LEVEL=trace -FROM docker.io/debian:11-slim AS stage-base +FROM docker.io/debian:12-slim AS stage-base ARG DEBIAN_FRONTEND ARG DOVECOT_COMMUNITY_REPO @@ -30,8 +30,6 @@ COPY target/scripts/helpers/log.sh /usr/local/bin/helpers/log.sh RUN /bin/bash /build/packages.sh && rm -r /build - - # ----------------------------------------------- # --- Compile deb packages ---------------------- # ----------------------------------------------- @@ -130,7 +128,8 @@ COPY \ # hadolint ignore=SC2016 RUN </etc/default/spamassassin sedfile -i -r 's/^\$INIT restart/supervisorctl restart amavis/g' /etc/spamassassin/sa-update-hooks.d/amavisd-new mkdir /etc/spamassassin/kam/ curl -sSfLo /etc/spamassassin/kam/kam.sa-channels.mcgrail.com.key https://mcgrail.com/downloads/kam.sa-channels.mcgrail.com.key @@ -189,7 +188,6 @@ RUN < /dev/null|/usr/bin/supervisorctl signal hup rsyslog >/dev/null|g' /usr/lib/rsyslog/rsyslog-rotate + # this change is for our alternative process manager rather than part of + # a fix related to the change preceding it. + echo -e '\n/usr/bin/supervisorctl signal hup rsyslog >/dev/null' >>/usr/lib/rsyslog/rsyslog-rotate EOF # ----------------------------------------------- diff --git a/VERSION b/VERSION index ac565bc1..c3d10c59 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -13.3.0 +13.3.1 diff --git a/docs/content/config/advanced/mail-sieve.md b/docs/content/config/advanced/mail-sieve.md index b3cef249..759e4f78 100644 --- a/docs/content/config/advanced/mail-sieve.md +++ b/docs/content/config/advanced/mail-sieve.md @@ -4,14 +4,22 @@ title: 'Advanced | Email Filtering with Sieve' ## User-Defined Sieve Filters -[Sieve](http://sieve.info/) allows to specify filtering rules for incoming emails that allow for example sorting mails into different folders depending on the title of an email. -There are global and user specific filters which are filtering the incoming emails in the following order: +!!! warning "Advice may be outdated" -- Global-before -> User specific -> Global-after + This section was contributed by the community some time ago and some configuration examples may be outdated. + +[Sieve][sieve-info] allows to specify filtering rules for incoming emails that allow for example sorting mails into different folders depending on the title of an email. + +!!! info "Global vs User order" + + There are global and user specific filters which are filtering the incoming emails in the following order: + + Global-before -> User specific -> Global-after Global filters are applied to EVERY incoming mail for EVERY email address. -To specify a global Sieve filter provide a `docker-data/dms/config/before.dovecot.sieve` or a `docker-data/dms/config/after.dovecot.sieve` file with your filter rules. -If any filter in this filtering chain discards an incoming mail, the delivery process will stop as well and the mail will not reach any following filters (e.g. global-before stops an incoming spam mail: The mail will get discarded and a user-specific filter won't get applied.) + +- To specify a global Sieve filter provide a `docker-data/dms/config/before.dovecot.sieve` or a `docker-data/dms/config/after.dovecot.sieve` file with your filter rules. +- If any filter in this filtering chain discards an incoming mail, the delivery process will stop as well and the mail will not reach any following filters (e.g. global-before stops an incoming spam mail: The mail will get discarded and a user-specific filter won't get applied.) To specify a user-defined Sieve filter place a `.dovecot.sieve` file into a virtual user's mail folder (e.g. `/var/mail/example.com/user1/home/.dovecot.sieve`). If this file exists dovecot will apply the filtering rules. @@ -32,6 +40,7 @@ An example of a sieve filter that moves mails to a folder `INBOX/spam` depending ``` !!! warning + That folders have to exist beforehand if sieve should move them. Another example of a sieve filter that forward mails to a different address: @@ -52,14 +61,25 @@ Just forward all incoming emails and do not save them locally: redirect "user2@not-example.com"; ``` -You can also use external programs to filter or pipe (process) messages by adding executable scripts in `docker-data/dms/config/sieve-pipe` or `docker-data/dms/config/sieve-filter`. This can be used in lieu of a local alias file, for instance to forward an email to a webservice. These programs can then be referenced by filename, by all users. Note that the process running the scripts run as a privileged user. For further information see [Dovecot's wiki](https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/Pipe). +You can also use external programs to filter or pipe (process) messages by adding executable scripts in `docker-data/dms/config/sieve-pipe` or `docker-data/dms/config/sieve-filter`. + +This can be used in lieu of a local alias file, for instance to forward an email to a webservice. + +- These programs can then be referenced by filename, by all users. +- Note that the process running the scripts run as a privileged user. +- For further information see [Dovecot's docs][dovecot-docs::sieve-pipe]. ```sieve require ["vnd.dovecot.pipe"]; pipe "external-program"; ``` -For more examples or a detailed description of the Sieve language have a look at [the official site](http://sieve.info/examplescripts). Other resources are available on the internet where you can find several [examples](https://support.tigertech.net/sieve#sieve-example-rules-jmp). +For more examples or a detailed description of the Sieve language have a look at [the official site][sieve-info::examples]. Other resources are available on the internet where you can find several [examples][third-party::sieve-examples]. + +[dovecot-docs::sieve-pipe]: https://doc.dovecot.org/configuration_manual/sieve/plugins/extprograms/#pigeonhole-plugin-extprograms +[sieve-info]: http://sieve.info/ +[sieve-info::examples]: http://sieve.info/examplescripts +[third-party::sieve-examples]: https://support.tigertech.net/sieve#sieve-example-rules-jmp ## Automatic Sorting Based on Subaddresses diff --git a/target/bin/debug-getmail b/target/bin/debug-getmail index c007ebd2..27834475 100644 --- a/target/bin/debug-getmail +++ b/target/bin/debug-getmail @@ -15,5 +15,5 @@ else fi for FILE in /etc/getmailrc.d/getmailrc*; do - /usr/local/bin/getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +7 + getmail --getmaildir "${GETMAILDIR}" --rcfile "${FILE}" --dump | tail -n +6 done diff --git a/target/bin/getmail-cron b/target/bin/getmail-cron index 8f31e3b1..8e1e4749 100644 --- a/target/bin/getmail-cron +++ b/target/bin/getmail-cron @@ -2,6 +2,6 @@ for FILE in /etc/getmailrc.d/getmailrc*; do if ! pgrep -f "${FILE}$" &>/dev/null; then - /usr/local/bin/getmail --getmaildir /var/lib/getmail --rcfile "${FILE}" + getmail --getmaildir /var/lib/getmail --rcfile "${FILE}" fi done diff --git a/target/postfix/main.cf b/target/postfix/main.cf index 8c1d4c89..a0d805cc 100644 --- a/target/postfix/main.cf +++ b/target/postfix/main.cf @@ -86,7 +86,7 @@ postscreen_dnsbl_sites = list.dnswl.org=127.0.[0..255].1*-3 list.dnswl.org=127.0.[0..255].[2..3]*-4 postscreen_dnsbl_threshold = 3 -postscreen_dnsbl_whitelist_threshold = -1 +postscreen_dnsbl_allowlist_threshold = -1 postscreen_greet_action = enforce postscreen_bare_newline_action = enforce @@ -121,7 +121,4 @@ smtp_header_checks = pcre:/etc/postfix/maps/sender_header_filter.pcre # http://www.postfix.org/COMPATIBILITY_README.html # If backwards-compaitibilty log messages appear, fix them by explicitly adding # the legacy or new default value (alternatively raise the compatibility_level) -# -# TODO: The next compatibility_level is 3.6, when Postfix 3.6 is available consider -# bumping this value after taking the compaitibilty changes into account. -compatibility_level = 2 +compatibility_level = 3.6 diff --git a/target/scripts/build/compile.sh b/target/scripts/build/compile.sh index a7010ba5..56feea72 100644 --- a/target/scripts/build/compile.sh +++ b/target/scripts/build/compile.sh @@ -17,7 +17,7 @@ function _compile_dovecot_fts_xapian() { tar xzvf dovecot-fts-xapian.tar.gz cd fts-xapian-1.5.5 USER=root dh_make -p dovecot-fts-xapian-1.5.5 --single --native --copyright gpl2 -y - rm debian/*.ex debian/*.EX + rm debian/*.ex cp PACKAGES/DEB/control debian/ cp PACKAGES/DEB/changelog debian/ cp PACKAGES/DEB/compat debian/ diff --git a/target/scripts/build/packages.sh b/target/scripts/build/packages.sh index e57cfe07..24dae8a7 100644 --- a/target/scripts/build/packages.sh +++ b/target/scripts/build/packages.sh @@ -1,10 +1,13 @@ #!/bin/bash -# -eE :: exit on error (do this in functions as well) -# -u :: show (and exit) when using unset variables +# -e :: exit on error (do this in functions as well) +# -E :: inherit the ERR trap to functions, command substitutions and sub-shells +# -u :: show (and exit) when using unset variables # -o pipefail :: exit on error in pipes set -eE -u -o pipefail +VERSION_CODENAME='bookworm' + # shellcheck source=../helpers/log.sh source /usr/local/bin/helpers/log.sh @@ -17,17 +20,38 @@ function _pre_installation_steps() { _log 'trace' 'Updating package signatures' apt-get "${QUIET}" update - _log 'trace' 'Installing packages that are needed early' - apt-get "${QUIET}" install --no-install-recommends apt-utils 2>/dev/null - _log 'trace' 'Upgrading packages' apt-get "${QUIET}" upgrade + + _log 'trace' 'Installing packages that are needed early' + # add packages usually required by apt to + # - not log unnecessary warnings + # - be able to add PPAs early (e.g., Rspamd) + local EARLY_PACKAGES=( + apt-utils # avoid useless warnings + apt-transport-https ca-certificates curl gnupg # required for adding PPAs + systemd-standalone-sysusers # avoid problems with SA / Amavis (https://github.com/docker-mailserver/docker-mailserver/pull/3403#pullrequestreview-1596689953) + ) + apt-get "${QUIET}" install --no-install-recommends "${EARLY_PACKAGES[@]}" 2>/dev/null +} + +function _install_utils() { + _log 'debug' 'Installing utils sourced from Github' + _log 'trace' 'Installing jaq' + curl -sSfL "https://github.com/01mf02/jaq/releases/latest/download/jaq-v1.2.0-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq + + _log 'trace' 'Installing swaks' + local SWAKS_VERSION='20240103.0' + local SWAKS_RELEASE="swaks-${SWAKS_VERSION}" + curl -sSfL "https://github.com/jetmore/swaks/releases/download/v${SWAKS_VERSION}/${SWAKS_RELEASE}.tar.gz" | tar -xz + mv "${SWAKS_RELEASE}/swaks" /usr/local/bin + rm -r "${SWAKS_RELEASE}" } function _install_postfix() { _log 'debug' 'Installing Postfix' - _log 'warn' 'Applying workaround for Postfix bug (see https://github.com//issues/2023#issuecomment-855326403)' + _log 'warn' 'Applying workaround for Postfix bug (see https://github.com/docker-mailserver/docker-mailserver/issues/2023#issuecomment-855326403)' # Debians postfix package has a post-install script that expects a valid FQDN hostname to work: mv /bin/hostname /bin/hostname.bak @@ -43,12 +67,17 @@ function _install_postfix() { function _install_packages() { _log 'debug' 'Installing all packages now' - ANTI_VIRUS_SPAM_PACKAGES=( - amavisd-new clamav clamav-daemon - pyzor razor spamassassin + local ANTI_VIRUS_SPAM_PACKAGES=( + clamav clamav-daemon + # spamassassin is used only with amavisd-new, while pyzor + razor are used by spamassasin + amavisd-new spamassassin pyzor razor + # the following packages are all for Fail2Ban + # https://github.com/docker-mailserver/docker-mailserver/pull/3403#discussion_r1306581431 + fail2ban python3-pyinotify python3-dnspython ) - CODECS_PACKAGES=( + # predominantly for Amavis support + local CODECS_PACKAGES=( altermime arj bzip2 cabextract cpio file gzip lhasa liblz4-tool @@ -57,26 +86,33 @@ function _install_packages() { unrar-free unzip xz-utils ) - MISCELLANEOUS_PACKAGES=( - apt-transport-https binutils bsd-mailx - ca-certificates curl dbconfig-no-thanks - dumb-init gnupg iproute2 libdate-manip-perl - libldap-common libmail-spf-perl - libnet-dns-perl locales logwatch - netcat-openbsd nftables rsyslog - supervisor uuid whois + local MISCELLANEOUS_PACKAGES=( + binutils bsd-mailx + dbconfig-no-thanks dumb-init iproute2 + libdate-manip-perl libldap-common libmail-spf-perl libnet-dns-perl + locales logwatch netcat-openbsd + nftables # primarily for Fail2Ban + rsyslog supervisor + uuid # used for file-locking + whois ) - POSTFIX_PACKAGES=( + local POSTFIX_PACKAGES=( pflogsumm postgrey postfix-ldap postfix-mta-sts-resolver postfix-pcre postfix-policyd-spf-python postsrsd ) - MAIL_PROGRAMS_PACKAGES=( - fetchmail opendkim opendkim-tools + local MAIL_PROGRAMS_PACKAGES=( + opendkim opendkim-tools opendmarc libsasl2-modules sasl2-bin ) + # These packages support community contributed features. + # If they cause too much maintenance burden in future, they are liable for removal. + local COMMUNITY_PACKAGES=( + fetchmail getmail6 + ) + # `bind9-dnsutils` provides the `dig` command # `iputils-ping` provides the `ping` command DEBUG_PACKAGES=( @@ -89,14 +125,12 @@ function _install_packages() { "${MISCELLANEOUS_PACKAGES[@]}" \ "${POSTFIX_PACKAGES[@]}" \ "${MAIL_PROGRAMS_PACKAGES[@]}" \ - "${DEBUG_PACKAGES[@]}" + "${DEBUG_PACKAGES[@]}" \ + "${COMMUNITY_PACKAGES[@]}" } function _install_dovecot() { - declare -a DOVECOT_PACKAGES - - # Dovecot packages for officially supported features. - DOVECOT_PACKAGES=( + local DOVECOT_PACKAGES=( dovecot-core dovecot-imapd dovecot-ldap dovecot-lmtpd dovecot-managesieved dovecot-pop3d dovecot-sieve dovecot-solr @@ -111,7 +145,8 @@ function _install_dovecot() { _log 'trace' 'Using Dovecot community repository' curl https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg - echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/bullseye bullseye main" > /etc/apt/sources.list.d/dovecot.list + # VERSION_CODENAME sourced from /etc/os-release + echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/dovecot.list _log 'trace' 'Updating Dovecot package signatures' apt-get "${QUIET}" update @@ -128,47 +163,31 @@ function _install_dovecot() { } function _install_rspamd() { - _log 'trace' 'Adding Rspamd package signatures' - local DEB_FILE='/etc/apt/sources.list.d/rspamd.list' - - curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg - local URL='[signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ bullseye main' - echo "deb ${URL}" >"${DEB_FILE}" - _log 'debug' 'Installing Rspamd' + _log 'trace' 'Adding Rspamd PPA' + curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg + echo \ + "deb [signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ ${VERSION_CODENAME} main" \ + >/etc/apt/sources.list.d/rspamd.list + + _log 'trace' 'Updating package index after adding PPAs' apt-get "${QUIET}" update - apt-get "${QUIET}" --no-install-recommends install 'rspamd' 'redis-server' + + _log 'trace' 'Installing actual package' + apt-get "${QUIET}" install rspamd redis-server } -function _install_fail2ban() { - local FAIL2BAN_DEB_URL='https://github.com/fail2ban/fail2ban/releases/download/1.0.2/fail2ban_1.0.2-1.upstream1_all.deb' - local FAIL2BAN_DEB_ASC_URL="${FAIL2BAN_DEB_URL}.asc" - local FAIL2BAN_GPG_FINGERPRINT='8738 559E 26F6 71DF 9E2C 6D9E 683B F1BE BD0A 882C' - local FAIL2BAN_GPG_PUBLIC_KEY_ID='0x683BF1BEBD0A882C' - local FAIL2BAN_GPG_PUBLIC_KEY_SERVER='hkps://keyserver.ubuntu.com' +function _post_installation_steps() { + _log 'debug' 'Running post-installation steps (cleanup)' + _log 'debug' 'Deleting sensitive files (secrets)' + rm /etc/postsrsd.secret - _log 'debug' 'Installing Fail2ban' - apt-get "${QUIET}" --no-install-recommends install python3-pyinotify python3-dnspython + _log 'debug' 'Deleting default logwatch cronjob' + rm /etc/cron.daily/00logwatch - gpg --keyserver "${FAIL2BAN_GPG_PUBLIC_KEY_SERVER}" --recv-keys "${FAIL2BAN_GPG_PUBLIC_KEY_ID}" 2>&1 - - curl -Lkso fail2ban.deb "${FAIL2BAN_DEB_URL}" - curl -Lkso fail2ban.deb.asc "${FAIL2BAN_DEB_ASC_URL}" - - FINGERPRINT=$(LANG=C gpg --verify fail2ban.deb.asc fail2ban.deb |& sed -n 's#Primary key fingerprint: \(.*\)#\1#p') - - if [[ -z ${FINGERPRINT} ]]; then - echo 'ERROR: Invalid GPG signature!' >&2 - exit 1 - fi - - if [[ ${FINGERPRINT} != "${FAIL2BAN_GPG_FINGERPRINT}" ]]; then - echo "ERROR: Wrong GPG fingerprint!" >&2 - exit 1 - fi - - dpkg -i fail2ban.deb 2>&1 - rm fail2ban.deb fail2ban.deb.asc + _log 'trace' 'Removing leftovers from APT' + apt-get "${QUIET}" clean + rm -rf /var/lib/apt/lists/* _log 'debug' 'Patching Fail2ban to enable network bans' # Enable network bans @@ -176,56 +195,10 @@ function _install_fail2ban() { sedfile -i -r 's/^_nft_add_set = .+/_nft_add_set = add set \\{ type \\; flags interval\\; \\}/' /etc/fail2ban/action.d/nftables.conf } -# Presently the getmail6 package is v6.14, which is too old. -# v6.18 contains fixes for Google and Microsoft OAuth support. -# using pip to install getmail. -# TODO This can be removed when the base image is updated to Debian 12 (Bookworm) -function _install_getmail() { - _log 'debug' 'Installing getmail6' - apt-get "${QUIET}" --no-install-recommends install python3-pip - pip3 install --no-cache-dir 'getmail6~=6.18.12' - ln -s /usr/local/bin/getmail /usr/bin/getmail - ln -s /usr/local/bin/getmail-gmail-xoauth-tokens /usr/bin/getmail-gmail-xoauth-tokens - apt-get "${QUIET}" purge python3-pip - apt-get "${QUIET}" autoremove -} - -function _install_utils() { - _log 'debug' 'Installing utils sourced from Github' - _log 'trace' 'Installing jaq' - curl -sL "https://github.com/01mf02/jaq/releases/latest/download/jaq-v1.2.0-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq && chmod +x /usr/bin/jaq - - _log 'trace' 'Installing swaks' - local SWAKS_VERSION='20240103.0' - local SWAKS_RELEASE="swaks-${SWAKS_VERSION}" - curl -sSfL "https://github.com/jetmore/swaks/releases/download/v${SWAKS_VERSION}/${SWAKS_RELEASE}.tar.gz" | tar -xz - mv "${SWAKS_RELEASE}/swaks" /usr/local/bin - rm -r "${SWAKS_RELEASE}" -} - -function _remove_data_after_package_installations() { - _log 'debug' 'Deleting sensitive files (secrets)' - rm /etc/postsrsd.secret - - _log 'debug' 'Deleting default logwatch cronjob' - rm /etc/cron.daily/00logwatch -} - -function _post_installation_steps() { - _log 'debug' 'Running post-installation steps (cleanup)' - apt-get "${QUIET}" clean - rm -rf /var/lib/apt/lists/* - - _log 'info' 'Finished installing packages' -} - _pre_installation_steps +_install_utils _install_postfix _install_packages _install_dovecot _install_rspamd -_install_fail2ban -_install_getmail -_install_utils -_remove_data_after_package_installations _post_installation_steps diff --git a/target/scripts/check-for-changes.sh b/target/scripts/check-for-changes.sh index 66417c09..2fa004b0 100755 --- a/target/scripts/check-for-changes.sh +++ b/target/scripts/check-for-changes.sh @@ -88,7 +88,7 @@ function _reload_amavis() { if [[ ${CHANGED} =~ ${DMS_DIR}/postfix-accounts.cf ]] || [[ ${CHANGED} =~ ${DMS_DIR}/postfix-virtual.cf ]]; then # /etc/postfix/vhost was updated, amavis must refresh it's config by # reading this file again in case of new domains, otherwise they will be ignored. - amavisd-new reload + amavisd reload fi } diff --git a/target/scripts/startup/setup.d/security/misc.sh b/target/scripts/startup/setup.d/security/misc.sh index 36d8905f..ec32a1e6 100644 --- a/target/scripts/startup/setup.d/security/misc.sh +++ b/target/scripts/startup/setup.d/security/misc.sh @@ -81,11 +81,6 @@ function __setup__security__spamassassin() { # shellcheck disable=SC2016 sed -i -r 's|^\$sa_kill_level_deflt (.*);|\$sa_kill_level_deflt = '"${SA_KILL}"';|g' /etc/amavis/conf.d/20-debian_defaults - # fix cron.daily for spamassassin - sed -i \ - 's|invoke-rc.d spamassassin reload|/etc/init\.d/spamassassin reload|g' \ - /etc/cron.daily/spamassassin - if [[ ${SA_SPAM_SUBJECT} == 'undef' ]]; then # shellcheck disable=SC2016 sed -i -r 's|^\$sa_spam_subject_tag (.*);|\$sa_spam_subject_tag = undef;|g' /etc/amavis/conf.d/20-debian_defaults diff --git a/target/supervisor/conf.d/supervisor-app.conf b/target/supervisor/conf.d/supervisor-app.conf index d64d3d72..50a7acc3 100644 --- a/target/supervisor/conf.d/supervisor-app.conf +++ b/target/supervisor/conf.d/supervisor-app.conf @@ -94,7 +94,7 @@ autostart=false autorestart=true stdout_logfile=/var/log/supervisor/%(program_name)s.log stderr_logfile=/var/log/supervisor/%(program_name)s.log -command=/usr/sbin/amavisd-new foreground +command=/usr/sbin/amavisd foreground [program:rspamd] startsecs=0 diff --git a/test/tests/parallel/set1/getmail.bats b/test/tests/parallel/set1/getmail.bats index 4b5c528a..ca1fdf87 100644 --- a/test/tests/parallel/set1/getmail.bats +++ b/test/tests/parallel/set1/getmail.bats @@ -25,9 +25,9 @@ function teardown_file() { _default_teardown ; } assert_line 'received = false' assert_line 'delivered_to = false' - _run_in_container stat /usr/local/bin/debug-getmail + _run_in_container_bash '[[ -f /usr/local/bin/debug-getmail ]]' assert_success - _run_in_container stat /usr/local/bin/getmail-cron + _run_in_container_bash '[[ -f /usr/local/bin/getmail-cron ]]' assert_success } diff --git a/test/tests/parallel/set1/spam_virus/amavis.bats b/test/tests/parallel/set1/spam_virus/amavis.bats index bc920234..d7a59cb2 100644 --- a/test/tests/parallel/set1/spam_virus/amavis.bats +++ b/test/tests/parallel/set1/spam_virus/amavis.bats @@ -51,10 +51,8 @@ function teardown_file() { export CONTAINER_NAME=${CONTAINER1_NAME} # give Amavis just a bit of time to print out its full debug log - run _repeat_in_container_until_success_or_timeout 5 "${CONTAINER_NAME}" grep 'ANTI-SPAM-SA' /var/log/mail/mail.log + run _repeat_in_container_until_success_or_timeout 20 "${CONTAINER_NAME}" grep 'SpamControl: init_pre_fork on SpamAssassin done' /var/log/mail/mail.log assert_success - assert_output --partial 'loaded' - refute_output --partial 'NOT loaded' } @test '(Amavis enabled) SA ENV should update Amavis config' { diff --git a/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats b/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats index ee26802a..b85d18dc 100644 --- a/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats +++ b/test/tests/parallel/set1/spam_virus/disabled_clamav_spamassassin.bats @@ -29,8 +29,12 @@ function teardown_file() { _default_teardown ; } } @test "SA - Amavis integration should not be active" { - _service_log_should_contain_string 'mail' 'ANTI-SPAM-SA code' - assert_output --partial 'NOT loaded' + # Wait until Amavis has finished initializing: + run _repeat_in_container_until_success_or_timeout 20 "${CONTAINER_NAME}" grep 'Deleting db files in /var/lib/amavis/db' /var/log/mail/mail.log + assert_success + + # Amavis module for SA should not be loaded (`SpamControl: scanner SpamAssassin, module Amavis::SpamControl::SpamAssassin`): + _service_log_should_contain_string 'mail' 'scanner SpamAssassin' } @test "SA - should not have been called" { diff --git a/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats b/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats index 5538d3bf..2609812e 100644 --- a/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats +++ b/test/tests/parallel/set1/spam_virus/postgrey_enabled.bats @@ -59,7 +59,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=greylist' \ 'reason=new' \ - 'client_address=127.0.0.1/32, sender=user@external.tld, recipient=user1@localhost.localdomain' + 'client_address=127.0.0.1, sender=user@external.tld, recipient=user1@localhost.localdomain' } # NOTE: This test case depends on the previous one @@ -73,7 +73,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=pass' \ 'reason=triplet found' \ - 'client_address=127.0.0.1/32, sender=user@external.tld, recipient=user1@localhost.localdomain' + 'client_address=127.0.0.1, sender=user@external.tld, recipient=user1@localhost.localdomain' } # NOTE: These two whitelist tests use `files/nc/` instead of `files/emails`. @@ -91,7 +91,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=pass' \ 'reason=client whitelist' \ - 'client_address=127.0.0.1/32, sender=test@whitelist.tld, recipient=user1@localhost.localdomain' + 'client_address=127.0.0.1, sender=test@whitelist.tld, recipient=user1@localhost.localdomain' } @test "should whitelist recipient 'user2@otherdomain.tld'" { @@ -100,7 +100,7 @@ function teardown_file() { _default_teardown ; } _should_have_log_entry \ 'action=pass' \ 'reason=recipient whitelist' \ - 'client_address=127.0.0.1/32, sender=test@nonwhitelist.tld, recipient=user2@otherdomain.tld' + 'client_address=127.0.0.1, sender=test@nonwhitelist.tld, recipient=user2@otherdomain.tld' } function _should_have_log_entry() { diff --git a/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats b/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats index 215b334d..044e38fb 100644 --- a/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats +++ b/test/tests/parallel/set1/spam_virus/rspamd_dkim.bats @@ -222,7 +222,7 @@ function __check_rsa_keys() { # Check the private key matches the specification _run_in_container_bash "openssl rsa -in '${BASE_FILE_NAME}.private.txt' -noout -text" assert_success - assert_line --index 0 "RSA Private-Key: (${KEYSIZE} bit, 2 primes)" + assert_line --index 0 "Private-Key: (${KEYSIZE} bit, 2 primes)" # Check the public key matches the specification # @@ -232,7 +232,7 @@ function __check_rsa_keys() { PUBKEY=$(_exec_in_container_bash "grep -o 'p=.*' ${BASE_FILE_NAME}.public.dns.txt") _run_in_container_bash "openssl enc -base64 -d <<< ${PUBKEY#p=} | openssl pkey -inform DER -pubin -noout -text" assert_success - assert_line --index 0 "RSA Public-Key: (${KEYSIZE} bit)" + assert_line --index 0 "Public-Key: (${KEYSIZE} bit)" } # Verify that all DKIM key files are present. diff --git a/test/tests/parallel/set3/container_configuration/process_check_restart.bats b/test/tests/parallel/set3/container_configuration/process_check_restart.bats index 0a54a60f..4f0fd90f 100644 --- a/test/tests/parallel/set3/container_configuration/process_check_restart.bats +++ b/test/tests/parallel/set3/container_configuration/process_check_restart.bats @@ -13,7 +13,7 @@ function teardown() { _default_teardown ; } # opendmarc (/usr/sbin/opendmarc) # postfix (/usr/lib/postfix/sbin/master) - Postfix main process (two ancestors, launched via pidproxy python3 script) # -# amavisd-new (usr/sbin/amavisd-new) +# amavisd (usr/sbin/amavisd) # clamd (/usr/sbin/clamd) # dovecot (/usr/sbin/dovecot) # fetchmail (/usr/bin/fetchmail) @@ -37,7 +37,7 @@ CORE_PROCESS_LIST=( # These processes can be toggled via ENV: # NOTE: clamd handled in separate test case ENV_PROCESS_LIST=( - amavisd-new + amavisd dovecot fail2ban-server fetchmail diff --git a/test/tests/serial/mail_with_oauth2.bats b/test/tests/serial/mail_with_oauth2.bats index 19693e34..c9a02eea 100644 --- a/test/tests/serial/mail_with_oauth2.bats +++ b/test/tests/serial/mail_with_oauth2.bats @@ -59,6 +59,10 @@ function teardown_file() { } @test "should authenticate with XOAUTH2" { + # curl packaged in Debian 12 (and the latest release as of Jan 2024) broke XOAUTH2 support + # https://github.com/docker-mailserver/docker-mailserver/pull/3403#issuecomment-1907100624 + skip 'unable to test XOAUTH mechanism due to bug since curl 7.80' + __should_login_successfully_with 'XOAUTH2' } diff --git a/test/tests/serial/open_dkim.bats b/test/tests/serial/open_dkim.bats index b4950d1f..e51fdedb 100644 --- a/test/tests/serial/open_dkim.bats +++ b/test/tests/serial/open_dkim.bats @@ -226,7 +226,7 @@ function __should_have_expected_files() { # DKIM private key for signing, parse it to verify private key size is correct: _run_in_container_bash "openssl rsa -in '${TARGET_DIR}/mail.private' -noout -text" assert_success - assert_line --index 0 "RSA Private-Key: (${EXPECTED_KEYSIZE} bit, 2 primes)" + assert_line --index 0 "Private-Key: (${EXPECTED_KEYSIZE} bit, 2 primes)" # DKIM record, extract public key (base64 encoded, potentially multi-line) # - tail to exclude first line, @@ -240,7 +240,7 @@ function __should_have_expected_files() { ) | openssl enc -base64 -d | openssl pkey -inform DER -pubin -noout -text " assert_success - assert_line --index 0 "RSA Public-Key: (${EXPECTED_KEYSIZE} bit)" + assert_line --index 0 "Public-Key: (${EXPECTED_KEYSIZE} bit)" # Contents is for expected DKIM_DOMAIN and selector (mail): _run_in_container cat "${TARGET_DIR}/mail.txt"