Merge branch 'master' into feat/env-vars-from-files

CHANGELOG.md

@@ -12,20 +12,12 @@ All notable changes to this project will be documented in this file. The format
 - **saslauthd** mechanism support via ENV `SASLAUTHD_MECHANISMS` with `pam`, `shadow`, `mysql` values has been removed. Only `ldap` and `rimap` remain supported ([#4259](https://github.com/docker-mailserver/docker-mailserver/pull/4259))
 - **getmail6** has been refactored: ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156))
   - The [DMS config volume](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/optional-config/#volumes) now has support for `getmailrc_general.cf` for overriding [common default settings](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/advanced/mail-getmail/#common-options). If you previously mounted this config file directly to `/etc/getmailrc_general` you should switch to our config volume support.
-  - IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail).
-  - ENV [`GETMAIL_POLL`](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/environment/#getmail_poll) now supports values above 30 minutes.
-  - Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling.
   - Generated getmail configuration files no longer set the `message_log` option. Instead of individual log files per config, the [default base settings DMS configures](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/target/getmail/getmailrc_general) now enables `message_log_syslog`. This aligns with how other services in DMS log to syslog where it is captured in `mail.log`.
   - Getmail configurations have changed location from the base of the DMS Config Volume, to the `getmail/` subdirectory. Any existing configurations **must be migrated manually.**
-  - DMS v14 mistakenly relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory.
+  - **DMS v14 mistakenly** relocated the _getmail state directory_ to the _DMS Config Volume_ as a `getmail/` subdirectory.
     - This has been corrected to `/var/lib/getmail` (_if you have mounted a DMS State Volume to `/var/mail-state`, `/var/lib/getmail` will be symlinked to `/var/mail-state/lib-getmail`_).
     - To preserve this state when upgrading to DMS v15, **you must manually migrate `getmail/` from the _DMS Config Volume_ to `lib-getmail/` in the _DMS State Volume_.**
 
-### Security
-
-- **Fail2ban:**
-  - Ensure a secure connection, when downloading the fail2ban package ([#4080](https://github.com/docker-mailserver/docker-mailserver/pull/4080))
-
 ### Added
 
 - **Internal:**
@@ -34,30 +26,40 @@ All notable changes to this project will be documented in this file. The format
 
 ### Updates
 
-- **Removed `VERSION` file** from the repo that releases of DMS prior to v13 (Nov 2023) would check to detect new releases ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677), [#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321))
+**Internal:**
-- **Fail2ban:**
+  - **Removed `VERSION` file** from the repo. Releases of DMS prior to v13 (Nov 2023) would check this to detect new releases ([#3677](https://github.com/docker-mailserver/docker-mailserver/issues/3677), [#4321](https://github.com/docker-mailserver/docker-mailserver/pull/4321))
-  - Updated to version [`1.1.0`](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0) ([#4045](https://github.com/docker-mailserver/docker-mailserver/pull/4045))
+  - During image build, ensure a secure connection when downloading the `fail2ban` package ([#4080](https://github.com/docker-mailserver/docker-mailserver/pull/4080))
 - **Documentation:**
   - Account Management and Authentication pages have been rewritten and better organized ([#4122](https://github.com/docker-mailserver/docker-mailserver/pull/4122))
   - Add a caveat for `DMS_VMAIL_UID` not being compatible with `0` / root ([#4143](https://github.com/docker-mailserver/docker-mailserver/pull/4143))
+- **Getmail:** ([#4156](https://github.com/docker-mailserver/docker-mailserver/pull/4156))
+  - Added `getmail` as a new service for `supervisor` to manage, replacing cron for periodic polling.
+  - IMAP/POP3 example configs added to our [`config-examples`](https://github.com/docker-mailserver/docker-mailserver/tree/v15.0.0/config-examples/getmail).
+  - ENV [`GETMAIL_POLL`](https://docker-mailserver.github.io/docker-mailserver/v15.0/config/environment/#getmail_poll) now supports values above 30 minutes.
 - **Postfix:**
   - By default opt-out from _Microsoft reactions_ for outbound mail ([#4120](https://github.com/docker-mailserver/docker-mailserver/pull/4120))
-- Updated `jaq` version from `1.3.0` to `2.0.0` ([#4190](https://github.com/docker-mailserver/docker-mailserver/pull/4190))
+- **Rspamd:**
-- Updated Rspamd GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191))
+  - Updated GTube settings and tests ([#4191](https://github.com/docker-mailserver/docker-mailserver/pull/4191))
+- Updated externally installed software ([#4357](https://github.com/docker-mailserver/docker-mailserver/pull/4357)):
+  - `DOVECOT_COMMUNITY_REPO=1` custom image build ARG now supports the latest Dovecot [`2.4.x`](https://github.com/dovecot/core/releases/tag/2.4.0) (_DMS provides Dovecot `2.3.19` by default_)
+  - Dovecot FTS Xapian module (`1.7.12` => [`1.9.0`](https://github.com/grosjo/fts-xapian/releases/tag/1.9))
+  - `jaq` (`1.3.0` => [`2.1.0`](https://github.com/01mf02/jaq/releases/tag/v2.1.0))
+  - Fail2Ban (`1.0.2-2` => [`1.1.0`](https://github.com/fail2ban/fail2ban/releases/tag/1.1.0)) ([#4045](https://github.com/docker-mailserver/docker-mailserver/pull/4045))
+  - Rspamd (`3.8.4` => [`3.11.0`](https://github.com/rspamd/rspamd/releases/tag/3.11.0)) - Implicitly upgraded during image build, as the third-party repo lacks version pinning support.
 
 ### Fixes
 
 - **Dovecot:**
   - The logwatch `ignore.conf` now also excludes Xapian messages about pending documents ([#4060](https://github.com/docker-mailserver/docker-mailserver/pull/4060))
-  - `dovecot-fts-xapian` plugin was updated to `1.7.13`, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095))
+  - `dovecot-fts-xapian` plugin was updated, fixing a regression with indexing ([#4095](https://github.com/docker-mailserver/docker-mailserver/pull/4095))
   - The "dummy account" workaround for _Dovecot Quota_ feature support no longer treats the alias as a regex when checking the Dovecot UserDB ([#4222](https://github.com/docker-mailserver/docker-mailserver/pull/4222))
 - **LDAP:**
-  - Correctly apply a compatibility fix for OAuth2 introduced in DMS v13.3.1 which had not been applied to the actual LDAP config changes ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175))
+  - Correctly apply a compatibility fix for OAuth2 introduced in DMS `v13.3.1` which had not been applied to the actual LDAP config changes ([#4175](https://github.com/docker-mailserver/docker-mailserver/pull/4175))
 - **Internal:**
   - The main `mail.log` (_which is piped to stdout via `tail`_) now correctly begins from the first log line of the active container run. Previously some daemon logs and potential warnings/errors were omitted ([#4146](https://github.com/docker-mailserver/docker-mailserver/pull/4146))
   - `start-mailserver.sh` removed unused `shopt -s inherit_errexit` ([#4161](https://github.com/docker-mailserver/docker-mailserver/pull/4161))
-  - Fixed a regression introduced in v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147))
+  - Fixed a regression introduced in DMS v14 where `postfix-main.cf` appended `stderr` output into `/etc/postfix/main.cf`, causing Postfix startup to fail ([#4147](https://github.com/docker-mailserver/docker-mailserver/pull/4147))
-  - Fixed a regression introduced in v14 to better support running `start-mailserver.sh` with container restarts, which now only skip calling `_setup()` ([#4323](https://github.com/docker-mailserver/docker-mailserver/pull/4323#issuecomment-2629559254))
+  - Fixed a regression introduced in DMS v14 to better support running `start-mailserver.sh` with container restarts, which now only skip calling `_setup()` ([#4323](https://github.com/docker-mailserver/docker-mailserver/pull/4323#issuecomment-2629559254))
   - The command `swaks --help` is now functional ([#4282](https://github.com/docker-mailserver/docker-mailserver/pull/4282))
 - **Rspamd:**
   - DKIM private key path checking is now performed only on paths that do not contain `$` ([#4201](https://github.com/docker-mailserver/docker-mailserver/pull/4201))

docs/content/config/best-practices/dkim_dmarc_spf.md

@@ -29,7 +29,7 @@ Cloudflare has written an [article about DKIM, DMARC and SPF][cloudflare-dkim-dm
 When DKIM is enabled:
 
 1. Inbound mail will verify any included DKIM signatures
-2. Outbound mail is signed (_when you're sending domain has a configured DKIM key_)
+2. Outbound mail is signed (_when your sending domain has a configured DKIM key_)
 
 DKIM requires a public/private key pair to enable **signing (_via private key_)** your outgoing mail, while the receiving end must query DNS to **verify (_via public key_)** that the signature is trustworthy.
 

target/scripts/build/compile.sh

@@ -11,28 +11,39 @@ source /usr/local/bin/helpers/log.sh
 # shellcheck disable=SC2310
 _log_level_is 'trace' && QUIET='-y' || QUIET='-qq'
 
-function _compile_dovecot_fts_xapian() {
+function _install_build_deps() {
   apt-get "${QUIET}" update
   apt-get "${QUIET}" install --no-install-recommends \
     automake libtool pkg-config libicu-dev libsqlite3-dev libxapian-dev make build-essential dh-make devscripts dovecot-dev
-
-  local XAPIAN_VERSION='1.7.13'
-  curl -sSfL -o dovecot-fts-xapian.tar.gz \
-    "https://github.com/grosjo/fts-xapian/releases/download/${XAPIAN_VERSION}/dovecot-fts-xapian-${XAPIAN_VERSION}.tar.gz"
-  tar xf dovecot-fts-xapian.tar.gz
-
-  cd "fts-xapian-${XAPIAN_VERSION}"
-  USER=root dh_make -p "dovecot-fts-xapian-${XAPIAN_VERSION}" --single --native --copyright gpl2 -y
-
-  rm debian/*.ex
-  cp PACKAGES/DEB/control debian/
-  cp PACKAGES/DEB/changelog debian/
-  cp PACKAGES/DEB/compat debian/
-
-  sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+|\1-${XAPIAN_VERSION}|g" debian/control
-  sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+ \(.*\)(.*)|\1-${XAPIAN_VERSION} (${XAPIAN_VERSION})\2|g" debian/changelog
-
-  debuild -us -uc -B | tee /tmp/debuild.log 2>&1
 }
 
-_compile_dovecot_fts_xapian
+function _build_package() {
+  local XAPIAN_VERSION='1.9'
+  curl -fsSL "https://github.com/grosjo/fts-xapian/releases/download/${XAPIAN_VERSION}/dovecot-fts-xapian-${XAPIAN_VERSION}.tar.gz" \
+    | tar -xz
+  cd "fts-xapian-${XAPIAN_VERSION}"
+
+  # Prepare for building DEB source package:
+  # https://manpages.debian.org/bookworm/dh-make/dh_make.1.en.html
+  # License LGPL 2.1: https://github.com/grosjo/fts-xapian/issues/174#issuecomment-2422404568
+  USER=root dh_make --packagename "dovecot-fts-xapian-${XAPIAN_VERSION}" --single --native --copyright lgpl2 -y
+  # Remove generated example files:
+  rm debian/*.ex
+  # Add required package metadata:
+  # https://www.debian.org/doc/manuals/maint-guide/dreq.en.html#control
+  curl -fsSL https://raw.githubusercontent.com/grosjo/fts-xapian/refs/tags/1.7.16/PACKAGES/DEB/control > debian/control
+  # Replace version number:
+  sed -i -E "s|(dovecot-fts-xapian)-[1-9\.-]+|\1-${XAPIAN_VERSION}|g" debian/control
+  # Required to proceed with debuild:
+  # https://www.debian.org/doc/manuals/maint-guide/dother.en.html#compat
+  # (13 is the default debhelper version from the original `dh_make` generated `debian/control`):
+  echo '13' > debian/compat
+
+  # Build arch specific binary package via debuild:
+  # https://manpages.debian.org/bookworm/devscripts/debuild.1.en.html
+  # https://manpages.debian.org/bookworm/dpkg-dev/dpkg-buildpackage.1.en.html
+  debuild --no-sign --build=any | tee /tmp/debuild.log 2>&1
+}
+
+_install_build_deps
+_build_package

target/scripts/build/packages.sh

@@ -24,13 +24,14 @@ function _pre_installation_steps() {
   apt-get "${QUIET}" upgrade
 
   _log 'trace' 'Installing packages that are needed early'
-  # add packages usually required by apt to
+  # Add packages usually required by apt to:
-  # - not log unnecessary warnings
-  # - be able to add PPAs early (e.g., Rspamd)
   local EARLY_PACKAGES=(
-    apt-utils # avoid useless warnings
+    # Avoid logging unnecessary warnings:
-    apt-transport-https ca-certificates curl gnupg # required for adding PPAs
+    apt-utils
-    systemd-standalone-sysusers # avoid problems with SA / Amavis (https://github.com/docker-mailserver/docker-mailserver/pull/3403#pullrequestreview-1596689953)
+    # Required for adding third-party repos (/etc/apt/sources.list.d) as alternative package sources (eg: Dovecot CE and Rspamd):
+    apt-transport-https ca-certificates curl gnupg
+    # Avoid problems with SA / Amavis (https://github.com/docker-mailserver/docker-mailserver/pull/3403#pullrequestreview-1596689953):
+    systemd-standalone-sysusers
   )
   apt-get "${QUIET}" install --no-install-recommends "${EARLY_PACKAGES[@]}" 2>/dev/null
 }
@@ -38,7 +39,7 @@ function _pre_installation_steps() {
 function _install_utils() {
   _log 'debug' 'Installing utils sourced from Github'
   _log 'trace' 'Installing jaq'
-  local JAQ_TAG='v2.0.0'
+  local JAQ_TAG='v2.1.0'
   curl -sSfL "https://github.com/01mf02/jaq/releases/download/${JAQ_TAG}/jaq-$(uname -m)-unknown-linux-gnu" -o /usr/bin/jaq
   chmod +x /usr/bin/jaq
 
@@ -136,43 +137,56 @@ function _install_dovecot() {
     dovecot-pop3d dovecot-sieve
   )
 
-  # Dovecot packages for community supported features.
+  # Additional Dovecot packages for supporting the DMS community (docs-only guide contributions).
   DOVECOT_PACKAGES+=(dovecot-auth-lua)
 
-  # Dovecot's deb community repository only provides x86_64 packages, so do not include it
+  # (Opt-in via ENV) Change repo source for dovecot packages to a third-party repo maintained by Dovecot.
-  # when building for another architecture.
+  # NOTE: AMD64 / x86_64 is the only supported arch from the Dovecot CE repo (thus noDMS built for ARM64 / aarch64)
+  # Repo: https://repo.dovecot.org/ce-2.4-latest/debian/bookworm/dists/bookworm/main/
+  # Docs: https://repo.dovecot.org/#debian
   if [[ ${DOVECOT_COMMUNITY_REPO} -eq 1 ]] && [[ "$(uname --machine)" == "x86_64" ]]; then
-    _log 'trace' 'Using Dovecot community repository'
+    # WARNING: Repo only provides Debian Bookworm package support for Dovecot CE 2.4+.
-    curl -sSfL https://repo.dovecot.org/DOVECOT-REPO-GPG | gpg --import
+    # As Debian Bookworm only packages Dovecot 2.3.x, building DMS with this alternative package repo may not yet be compatible with DMS:
-    gpg --export ED409DA1 > /etc/apt/trusted.gpg.d/dovecot.gpg
+    # - 2.3.19: https://salsa.debian.org/debian/dovecot/-/tree/stable/bookworm
-    echo "deb https://repo.dovecot.org/ce-2.3-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/dovecot.list
+    # - 2.3.21: https://salsa.debian.org/debian/dovecot/-/tree/stable/bookworm-backports
 
-    _log 'trace' 'Updating Dovecot package signatures'
+    _log 'trace' 'Adding third-party package repository (Dovecot)'
+    curl -fsSL https://repo.dovecot.org/DOVECOT-REPO-GPG-2.4 | gpg --dearmor > /usr/share/keyrings/upstream-dovecot.gpg
+    echo \
+      "deb [signed-by=/usr/share/keyrings/upstream-dovecot.gpg] https://repo.dovecot.org/ce-2.4-latest/debian/${VERSION_CODENAME} ${VERSION_CODENAME} main" \
+      > /etc/apt/sources.list.d/upstream-dovecot.list
+
+    # Refresh package index:
     apt-get "${QUIET}" update
 
-    # Additional community package needed for Lua support if the Dovecot community repository is used.
+    # This repo instead provides `dovecot-auth-lua` as a transitional package to `dovecot-lua`,
+    # thus this extra package is required to retain lua support:
     DOVECOT_PACKAGES+=(dovecot-lua)
   fi
 
   _log 'debug' 'Installing Dovecot'
   apt-get "${QUIET}" install --no-install-recommends "${DOVECOT_PACKAGES[@]}"
 
-  # dependency for fts_xapian
+  # Runtime dependency for fts_xapian (built via `compile.sh`):
   apt-get "${QUIET}" install --no-install-recommends libxapian30
 }
 
 function _install_rspamd() {
-  _log 'debug' 'Installing Rspamd'
+  # NOTE: DMS only supports the rspamd package via using the third-party repo maintained by Rspamd (AMD64 + ARM64):
-  _log 'trace' 'Adding Rspamd PPA'
+  # Repo: https://rspamd.com/apt-stable/dists/bookworm/main/
-  curl -sSfL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor >/etc/apt/trusted.gpg.d/rspamd.gpg
+  # Docs: https://rspamd.com/downloads.html#debian-and-ubuntu-linux
-  echo \
+  # NOTE: Debian 12 provides Rspamd 3.4 (too old) and Rspamd discourages it's use
-    "deb [signed-by=/etc/apt/trusted.gpg.d/rspamd.gpg] http://rspamd.com/apt-stable/ ${VERSION_CODENAME} main" \
-    >/etc/apt/sources.list.d/rspamd.list
 
-  _log 'trace' 'Updating package index after adding PPAs'
+  _log 'trace' 'Adding third-party package repository (Rspamd)'
+  curl -fsSL https://rspamd.com/apt-stable/gpg.key | gpg --dearmor > /usr/share/keyrings/upstream-rspamd.gpg
+  echo \
+    "deb [signed-by=/usr/share/keyrings/upstream-rspamd.gpg] https://rspamd.com/apt-stable/ ${VERSION_CODENAME} main" \
+    > /etc/apt/sources.list.d/upstream-rspamd.list
+
+  # Refresh package index:
   apt-get "${QUIET}" update
 
-  _log 'trace' 'Installing actual package'
+  _log 'debug' 'Installing Rspamd'
   apt-get "${QUIET}" install rspamd redis-server
 }
 

target/scripts/helpers/utils.sh

@@ -122,7 +122,7 @@ function _reload_postfix() {
 # you can set the environment variable `POSTFIX_README_DIRECTORY='/new/dir/'`
 # (`POSTFIX_` is an arbitrary prefix, you can choose the one you like),
 # and then call this function:
-# `_replace_by_env_in_file 'POSTFIX_' 'PATH TO POSTFIX's main.cf>`
+# `_replace_by_env_in_file 'POSTFIX_' '<PATH TO POSTFIX's main.cf>`
 #
 # ## Panics
 #