* chore: Disable `smtputf8` support in config
This was always configured disabled at runtime, better to just set explicitly in `main.cf` unless config diverges when Dovecot is enabled to opt-out of this feature.
* scripts: Install rspamd from official repository instead of debian backports on arm64 architecture
* Remove unnecessary deb-src repository for rspamd
* Remove note about ARM64 rspamd version, update CHANGELOG.md
---------
Co-authored-by: Peter Adam <p.adam@cygnusnetworks.de>
* fix: Source `VERSION` from image ENV
Now CI builds triggered from tagged releases will always have the correct version. No need for manually updating a separate file.
* fix: Query latest GH release tag
Compare to the remote GH release tag published, rather than contents of a `VERSION` file.
`VERSION` file remains in source for now as prior releases still rely on it for an update notification.
* chore: Switch from `yq` to `jaq`
- Can more easily express a string subslice.
- Lighter weight: 9.3M vs 1.7M.
- Drawback, no YAML input/output support.
If `yq` is preferred, the `v` prefix could be removed via BASH easily enough.
* chore: Add entry to `CHANGELOG.md`
* ci: `VERSION` has no relevance to `:edge`
* docs: Update build guide + simplify `make build`
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* added check for Rspamd DKIM on startup
The newly added function `__rspamd__check_dkim_permissions` performs a
check on DKIM private key files. This is useful to prevent issues
like #3621 in the future. The function is deliberately kept simple and
may not catch every single misconfiguration in terms of permissions and
ownership, but it should be quite accurate.
Please note that the Rspamd setup does NOT change at all, and the checks
will not abort the setup in case they fail. A simple warning is emmited.
* add more documentation to Rspamd functions
* Apply suggestions from code review
* improve `__do_as_rspamd_user`
* rework check similar to review suggestion
see https://github.com/docker-mailserver/docker-mailserver/pull/3627#discussion_r1388697547
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* update K8s deployment
Because `allowPrivilegeEscalation` controls SUID/SGID, we require it
when postdrop is invoked.
* correct permissions for maildrop/public
The reason our permissions previously worked out as that in setups where
SUID/SGID worked, the binaries used to place files in these directories
already have SGID set; the current set of permissions makes less sense
(as explained in this comment:
https://github.com/docker-mailserver/docker-mailserver/issues/3619#issuecomment-1793816412)
Since the binaries used to place files inside these directories alredy
have SUID/SGID set, we do not require these bits (or the sticky bit) to
be set on the directories.
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* Dovecot: add deb package dovecot-lua to support Lua scripting
* Adding documentation for Lua authentication
* Updated documentation and made a better distinction between Dovecot packages for officially supported features and for community supported features.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* outsource Rspamd ENVs into explicit helper
This will allow us to uniformly source the helper and get the values
from everywhere consistently. This is more than desirable since we will
be using these values not only for the Rspamd setup, but also for DKIM
management and during change-detection.
* integrate Rspamd into changedetection
We outsource one more function to reside in the helper script for Rspamd
so that we can call this function from the Rspamd setup and from the
changedetection functionality too.
* realize deprecation of old commands file for Rspamd
THIS IS A BREAKING CHANGE!
This change realizes the log message: "Using old file location now
(deprecated) - this will prevent startup in v13.0.0" Startup will now
fail.
* added '--force' option to Rspamd DKIM script
* use new helper to get ENVs for Rspamd in DKIM script
* remove the need for linking directories
This was unnecessary, as explained in
https://github.com/docker-mailserver/docker-mailserver/pull/3597#discussion_r1369413599
* Apply suggestions from code review
review by @polarathene
* apply more review feedback from @polarathene
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370885519>
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370904201>
* update documentation
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of using `etc/rspamd/override.d/dkim_signing.conf`, we will now
be using `/tmp/docker-mailserver/rspamd/override.d/dkim_signing.conf`.
The new location is persisted (and linked again during startup) and
hence better suited.
* simplify `_setup_logrotate`
* adjust Rspamd's log file and improve it's management
* add information to docs about Rspamd log
* update log query helper to allow another file location
* bail in case `LOGROTATE_INTERVAL` is invalid
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Some deployment scenarios are not compatible with `5000:5000` static vmail user with `/var/mail`. This feature allows adjusting the defaults to a UID / GID that is compatible.
Signed-off-by: vincent <vincent@ducamps.win>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Drop management of `SASLAUTHD_*` ENV
- `variables-stack.sh` does not need to manage all these extra ENV or store them. They're not used anywhere else.
- `saslauthd.sh` is the only consumer of these ENV which are effectively direct key/value mappings, with some defaults provided / inherited.
Instead of trying to conditionally support key/value pairs when ENV is set, we could instead use `sed` to delete lines with empty values.
* chore: Drop fallbacks + update configs to match docs
- Drop deprecated support:
- `DOVECOT_HOSTS` is an ENV deprecated since v10.
- Fallback for missing URI scheme introduced for Dovecot and SASLAuthd in v10.
- Adding error log message when no LDAP URI scheme is detected for the supported ENV (when set).
- Docs updated for ENV to reflect the mandatory requirement. `mailserver.env` partially synced equivalent sections.
- Provided base LDAP configs (for overriding) likewise updated from `domain.com` to `example.com`.
- LDAP test updated for required `ldap://` URI scheme. Common ENV shared across LDAP configs hoisted out of the Postfix group.
* chore: Remove unset lines in generated `saslauthd.conf`
- Deprecation startup script check is kept for `ENABLE_LDAP=1` but adjusted to emit an error instead. It can be dropped in a future release. Just a precaution for those who mistakenly update (_possibly via automation_) without checking the release notes, an error log is somewhat helpful, although it could alternatively panic?
- Docs updated to remove the `ENABLE_LDAP=1` usage
- ENV docs updated to reference a maintained LDAP image.
- Changelog includes the breaking change, and slight revision to prior release mention of deprecation.
The new function can
1. update/append
2. update/prepend
3. initialize if non-existent
options in `/etc/postfix/main.cf` in a safe and secure manner. When the
container is improperly restarted, the option is not applied twice.
---
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* Fix issue with concatenating $dmarc_milter and $dkim_milter in main.cf
Upon each start the `smtpd_milters` and `non_smtpd_milters` would be extended with the following:
```
smtpd_milters = $dmarc_milter $dkim_milter
non_smtpd_milters = $dkim_milter
```
In my case they became long enough that mail delivery stopped. I think this was because of the extra headers that are added by these steps. (which in turn would cause the mail to be dropped)
* fix sed to work when the variables are there and when they are not.
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Previously it was assumed the sed operation was applying the sed expressions as a sequence, but it did not seem to filter entries being looked up correctly.
Instead any line that matched either sed expression pattern was output (_value without matching key, values split by the delimiter_), then grep would match any of that causing false-positives.
Resolved by piping the first sed expression into the next.
* add new home dir for Dovecot
I tried changing the mail dir, but this is a _very_ disruptive change,
so I took approach 3 on
<https://doc.dovecot.org/configuration_manual/home_directories_for_virtual_users/>,
whereby the home directory is now inside the mail directory.
The MDBOX/SDBOX formats are not touched by this change. The change
itself could be considered breaking though.
* adjust Sieve tests accordingly
* Update target/dovecot/10-mail.conf
* Update target/dovecot/auth-passwdfile.inc
---------
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* adjust learning of ham
See #3333
When moving a mail from the Junk folder to the Trash folder, the mail
previously classified as ham due to the wildcard match of `*`. Because
the syntax does not allow for negation, we can only change the behavior
in a way that mails are learned as ham when they are moved into `INBOX`
from `Junk`. This is reasonable though.
* adjust tests accordingly
* adjust docs accordingly
* add sanity check for Clam size & adjusted MaxScanSize
The second part is of special importance! See
<https://askubuntu.com/a/1448525>, which explains that the maximum scan
size is important as well. We previously just set the maximum file size,
which actually is pretty insecure as we silently not scan mile bigger
than `MaxScanSize`. This is corrected now.
* add SlamAV size configuration to Rspamd
For an upcoming PR, these changes are required, because the script that
is using the helpers uses `set -eE`. This leads to situations where
errors are not properly handled in our helpers (yet; I plan on changing
that in the future).
* move modules adjustment file to new location
Because we link `/tmp/docker-mailserver/rspamd/override.d` to
`/etc/rspamd/override.d`, I think it makes sense to move the modules
adjustment file into `/tmp/docker-mailserver/rspamd/` as well.
I write the code in a way that it is backwards compatible for now, so
this is NOT a breaking change.
* minor improvement to `__rspamd__handle_user_modules_adjustments`
The expansion of `ARGUMENT3` is now done in a way that only adds the
whitespace in case the variable is set and not null.
* move test file structure to respect latest changes
Because we're now linking `rspamd/override.d/`, we can simplify the
setup a bit. But this requires a change in directory structure.
The current Rspamd test will be renamed to `rspamd_full.bats`, because I
plan on adding more tests in different files for different feature sets.
This is done to make this feature well-tested!
* improved and added tests to Rspamd-full
FYI: The line
```bats
_run_in_container grep 'sieve_global_extensions.*\+vnd\.dovecot\.pipe'
"${SIEVE_CONFIG_FILE}"
```
was testing a condition that should actually not be met, but when I
started working on this feature, I thought this was the correct
configuration. Adding the `assert_success` statements revealed this
wrong line.
I also added tests to check whether `override.d` is linked correctly.
* renamed: `rspamd.bats` => `rspamd_full.bats`
* added new tests for incomplete Rspamd feature set
We now test that warnings are emitted & features are disabled correctly.
* update documentation
* added checks whether OpenDKIM/OpenDMARC/policyd-spf are enabled
* added functions to check if VAR is 0/0 or an int
and also added tests.
I also adjusted the test file to not run in a container, because there
is no need. This also decreases test time, which, in turn, increases
maintainers' happiness.
* added more checks to Rspamd setup
I added the helpers from the previous commit to the Rspamd setup to make
the whole setup more robust, and indicate to the user that an ENV
variable's value is incorrect.
While we did not issues for this in the past, I believe it to be
worthwhile for the future.
* added canonical directory for users to place files in
This dir is canonical with DMS's optional configuration dirs, as it
lives in well-known volume mounts. Hence, users will not need to adjust
`/etc/rspamd/override.d` manually anymore, or mount a volume to this
place.
The docs explain this now, but the DKIM page needs a slight update on
this too I guess. I will follow-up here.
* misc minor improvements
* use variables for common directories
* fixed special bits for maildrop and public dir
After changing the group, special bits are lost, but they should be set for the directories `/var/spool/postfix/{maildrop,public}`, otherwise you see the following error:
```
postfix/postdrop[17400]: warning: mail_queue_enter: create file maildrop/729504.17400: Permission denied
```
* fix: Match octal permissions originally provided
Officially Postfix source seems to imply:
- `730` for `maildrop/` (_but has mentioned a sticky bit in the past, set-gid bit only for the postdrop binary involved_)
- `710` for `public/`
Both folders are assigned the same group that `postdrop` belongs to which has the SGID permission for it's executable. SGID special bit on`public/` doesn't seem necessary, but left as-is to match the default from Debian.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* remove PostSRSD wrapper
The setup is now completely done during _actual_ setup phase. The
wrapper did not even catch signals (SIGINT, etc.), which I think is
strange.
I also added all the ENVs the wrapper relied on (which previously could
have been unset) to the variables script.
* forgot adjusting the `Dockerfile`
* chore: Only replace `CHKSUM_FILE` when a change has been processed
* chore: Change Detection service should be the last daemon started
* chore: Remove 10 second startup delay for change detector
There should be no concern with conflicts as any writes should have already been done by the time this daemon service is started.
* tests(fix): `smtp_delivery.bats` must wait for Amavis
The change event for adding a user can be processed much sooner now, which means Amavis may not yet be ready.
Added extra condition to wait on at least the Amavis port being reachable, and some failure asserts with the mail queue to better catch / debug when this problem occurs.
* chore: Add some minor delay to avoid Amavis failing to connect
* tests(refactor): Make test cases for opendkim keysizes DRY
- These all do roughly the same logic that can be split into two separate methods.
- `_should_generate_dkim_key()` covers a bit more logic as it can be leveraged to handle other test cases that also perform the same logic.
- The `config/opendkim/` doesn't seem necessary for tests. Only the first few test cases here are testing against it, so we can conditionally make that available. `process_check_restart.bats` also depended on it to run OpenDKIM successfully, but this was due to the `setup-stack.sh` config defaults failing to find an "empty" file forcing `supervisord` to constantly restart the process..
- With this, there we inverse the default opendkim config, so we don't have to mount unique / empty subfolders for each test case, followed by copying over the two extra configs.
* tests(refactor): DRY up more test cases
All the remaining test cases but the last one were refactored here for a clean commit diff. The last test case will be refactored in the following commit.
Plenty of repeated logic spread across these test cases, now condensed into shared methods.
* tests(refactor): Make final test case DRY
* chore: Migrate to new testing helpers
* chore: Revise test case descriptions
* tests(refactor): Improve and simplify assertions
* tests(refactor): Use common container setup instead of `docker run`
- As the majority of test cases are only running `open-dkim` helper, we don't actually have to wait for a full container setup. So an alternative container start is called.
- Also improves assertions a bit more instead of just counting lines.
- Some test cases don't bind mount all of `/tmp/docker-mailserver` contents, thus don't raise permission errors on subsequent test runs.
- Instead of `rm -f` on some config files, have opted to mount them read-only instead, or alternatively mount an anonymous empty volume instead.
- Collapsed the first three test cases into one, thus no `setup_file()` necessary.
- Shift the `_wait_for_finished_setup_in_container()` method into `_common_container_setup()` instead since nothing else is using `_common_container_start()` yet, this allows for avoiding the wait.
* tests(refactor): Collapse dkim key size test cases into single test case
This makes these tests a bit more DRY, and enhances the raised quality issue with these tests. Now not only is the domain checked in the generated DNS dkim record, but we also verify the key size is corrected in the public and private keys via openssl.
* chore: Revise container names
* chore: Swap order of test case 1 and 2
* tests(refactor): Assert generated log output
- `__should_have_tables_trustedhosts_for_domain` shifted in each test case to just after generating the domains keys.
- Asserts `open-dkim` logs instead of just counting them.
- Added checks for domains that should not be present in a test case.
- Additional coverage and notes about the alias from vhost `@localdomain.com`
- Single assert statement with switch statement as all are using common args.
* chore: Minor changes
* tests(refactor): Share `find` logic in helpers and tests
* tests(fix): Listing file content does not need to match line order
The order printed from local system vs CI differed causing the CI to fail. The order of lines is irrelevant so `--index` is not required.
Additionally correct the prefix of the called method to be only one `_` now that it's a `common.bash` helper method.
* chore: Collapse custom DKIM selector test into custom DKIM domain test
These cover the same test logic for the most part, the first domain could also be testing the custom selector.
`special_use_folders.bats` + `mailbox_format_dbox` can assert lines instead, removing the need for `--partial`.
* Apply suggestions from code review
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* chore: Split switch statement method into wrapper methods
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* only add Amavis configuration to Postfix when enabled
Since I am running Rspamd nowadays, I noticed there still are ports open
that belong to Amavis. This is because the Amavis configuration is a
fixed part of Postfix's `master.cf`. I changed that. Now, the Amavis
section is added when Amavis really is enabled.
I took the chance and added proper indentation to `master.cf`; hence the
diff is a bit fuzzy. **But**, only the Amavis part was adjusted, the
rest is just styling.
This appears to have been added to replace the `fam` package in an early version of DMS with Courier for IMAP instead of Dovecot on an Ubuntu 14.04 base image.
It does not appear to serve a purpose anymore.
* chore: Remove the wrapper script for `fail2ban`
- This does not appear necessary. The server can be run with foreground mode.
- `daemons-stack.sh` removal of the socket can be handled by the fail2ban server when using the `-x` option.
* chore: Remove `touch /var/log/auth.log`
These were both added as supposed fixes in 2016 for the then Ubuntu 2014 base image.
Removing them causes no failures in tests.
* fix: Install optional python packages for `fail2ban`
These have barely any overhead in layer weight. The DNS package may provide some QoL improvements, while the `pyinotify` is a better alternative than polling logs to check for updates.
We have `gamin` package installed but `fail2ban` would complain in the log that it was not able to initialize the module for it. There only appears to be a `python-gamin` dependent on EOL python 2, no longer available from Debian Bullseye.
Brennan Kinney
Georg Lauterbach
Casper
René Plötz
Peter Adam
Zepmann
Vincent Ducamps
H4R0
wligtenberg
Claude Brisson
Thomas Butter
LucidityCrash
Andreas Perhab
ghnp5
James
Jack Pearson