Refactoring this `setup` CLI command as part of the effort to unify our DKIM feature support between OpenDKIM + Rspamd:
- Adds a `main()` method similar to other setup CLI commands.
- Help text more aligned with equivalent rspamd DKIM setup CLI command.
- DRY some repetition such as hard-coded paths to use variables.
- OpenDKIM config files are created / initialized early on now with `_create_opendkim_configs()`. `while` loop only needs to append entries, so is easier to grok.
- `_create_dkim_key()` to scope just the logic (_and additional notes_) to key generation via `opendkim-genkey`
- Now overall logic with the `while` loop of the script occurs in `_generate_dkim_keys()`:
- Ownership fixes are now applied after the `while` loop as that seems more appropriate than per iteration.
- Temporary VHOST config is now removed since it's no longer useful after running.
- Tests adjusted for one new log for adding of default trusted hosts content.
Overall this should be nicer to grok/maintain. Some of this logic will be reused for the unified DKIM generation command in future, which is more likely to shift towards all domains using the same keypair by default with rspamd/opendkim config generated at runtime rather than reliant upon DMS config volume to provide that (_still expected for private key_).
---------
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Custom parameters must be referenced to be retained when `postconf -n` is run. If those parameters are referenced by `postfix-master.cf` this needs to update `master.cf` before updating `main.cf`.
* chore: Detect missing final newline in configs read
These lines will be not be processed by `read`, emit a warning to raise awareness.
* fix: Ensure parsed config has final newline appended (when possible)
This functionality was handled in `accounts.sh` via a similar sed command (that the linked references also offer).
`printf` is better for this, no shellcheck comment required either.
We additionally don't attempt to modify files that are read-only.
* fix: Ensure parsed configs have CRLF to LF corrected (where possible)
Likewise, this runtime fix was only covering two config files. It now applies to all callers of this method.
* fix: Sanitize `postfix-master.cf` via helper
This feature should have been using the helper to avoid user error from their config updates accidentally introducing subtle breakage implicitly (due to CRLF or missing final newline).
* tests: Add test cases for new helpers
* tests: `rm` is redundant when using `BATS_TEST_TMPDIR`
This temporary directory is created and removed implicitly. Even after a test failure.
* chore: Remove old `postfix-virtual.cf` migration logic
This was introduced in 2018, there should be no one needing to rely on this anymore?
* tests: Remove comment on sed failure concern
* chore: Add entry to `CHANGELOG.md`
* Apply suggestions from code review
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* move log/filter functions into own file
* add ShellCheck global directives
* use new function for tracking logs
The new function, called `_send_email_with_mid`, aligns with suggestions
from @polarethene and is heavily simplified compared to its predecessor
`_send_email_and_get_id`. New helpers will be introduced to filter logs
according to the MID constructed in this function.
* new filters for searching logs with MID
* use new filters (and sending) functions
* add new helper for asserting non-existence of log message
* use new filters in tests
* Apply suggestions from code review
- `_mid` / `MID` => `_msgid` / `MSG_ID`
- Revised documentation / tooltip comments
* Apply suggestions from code review
* fix tests
* use more distinct names for MSG_ID headers
* update `_filter_service_log` to not use `-i -E`
Moreover, I added a function to print the whole mail log. Appropriate
comments were added to this function to indicate that one should only
use this function when necessary.
* adjust helpers to new helper filter
* follow-up of previous commit
* add CHANGELOG entry
* Apply suggestions from code review
* chore: Update OAuth2 to use new log helper
* Apply suggestions from code review
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* added explicit `_regexp` filters for logs
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* fix: Dovecot PassDB should restrict allowed auth mechanisms
This prevents PassDBs incompatible with certain auth mechanisms from logging failures which accidentally triggers Fail2Ban.
Instead only allow the PassDB to be authenticated against when it's compatible with the auth mechanism used.
* tests: Use `curl` for OAuth2 login test-cases instead of netcat
`curl` provides this capability for both IMAP and SMTP authentication with a bearer token. It supports both `XOAUTH2` and `OAUTHBEARER` mechanisms, as these updated test-cases demonstrate.
* chore: Add entry to `CHANGELOG.md`
* tests: OAuth2 - Replace Python `/userinfo` endpoint with Caddy
Better documented, easier flow and separation of concerns via Caddy.
The python code had additional noise related to setting up a basic API which is abstracted away via `Caddyfile` config that's dedicated to this task.
* tests: OAuth2 - Minimize noise + Improve test assertion
Caddyfile can use an Access Token instead of a JWT. Much smaller and correct for this OAuth2 configuration. This new value has been documented inline.
Likewise the `sub` field returned is not important to this test. `email_verified` is kept as it may be helpful for further coverage testing.
The actual test-case has better assertions for success and failure by checking for Dovecot logs we expect instead of netcat response.
`oauth2` to `auth` for the Caddy container hostname is not necessary, just a more generic subdomain choice.
* tests: OAuth2 - Caddyfile `imap/xoauth2` route dynamic via query string
This way is more flexible and doesn't require modifying the `Caddyfile` directly, while still easy to use.
Additionally simplifies understanding the Caddyfile to maintainers by removing the `route` directive that was required to ensure a deterministic order of vars.
* tests: OAuth2 - `/imap/xoauth2` respond with IMAP commands for netcat
Since this is the only intended usage, might as well have it respond with the full file content.
* tests: OAuth2 - Implement coverage for `OAUTHBEARER`
Caddyfile route for `/imap/` now accepts any subpath to support handling both `xoauth2` and `oauthbearer` subpaths.
Both SASL mechanisms represent the same information, with `XOAUTH2` being a common mechanism to encounter defined by Google, whilst `OAUTHBEARER` is the newer variant standardized by RFC 7628 but not yet as widely adopted.
The request to `/userinfo` endpoint will be the same, only the `credentials` value to be encoded differs.
Instead of repeating the block for a similar route, this difference is handled via the Caddyfile `map` directive.
We match the path context (_`/xoauth2` or `/oauthbearer`, the `/imap` prefix was stripped by `handle_path` earlier_), when there is a valid match, `sasl_mechanism` and `credentials` map vars are created and assigned to be referenced by the later `respond` directive.
---
Repeat the same test-case logic, DRY with log asserts extracted to a common function call. This should be fine as the auth method will be sufficient to match against or a common failure caught.
* tests: OAuth2 - Minor revisions
Separate test cases and additional comment on creating the same base64 encoded credentials via CLI as an alternative to running Caddy.
Added a simple `compose.yaml` for troubleshooting or running the container for the `/imap/xoauth2` / `/imap/oauthbearer` endpoints.
* tests: OAuth2 - Route endpoints in Caddyfile with snippets instead
`reverse_proxy` was a bit more convenient, but the additional internal ports weren't really relevant. It also added noise to logging when troubleshooting.
The `import` directive with Snippet blocks instead is a bit cleaner, but when used in a single file snippets must be defined prior to referencing them with the `import` directive.
---
`compose.yaml` inlines the examples, with slight modification to `localhost:80`, since the Caddyfile examples `auth.example.test` is more relevant to the tests which can use it, and not applicable to troubleshooting locally outside of tests.
* chore: Add entry to `CHANGELOG.md`
* chore: Additional context on access token
* feat: add support for MTA-STS for outgoing mails
* Hook-up mta-sts-daemon into basic process handling test
* fix: Call python script directly
The python3 shebang will run it, which will now meet the expectations of the process testing via pgrep. fail2ban has the same approach.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Disable `smtputf8` support in config
This was always configured disabled at runtime, better to just set explicitly in `main.cf` unless config diverges when Dovecot is enabled to opt-out of this feature.
* fix: `supervisor-app.conf` - Correct `postgrey` log location
Looks like this should have been like every other service and reference a log file(s) based on program name in the supervisor log directory.
* tests: Adjust log location for `postgrey_enabled.bats`
* chore: Extract out Dovecot Quota test cases into new test file
Test cases are just cut + paste, no logic changed there yet.
* chore: Rename test case descriptions
* chore: Use `setup ...` methods instead of direct calls
* chore: Adjust `_run_in_container_bash` to `_run_in_container`
Plus some additional bug fixes in the disabled test case
* tests(refactor): Revise ENV test cases for max mailbox and message sizes
* tests(refactor): Revise ENV test cases for mailbox and message limits v2
Removes the extra variables and filtering in favour of explicit values instead of matching for comparison.
- Easier at a glance to know what is actually expected.
- Additionally reworks the quota limit checks in other test cases. Using a different formatter for `doveadm` is easier to match the desired value (`Limit`).
* chore: Sync improvement from `tests.bats` master
---
NOTE: This PR has been merged to avoid additional maintenance burden without losing the improvements. It was not considered complete, but remaining tasks were not documented in the PR.
* outsource Rspamd ENVs into explicit helper
This will allow us to uniformly source the helper and get the values
from everywhere consistently. This is more than desirable since we will
be using these values not only for the Rspamd setup, but also for DKIM
management and during change-detection.
* integrate Rspamd into changedetection
We outsource one more function to reside in the helper script for Rspamd
so that we can call this function from the Rspamd setup and from the
changedetection functionality too.
* realize deprecation of old commands file for Rspamd
THIS IS A BREAKING CHANGE!
This change realizes the log message: "Using old file location now
(deprecated) - this will prevent startup in v13.0.0" Startup will now
fail.
* added '--force' option to Rspamd DKIM script
* use new helper to get ENVs for Rspamd in DKIM script
* remove the need for linking directories
This was unnecessary, as explained in
https://github.com/docker-mailserver/docker-mailserver/pull/3597#discussion_r1369413599
* Apply suggestions from code review
review by @polarathene
* apply more review feedback from @polarathene
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370885519>
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370904201>
* update documentation
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of using `etc/rspamd/override.d/dkim_signing.conf`, we will now
be using `/tmp/docker-mailserver/rspamd/override.d/dkim_signing.conf`.
The new location is persisted (and linked again during startup) and
hence better suited.
* simplify `_setup_logrotate`
* adjust Rspamd's log file and improve it's management
* add information to docs about Rspamd log
* update log query helper to allow another file location
* bail in case `LOGROTATE_INTERVAL` is invalid
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Some deployment scenarios are not compatible with `5000:5000` static vmail user with `/var/mail`. This feature allows adjusting the defaults to a UID / GID that is compatible.
Signed-off-by: vincent <vincent@ducamps.win>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of exit status of `124` (_signifies timeout_), it should fail with `1` (failure) like the others. Handled via using `_run_in_container_bash()` (_`timeout` failure `124` does not propagate and is treated as `1` instead_).
In this case we are waiting on the status of the mail being sent, the pattern provided to `grep` is too specific and results in a timeout. Instead since we only expect the one log entry, match any status and assert the expected pattern afterwards.
This provides a more helpful failure output that informs us that mail was at least processed by Postfix, but the sent status is not what we expected.
### Before
```
✗ [ENV] (POSTFIX_DAGENT) delivers mail to existing account [60327]
(from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
in test file test/tests/parallel/set3/mta/lmtp_ip.bats, line 47)
`assert_success' failed
-- command failed --
status : 124
output :
--
```
### After
```
✗ [ENV] (POSTFIX_DAGENT) delivers mail to existing account [1425]
(from function `assert_output' in file test/test_helper/bats-assert/src/assert_output.bash, line 178,
in test file test/tests/parallel/set3/mta/lmtp_ip.bats, line 48)
`assert_output --regexp "${MATCH_LOG_LINE}=sent .* Saved)"' failed
-- regular expression does not match output --
regexp : postfix/lmtp.* status=sent .* Saved)
output : Sep 28 04:12:52 mail postfix/lmtp[721]: 23701B575: to=<user1@localhost.localdomain>, relay=127.0.0.1[127.0.0.1]:24, delay=0.08, delays=0.07/0/0.01/0, dsn=4.2.0, status=deferred (host 127.0.0.1[127.0.0.1] said: 451 4.2.0 <user1@localhost.localdomain> Internal error occurred. Refer to server log for more information. [2023-09-28 04:12:52] (in reply to end of DATA command))
--
```
The expected pattern is logged as `assert_success` confirms a valid match for the log line of interest was found, and we have the mismatched value to debug the failure against.
This was missed during original review.
On a linux host, processes running within a container have been visible via commands like `pgrep`. This is does not appear to be the case with WSL2 + Docker Desktop (Windows), resulting in test failure.
The command should have been run from within the container regardless.
* chore: Use `.yml` extension
Both of these files support the `.yml` extension. Normalize on that.
* fix: Add `.gitattributes` to ensure `LF` line-endings are committed
Avoids accidentally committing files with `CRLF` when they're created on Windows. Or worse, if some editors don't detect `LF` and would introduce mixed line-endings with `CRLF`.
Shouldn't be a problem in practice as we already have a linting check to catch this via CI during PRs. This file is complimentary, in that it should automate that concern away.
* chore: Drop management of `SASLAUTHD_*` ENV
- `variables-stack.sh` does not need to manage all these extra ENV or store them. They're not used anywhere else.
- `saslauthd.sh` is the only consumer of these ENV which are effectively direct key/value mappings, with some defaults provided / inherited.
Instead of trying to conditionally support key/value pairs when ENV is set, we could instead use `sed` to delete lines with empty values.
* chore: Drop fallbacks + update configs to match docs
- Drop deprecated support:
- `DOVECOT_HOSTS` is an ENV deprecated since v10.
- Fallback for missing URI scheme introduced for Dovecot and SASLAuthd in v10.
- Adding error log message when no LDAP URI scheme is detected for the supported ENV (when set).
- Docs updated for ENV to reflect the mandatory requirement. `mailserver.env` partially synced equivalent sections.
- Provided base LDAP configs (for overriding) likewise updated from `domain.com` to `example.com`.
- LDAP test updated for required `ldap://` URI scheme. Common ENV shared across LDAP configs hoisted out of the Postfix group.
* chore: Remove unset lines in generated `saslauthd.conf`
* chore: Use white-space in query filters to improve readability
* tests: LDAP ENV query filters documented
- These filters remain roughly the same as they were before. The conditions are the same, but restructured to make the complimentary constraints more separated from the actual target attribtues.
- The DOMAIN query additionally includes the `mailAlias` from `PostfixBookMailAccount` class as well.
- Heavy inline documentation breaking down how the filters work for any maintainer to reference. This will likely be migrated after revision into our user docs for LDAP. Some quirks have also been noted with advice for future changes.
* tests: LDAP - Support test-case specific containers
A bit more complicated than other test files due to the larger ENV config array that most containers may need to share.
Example introduced with the test-case checking custom config file support.
* tests: Adjust LDAP test configs
- Paths for `.ldif` files used with volumes shortened
- Postfix LDAP `.cf` files adjusted to conventions used in LDAP tests.
Noah Overcash
Brennan Kinney
Moritz Poldrack
pitilux
Georg Lauterbach
Casper
Ikko Eltociear Ashimine
Kirill Kirilenko
Joerg Sonnenberger
Keval Kapdee
allddd
Vincent Ducamps