Refactoring this `setup` CLI command as part of the effort to unify our DKIM feature support between OpenDKIM + Rspamd:
- Adds a `main()` method similar to other setup CLI commands.
- Help text more aligned with equivalent rspamd DKIM setup CLI command.
- DRY some repetition such as hard-coded paths to use variables.
- OpenDKIM config files are created / initialized early on now with `_create_opendkim_configs()`. `while` loop only needs to append entries, so is easier to grok.
- `_create_dkim_key()` to scope just the logic (_and additional notes_) to key generation via `opendkim-genkey`
- Now overall logic with the `while` loop of the script occurs in `_generate_dkim_keys()`:
- Ownership fixes are now applied after the `while` loop as that seems more appropriate than per iteration.
- Temporary VHOST config is now removed since it's no longer useful after running.
- Tests adjusted for one new log for adding of default trusted hosts content.
Overall this should be nicer to grok/maintain. Some of this logic will be reused for the unified DKIM generation command in future, which is more likely to shift towards all domains using the same keypair by default with rspamd/opendkim config generated at runtime rather than reliant upon DMS config volume to provide that (_still expected for private key_).
---------
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Changes:
- `jaq` should probably live in `/usr/local/bin` with other third-party sourced binaries.
- `swaks` install properly with just `tar`, no `mv` + `rm` needed.
- Added Smallstep `step` CLI. This serves similar purpose to `openssl` commands, but is generally nicer for usage with generation and inspection of certs/keys. I've talked up using in DMS a few times in the past for our TLS helper and unifying DKIM support (_instead of separate OpenDKIM/Rspamd generators_).
- Including `step` for both AMD64 / ARM64 archs needs the alternate naming convention that it's published to GH releases with.
- Added commentary about the `tar` usage. The ownership is a common concern with GH release sources, technically a non-issue when running as `root`
* fix: `setup email restrict` configs should only prepend once
* chore: Prepend to our custom parameter variant to retain applying to all `smtpd` ports
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
- Bump to [`jaq` v2 release](https://github.com/01mf02/jaq/releases/tag/v2.0.0), artifact naming convention changed.
- Tidied up the changelog a little bit unrelated to this `jaq` update.
- Fixed a typo with an `rspamd.sh` comment + minor revision to the comment.
**Overview of changes:**
- Runner bumped from Ubuntu 22.04 => 24.04
- Revised inline documentation for maintainers.
- The output of `build-docs.sh` is now grouped in the steps action log, and now hides the noise from pulling the image via `docker run`.
- Removed the separate `tar` steps with ZSTD as there is only a directory to archive with recent changes to this workflow. The `upload` + `download` actions are sufficient.
- The `workflow_run` job has had the PR context restore step extracted to a separate job to minimize noise.
- `actions-netlify` is still effectively the same functionality.
- `github-token` is no longer configured as it doesn't appear needed with the functions disabled.
- Opt-out of the GH deployments feature which is not needed.
- Fixes the `if` condition that was recently adjusted.
- Better documents concerns for maintainers to be aware of.
- Reference the `pull_requests` ENV at runtime instead of embedding content into the script via GHA context expression. This is a better practice which prevent exploits from untrusted inputs (_notably for context objects which might introduce new fields in future_).
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Changes ClamAV image source from DockerHub clamav/clamav (Alpine) to clamav/clamav-debian. Only the Debian variant offers multi-platform images.
This isn't too important since we are only interested in taking a copy of the database from the image. It should however resolve a CI warning.
`stderr` is filtered by `grep` to discard unwanted (expected) log noise when appending the override `postfix-main.cf` content (_updated settings did not replace earlier defined instances_).
That `grep` filter introduced a regression into DMS v14 release, since any other `stderr` content not being excluded was now blended into `stdout` and redirected with the original `stdout` output for the `postconf -n` command.
The fix is to ensure the `grep` output is redirect to `stderr` to avoid that mishap.
* docs: Relocate account / auth pages into a common section
* docs: Update references to relocated pages
* docs: Add account management overview page
Updates remaining links to account sections on this page instead (_for `accounts`, `aliases`, `quotas`_).
This page will cover the features and defer to separate pages for more specific content where relevant.
* docs: Correct relocated pages titles and links
* docs: Accounts (Dovecot Master) - Minor revisions
* docs: Fix highlighting roundcube PHP snippet in OAuth2 page
* docs: Accounts (File) - Refactor
- Manual method not necessary to document.
- Condense `setup` example guidance.
- Quotas / Aliases content migrated to Overview when not specific about file provisioner.
Some of the content is this commit is not a complete revision.
* chore: Temporary commit
* docs(refactor): Sub-addressing section
Much better docs on the sub-addressing feature supported by Postfix and Dovecot, along with the guidance with usage in Sieve.
* docs: Revise accounts section
Add some context regarding DMS accounts and their distinction/overlap from the email address functionality, and it's relevant context for receiving/sending.
File provisioner, minor revisions to referencing associated config files and account management.
* docs: Minor adjustments
* docs: Refactor the quota section
Better documented with links and coverage over the workaround details we've implemented.
* docs: Revise the quota section
Minor revisions with phrasing, admonitions for structure and better explanation of the feature functionality/purpose.
* docs: Alias section refactor
Extensively covers known issues and technical details that have been discussed often enough.
The improvements should benefit both users and maintainers.
* docs: Refactor master accounts page
This rewrite should more clearly document the feature, along with a better example and additional links for reference.
* docs: OAuth2 revision
Minor update to this page:
- Links extracted to bottom of page as per convention.
- ENV file example converted to preferred `compose.yaml` ENV settings.
* docs: Sieve minor revisions
- Correct link to subaddressing section
- Make the config file example snippets intended filename less ambiguous.
- Minor rephrasng.
* docs: Revise accounts overview section
Revised the account section and added additional clarity for common confusion with relation to sender address and multi-domain support.
Top of the page now clarifies it's a technical reference and directs users to the related pages for configuration / caveats.
Technical Overview links to Dovecot docs were missing.
* docs: Another revision pass
File based provisioner docs:
- Sections indent with info admonitions.
- Accounts section expanded with config format and example.
- Quotas section expanded and shifted to bottom (alphabetical sort).
- Split into `setup` CLI and config reference groups.
Overview page:
- Sections indent with info admonitions.
- Revised content.
* docs(chore): Shift sub-addressing section
This is related to accounts and aliases, but not provisioners, thus extract out of the accounts parent section.
* docs: Document `postfix-accounts.cf` third column
This lacked documentation but was community contributed feature to allow further customization of a Dovecot Account.
It has caveats as DMS does not take these into consideration anywhere in scripts. Documenting officially for better awareness.
* docs: Revise and expand supplementary pages
Better outline the OAuth2 login process, the two supported login mechanisms and their docs/rfcs, along with documenting caveat with mail client compatibility.
Add a verification tip for the OAuth2 support, showing how `curl` can be used, along with caveat presently affecting the `curl` in DMS v14.
Additionally note the feature still isn't documented fully, providing the user with additional references for more information.
`ACCOUNT_PROVISIONER` ENV docs minimized. No `OIDC` provisioner plans, the OAuth2 docs page now mentions SCIM 2.0 API as the next step towards resolving that concern. The tip admonition was removed as it no longer provides value, instead we link to the Account Management overview page.
Dovecot Master Accounts docs page now lightly document the `setup` CLI and config format for the feature.
* docs: Fix broken anchor links
Some anchor links to different parts of our docs have gone stale. This branch also broke a few itself that I missed.
The build now only reports issues with anchor links to Content Tabs, which it must not be aware of during the build (_MKDocs Material specific feature?_)
* docs(lint): Fix indentation level
* chore: Add entry to `CHANGELOG.md` + corrections
The `image` field is used for the default tag, if it's not specified Compose will infer one in addition to any extra `tags` provided.
Better to use `image` for the tag assignment, and a clear `pull_policy` to prevent trying to pull a remote image of the same name.
* fix: Update `dovecot-fts-xapian` to `1.7.13`
Contains a fix to a regression introduced that broke indexing
---------
Co-authored-by: casperklein <casperklein@users.noreply.github.com>
Describe how to use Apache Solr as a Dovecot FTS backend.
---------
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* update `compile.sh` and Dovecot FTS Xapian to 1.7.12
- I updated from 1.5.5. Moreover, I adjusted the script to have what I
consider better style.
* update Dockerfile to use recent updates
* update CHANGELOG
Fixes an issue with the Getmail service, view PR thread for additional details.
- Log an error when the expected service state directory doesn't exist.
- The location `/var/lib/getmail/` doesn't seem like it should have been introduced. Drop it in favor of `/tmp/docker-mailserver/getmail`. It appears to be for storing remote mail that was retrieved if not configured to send to Dovecot like our docs advise. This location was never valid anyway (_as referenced issue covers_).
Added the missing "s" on "submissions", otherwise this error comes up:
The Service "mailserver" is invalid: spec.ports[2].name: Duplicate value: "submission"
Custom parameters must be referenced to be retained when `postconf -n` is run. If those parameters are referenced by `postfix-master.cf` this needs to update `master.cf` before updating `main.cf`.
* add FAQ entry about DNS servers
I also opted for including a quote from @polarthene which illustrates
how DNS servers are a difficult topic and should not be DMS'
responsibility.
* link to DNS FAQ from Rspamd page & drop feature request
The feature request annotation has been removed because we decided it's
not DMS responsibility to ensure correctly working DNS servers.
* move `policies_group.conf` to correct location
I originally assumed the file had to be placed into `scores.d`, but I
now know that `local.d` is actually correct.
* add configuration for composite symbols
See updates to #3690:
Additional Rspamd Symbols
Rspamd has so-called composite symbols that trigger when a condition
is met. Especially AUTH_NA and AUTH_NA_OR_FAIL will adjust the scores
of various lines in the table above. This needs to be taken into account.
* update CHANGELOG
* remove leftover statement on `/etc/os-release`
* update wording on the PR template
* add section about other services to Rspamd docs
* remove more outdated information from Rspamd docs
* moved links and minor rewording in Rspamd docs
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: `relay.sh` helper - Reference user config paths via variables
* chore: Better document postfix helper `_vhost_collect_postfix_domains()`
The functionality is effectively the same for the two configs for the most part when it comes to parsing out a domain from the target value.
Virtual aliases is more flexible in value, which may not have a domain-part present (manual user edit).
* chore: `check-for-change.sh` - Support VHOST change visibility
- Moves the "handle changes" logic into it's own scoped function, out of the main change detection loop logic.
- This will be benefit a future commit change that will rely on `VHOST_UPDATED=1`.
* chore: `relay.sh` - Minor revisions to minimize diff noise
- Better phrasing of the current logic comments.
- Regex patterns assigned to variables (easier to grok intention)
- Bulk of the logic for generating `/etc/postfix/relayhost_map` wrapped into a separate function with Postfix config setting handled separately.
* refactor: `relay.sh` opt-out logic
- Split the two distinct features that configure `/etc/postfix/relayhost_map` into separate functions (_`MATCH_VALID` var no longer needed for legacy support_).
- Instead of extracting domains from `postfix-accounts.cf` + `postfix-virtual.cf`, this has already been handled at `/etc/postfix/vhost`, sourcing from there is far less complicated.
- Rename loop var `DOMAIN_PART`to `SENDER_DOMAIN` for better context of what it represents when appended to the config file.
- Revised maintenance notes + guidance towards a future refactor of this relayhost feature support.
* docs: `relay.sh` - Additional comment revisions
* feat: `DEFAULT_RELAY_HOST` can now also use relay credentials ENV
- Remove comment regarding `smtp_sasl_password_maps = static:${RELAY_USER}:${RELAY_PASSWORD}`, it could be used but `main.cf` presently has `644` permissions vs the `sasl_passwd` file permissions of `600`, less secure at preventing leaking of secrets (ignoring the ENV exposure itself).
- Move the `main.cf` settings specific to relayhost credentials support / security into to the relevant function scope instead. This also allows for the configuration to be applied by a change detection event without container restart requirement.
- Outer functions for setup and change detection to call have a clearer config dependency guard, as does the `_legacy_support()`.
- These changes now support `DEFAULT_RELAY_HOST` to leverage the relay credentials ENV as well.
- `DATABASE_RELAYHOSTS` is available in scope to the functions called here that reference it.
* docs: Revised ENV docs on relay host config
Better quality guidance on configuring relay hosts.
* chore: Add entry to `CHANGELOG.md`
* fix: `relay.sh` - `grep` regex compatibility with `+` requires `-E`
* chore: `postfix.sh` - `FIRST_FIELD` => More descriptive field name
* docs: Better document DMS volumes
* docs: Remove any mention of `ONE_DIR` ENV
* chore: Remove `ONE_DIR` ENV from scripts
Only `ONE_DIR=0` has any effect. As the actual feature is now dependent upon the `/var/mail-state` location existing.
It is advised not mounting anything there instead if wanting to avoid runtime state consolidation.
* docs: Adjust link ref convention
This is more search friendly / organized to find references to all DMS volumes.
* lint: Ensure final newline is present
VSCode by default excludes this if the last line rendered is removed (rendered as a separate blank line).
A separate setting can enforce adding the final newline upon save regardless.
* setup-stack: fix error when RSPAMD_DMS_DKIM_D is not set
prevent messages like this
chown: cannot access '': No such file or directory
when RSPAMD_DMS_DKIM_D has no value
* Update target/scripts/startup/setup-stack.sh
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* only declare Rspamd vars when not already declared
* update CHANGELOG
* Update CHANGELOG.md
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Detect missing final newline in configs read
These lines will be not be processed by `read`, emit a warning to raise awareness.
* fix: Ensure parsed config has final newline appended (when possible)
This functionality was handled in `accounts.sh` via a similar sed command (that the linked references also offer).
`printf` is better for this, no shellcheck comment required either.
We additionally don't attempt to modify files that are read-only.
* fix: Ensure parsed configs have CRLF to LF corrected (where possible)
Likewise, this runtime fix was only covering two config files. It now applies to all callers of this method.
* fix: Sanitize `postfix-master.cf` via helper
This feature should have been using the helper to avoid user error from their config updates accidentally introducing subtle breakage implicitly (due to CRLF or missing final newline).
* tests: Add test cases for new helpers
* tests: `rm` is redundant when using `BATS_TEST_TMPDIR`
This temporary directory is created and removed implicitly. Even after a test failure.
* chore: Remove old `postfix-virtual.cf` migration logic
This was introduced in 2018, there should be no one needing to rely on this anymore?
* tests: Remove comment on sed failure concern
* chore: Add entry to `CHANGELOG.md`
* Apply suggestions from code review
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* move log/filter functions into own file
* add ShellCheck global directives
* use new function for tracking logs
The new function, called `_send_email_with_mid`, aligns with suggestions
from @polarethene and is heavily simplified compared to its predecessor
`_send_email_and_get_id`. New helpers will be introduced to filter logs
according to the MID constructed in this function.
* new filters for searching logs with MID
* use new filters (and sending) functions
* add new helper for asserting non-existence of log message
* use new filters in tests
* Apply suggestions from code review
- `_mid` / `MID` => `_msgid` / `MSG_ID`
- Revised documentation / tooltip comments
* Apply suggestions from code review
* fix tests
* use more distinct names for MSG_ID headers
* update `_filter_service_log` to not use `-i -E`
Moreover, I added a function to print the whole mail log. Appropriate
comments were added to this function to indicate that one should only
use this function when necessary.
* adjust helpers to new helper filter
* follow-up of previous commit
* add CHANGELOG entry
* Apply suggestions from code review
* chore: Update OAuth2 to use new log helper
* Apply suggestions from code review
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* added explicit `_regexp` filters for logs
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
- The old Dovecot wiki link needed to be updated to the new location.
- The new docs are not entirely compatible AFAIK, thus making the existing examples/docs a bit outdated / incompatible. A warning admonition has been added early on to raise awareness to the reader.
- Minor formatting revisions to the content.
* fix: Dovecot PassDB should restrict allowed auth mechanisms
This prevents PassDBs incompatible with certain auth mechanisms from logging failures which accidentally triggers Fail2Ban.
Instead only allow the PassDB to be authenticated against when it's compatible with the auth mechanism used.
* tests: Use `curl` for OAuth2 login test-cases instead of netcat
`curl` provides this capability for both IMAP and SMTP authentication with a bearer token. It supports both `XOAUTH2` and `OAUTHBEARER` mechanisms, as these updated test-cases demonstrate.
* chore: Add entry to `CHANGELOG.md`
The UID / GID shifted during a new release. Until DKIM handling is refactored in a new major release, this fix ensures the content maintains the expected `_rspamd` ownership.
* tests: OAuth2 - Replace Python `/userinfo` endpoint with Caddy
Better documented, easier flow and separation of concerns via Caddy.
The python code had additional noise related to setting up a basic API which is abstracted away via `Caddyfile` config that's dedicated to this task.
* tests: OAuth2 - Minimize noise + Improve test assertion
Caddyfile can use an Access Token instead of a JWT. Much smaller and correct for this OAuth2 configuration. This new value has been documented inline.
Likewise the `sub` field returned is not important to this test. `email_verified` is kept as it may be helpful for further coverage testing.
The actual test-case has better assertions for success and failure by checking for Dovecot logs we expect instead of netcat response.
`oauth2` to `auth` for the Caddy container hostname is not necessary, just a more generic subdomain choice.
* tests: OAuth2 - Caddyfile `imap/xoauth2` route dynamic via query string
This way is more flexible and doesn't require modifying the `Caddyfile` directly, while still easy to use.
Additionally simplifies understanding the Caddyfile to maintainers by removing the `route` directive that was required to ensure a deterministic order of vars.
* tests: OAuth2 - `/imap/xoauth2` respond with IMAP commands for netcat
Since this is the only intended usage, might as well have it respond with the full file content.
* tests: OAuth2 - Implement coverage for `OAUTHBEARER`
Caddyfile route for `/imap/` now accepts any subpath to support handling both `xoauth2` and `oauthbearer` subpaths.
Both SASL mechanisms represent the same information, with `XOAUTH2` being a common mechanism to encounter defined by Google, whilst `OAUTHBEARER` is the newer variant standardized by RFC 7628 but not yet as widely adopted.
The request to `/userinfo` endpoint will be the same, only the `credentials` value to be encoded differs.
Instead of repeating the block for a similar route, this difference is handled via the Caddyfile `map` directive.
We match the path context (_`/xoauth2` or `/oauthbearer`, the `/imap` prefix was stripped by `handle_path` earlier_), when there is a valid match, `sasl_mechanism` and `credentials` map vars are created and assigned to be referenced by the later `respond` directive.
---
Repeat the same test-case logic, DRY with log asserts extracted to a common function call. This should be fine as the auth method will be sufficient to match against or a common failure caught.
* tests: OAuth2 - Minor revisions
Separate test cases and additional comment on creating the same base64 encoded credentials via CLI as an alternative to running Caddy.
Added a simple `compose.yaml` for troubleshooting or running the container for the `/imap/xoauth2` / `/imap/oauthbearer` endpoints.
* tests: OAuth2 - Route endpoints in Caddyfile with snippets instead
`reverse_proxy` was a bit more convenient, but the additional internal ports weren't really relevant. It also added noise to logging when troubleshooting.
The `import` directive with Snippet blocks instead is a bit cleaner, but when used in a single file snippets must be defined prior to referencing them with the `import` directive.
---
`compose.yaml` inlines the examples, with slight modification to `localhost:80`, since the Caddyfile examples `auth.example.test` is more relevant to the tests which can use it, and not applicable to troubleshooting locally outside of tests.
* chore: Add entry to `CHANGELOG.md`
* chore: Additional context on access token
I figured this was a useful comment to reference related to the setting if it's ever being changed or needs to be better understood (linked issue is a common failure that can be encountered related to this restriction).
This is a more explicit reminder for any future contributors that get thrown off by the usage of `sed` here and may be inclined to change it.
Add a link to reference a comment where it's already been explored what the alternative `sed` invocations available are.
* correct misc typos
We also seem to be favoring `behavior` over `behaviour`.
* bump MkDocs version
* resolve errors shown when buildg docs
* improve the Rspamd page
* behaviour -> behavior
Streamline the usage of this word. The majority used behavior, so I
opted to go with this way of spelling it.
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
docs: Rspamd DKIM config (`dkim_signing.conf`) example has been simplified via `path` + `selector` settings.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
In Docker Compose `.env` files are parsed properly when values are wrapped with quotes. Trailing white-space is also discarded, like it would be with shell variables.
This is not the case with `docker run` or other CRI like `podman` (_including it's compose equivalent support_). Those will parse the quotes to be included in a literal string value. Trailing white-space is also retained.
Hence a default with a trailing space is not compatible across CRI. This change documents the default with additional context on how to include a trailing white-space with a custom value for the users CRI choice. It additionally clearly communicates the opt-out value for this feature.
* feat: add support for MTA-STS for outgoing mails
* Hook-up mta-sts-daemon into basic process handling test
* fix: Call python script directly
The python3 shebang will run it, which will now meet the expectations of the process testing via pgrep. fail2ban has the same approach.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* ci: `.gitattributes` - Ensure `eol=lf` for shell scripts
- These files should always use LF for line endings during a checkout.
- `Dockerfile` does not like building with HereDoc `RUN` scripts that expect LF.
* chore: Disable `smtputf8` support in config
This was always configured disabled at runtime, better to just set explicitly in `main.cf` unless config diverges when Dovecot is enabled to opt-out of this feature.
* fix: `supervisor-app.conf` - Correct `postgrey` log location
Looks like this should have been like every other service and reference a log file(s) based on program name in the supervisor log directory.
* tests: Adjust log location for `postgrey_enabled.bats`
* docs: Debugging - Delivery failure from service downtime
Services may be temporarily down, such as when restarted when certificates are updated due to the `check-for-changes.sh` service. This is another known source of intermittent delivery failures.
* ci: Allow lint workflow to be manually triggered
Without this a different event must occur to trigger the workflow, which is inconvenient for automated PRs.
The file is managed by the `contributors.yml` workflow, no need for linting to be triggered on PRs for that change.
This should ideally skip the required check status for the lint workflow which cannot trigger implicitly for automated PRs. If this doesn't work the change should be reverted.
* chore: Extract out Dovecot Quota test cases into new test file
Test cases are just cut + paste, no logic changed there yet.
* chore: Rename test case descriptions
* chore: Use `setup ...` methods instead of direct calls
* chore: Adjust `_run_in_container_bash` to `_run_in_container`
Plus some additional bug fixes in the disabled test case
* tests(refactor): Revise ENV test cases for max mailbox and message sizes
* tests(refactor): Revise ENV test cases for mailbox and message limits v2
Removes the extra variables and filtering in favour of explicit values instead of matching for comparison.
- Easier at a glance to know what is actually expected.
- Additionally reworks the quota limit checks in other test cases. Using a different formatter for `doveadm` is easier to match the desired value (`Limit`).
* chore: Sync improvement from `tests.bats` master
---
NOTE: This PR has been merged to avoid additional maintenance burden without losing the improvements. It was not considered complete, but remaining tasks were not documented in the PR.
* scripts: Install rspamd from official repository instead of debian backports on arm64 architecture
* Remove unnecessary deb-src repository for rspamd
* Remove note about ARM64 rspamd version, update CHANGELOG.md
---------
Co-authored-by: Peter Adam <p.adam@cygnusnetworks.de>
* fix: Source `VERSION` from image ENV
Now CI builds triggered from tagged releases will always have the correct version. No need for manually updating a separate file.
* fix: Query latest GH release tag
Compare to the remote GH release tag published, rather than contents of a `VERSION` file.
`VERSION` file remains in source for now as prior releases still rely on it for an update notification.
* chore: Switch from `yq` to `jaq`
- Can more easily express a string subslice.
- Lighter weight: 9.3M vs 1.7M.
- Drawback, no YAML input/output support.
If `yq` is preferred, the `v` prefix could be removed via BASH easily enough.
* chore: Add entry to `CHANGELOG.md`
* ci: `VERSION` has no relevance to `:edge`
* docs: Update build guide + simplify `make build`
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Previously, we did not run the workflow on push on `master` when a
release happened because the push on master is guarded by a check on
which files were changed.
With this change, I added `VERSION` to the list of files to consider
when updating `:edge`.
* added check for Rspamd DKIM on startup
The newly added function `__rspamd__check_dkim_permissions` performs a
check on DKIM private key files. This is useful to prevent issues
like #3621 in the future. The function is deliberately kept simple and
may not catch every single misconfiguration in terms of permissions and
ownership, but it should be quite accurate.
Please note that the Rspamd setup does NOT change at all, and the checks
will not abort the setup in case they fail. A simple warning is emmited.
* add more documentation to Rspamd functions
* Apply suggestions from code review
* improve `__do_as_rspamd_user`
* rework check similar to review suggestion
see https://github.com/docker-mailserver/docker-mailserver/pull/3627#discussion_r1388697547
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* update K8s deployment
Because `allowPrivilegeEscalation` controls SUID/SGID, we require it
when postdrop is invoked.
* correct permissions for maildrop/public
The reason our permissions previously worked out as that in setups where
SUID/SGID worked, the binaries used to place files in these directories
already have SGID set; the current set of permissions makes less sense
(as explained in this comment:
https://github.com/docker-mailserver/docker-mailserver/issues/3619#issuecomment-1793816412)
Since the binaries used to place files inside these directories alredy
have SUID/SGID set, we do not require these bits (or the sticky bit) to
be set on the directories.
* Apply suggestions from code review
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
- Make this easier to find when browsing the example environment file.
- Adjust ENV documentation to properly mark the actual default value.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* Dovecot: add deb package dovecot-lua to support Lua scripting
* Adding documentation for Lua authentication
* Updated documentation and made a better distinction between Dovecot packages for officially supported features and for community supported features.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* improvide docs about how to work with logs
Most importantly,
1. I added information on the recently adopted `less` / `nano`
2. I added information about `/var/log/mail/`
* fix typos
* Apply suggestions from code review
* Update docs/content/config/debugging.md
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* outsource Rspamd ENVs into explicit helper
This will allow us to uniformly source the helper and get the values
from everywhere consistently. This is more than desirable since we will
be using these values not only for the Rspamd setup, but also for DKIM
management and during change-detection.
* integrate Rspamd into changedetection
We outsource one more function to reside in the helper script for Rspamd
so that we can call this function from the Rspamd setup and from the
changedetection functionality too.
* realize deprecation of old commands file for Rspamd
THIS IS A BREAKING CHANGE!
This change realizes the log message: "Using old file location now
(deprecated) - this will prevent startup in v13.0.0" Startup will now
fail.
* added '--force' option to Rspamd DKIM script
* use new helper to get ENVs for Rspamd in DKIM script
* remove the need for linking directories
This was unnecessary, as explained in
https://github.com/docker-mailserver/docker-mailserver/pull/3597#discussion_r1369413599
* Apply suggestions from code review
review by @polarathene
* apply more review feedback from @polarathene
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370885519>
- <https://github.com/docker-mailserver/docker-mailserver/pull/3599#discussion_r1370904201>
* update documentation
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of using `etc/rspamd/override.d/dkim_signing.conf`, we will now
be using `/tmp/docker-mailserver/rspamd/override.d/dkim_signing.conf`.
The new location is persisted (and linked again during startup) and
hence better suited.
* simplify `_setup_logrotate`
* adjust Rspamd's log file and improve it's management
* add information to docs about Rspamd log
* update log query helper to allow another file location
* bail in case `LOGROTATE_INTERVAL` is invalid
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Some deployment scenarios are not compatible with `5000:5000` static vmail user with `/var/mail`. This feature allows adjusting the defaults to a UID / GID that is compatible.
Signed-off-by: vincent <vincent@ducamps.win>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Instead of exit status of `124` (_signifies timeout_), it should fail with `1` (failure) like the others. Handled via using `_run_in_container_bash()` (_`timeout` failure `124` does not propagate and is treated as `1` instead_).
In this case we are waiting on the status of the mail being sent, the pattern provided to `grep` is too specific and results in a timeout. Instead since we only expect the one log entry, match any status and assert the expected pattern afterwards.
This provides a more helpful failure output that informs us that mail was at least processed by Postfix, but the sent status is not what we expected.
### Before
```
✗ [ENV] (POSTFIX_DAGENT) delivers mail to existing account [60327]
(from function `assert_success' in file test/test_helper/bats-assert/src/assert_success.bash, line 42,
in test file test/tests/parallel/set3/mta/lmtp_ip.bats, line 47)
`assert_success' failed
-- command failed --
status : 124
output :
--
```
### After
```
✗ [ENV] (POSTFIX_DAGENT) delivers mail to existing account [1425]
(from function `assert_output' in file test/test_helper/bats-assert/src/assert_output.bash, line 178,
in test file test/tests/parallel/set3/mta/lmtp_ip.bats, line 48)
`assert_output --regexp "${MATCH_LOG_LINE}=sent .* Saved)"' failed
-- regular expression does not match output --
regexp : postfix/lmtp.* status=sent .* Saved)
output : Sep 28 04:12:52 mail postfix/lmtp[721]: 23701B575: to=<user1@localhost.localdomain>, relay=127.0.0.1[127.0.0.1]:24, delay=0.08, delays=0.07/0/0.01/0, dsn=4.2.0, status=deferred (host 127.0.0.1[127.0.0.1] said: 451 4.2.0 <user1@localhost.localdomain> Internal error occurred. Refer to server log for more information. [2023-09-28 04:12:52] (in reply to end of DATA command))
--
```
The expected pattern is logged as `assert_success` confirms a valid match for the log line of interest was found, and we have the mismatched value to debug the failure against.
This was missed during original review.
On a linux host, processes running within a container have been visible via commands like `pgrep`. This is does not appear to be the case with WSL2 + Docker Desktop (Windows), resulting in test failure.
The command should have been run from within the container regardless.
* Update update-and-cleanup.md
spotify dockergc is UNMAINTAINED, they advice to consider using the `docker system prune` command instead.
"This repository has been archived by the owner on Feb 2, 2021. It is now read-only."
https://github.com/spotify/docker-gc
* Revise `update-and-cleanup.md`
Merges the image update + cleanup sections.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Use `.yml` extension
Both of these files support the `.yml` extension. Normalize on that.
* fix: Add `.gitattributes` to ensure `LF` line-endings are committed
Avoids accidentally committing files with `CRLF` when they're created on Windows. Or worse, if some editors don't detect `LF` and would introduce mixed line-endings with `CRLF`.
Shouldn't be a problem in practice as we already have a linting check to catch this via CI during PRs. This file is complimentary, in that it should automate that concern away.
The subnet must be specified as part of `ipam.configs`.
This was unfortunately slightly incorrect due to a mistake in the official Docker docs being propagated, which has since been fixed upstream.
Refer to the official Compose Spec for more details:
* https://docs.docker.com/compose/compose-file/06-networks/#ipam
* chore: Drop management of `SASLAUTHD_*` ENV
- `variables-stack.sh` does not need to manage all these extra ENV or store them. They're not used anywhere else.
- `saslauthd.sh` is the only consumer of these ENV which are effectively direct key/value mappings, with some defaults provided / inherited.
Instead of trying to conditionally support key/value pairs when ENV is set, we could instead use `sed` to delete lines with empty values.
* chore: Drop fallbacks + update configs to match docs
- Drop deprecated support:
- `DOVECOT_HOSTS` is an ENV deprecated since v10.
- Fallback for missing URI scheme introduced for Dovecot and SASLAuthd in v10.
- Adding error log message when no LDAP URI scheme is detected for the supported ENV (when set).
- Docs updated for ENV to reflect the mandatory requirement. `mailserver.env` partially synced equivalent sections.
- Provided base LDAP configs (for overriding) likewise updated from `domain.com` to `example.com`.
- LDAP test updated for required `ldap://` URI scheme. Common ENV shared across LDAP configs hoisted out of the Postfix group.
* chore: Remove unset lines in generated `saslauthd.conf`
* chore: Use white-space in query filters to improve readability
* tests: LDAP ENV query filters documented
- These filters remain roughly the same as they were before. The conditions are the same, but restructured to make the complimentary constraints more separated from the actual target attribtues.
- The DOMAIN query additionally includes the `mailAlias` from `PostfixBookMailAccount` class as well.
- Heavy inline documentation breaking down how the filters work for any maintainer to reference. This will likely be migrated after revision into our user docs for LDAP. Some quirks have also been noted with advice for future changes.
* tests: LDAP - Support test-case specific containers
A bit more complicated than other test files due to the larger ENV config array that most containers may need to share.
Example introduced with the test-case checking custom config file support.
* tests: Adjust LDAP test configs
- Paths for `.ldif` files used with volumes shortened
- Postfix LDAP `.cf` files adjusted to conventions used in LDAP tests.
- Deprecation startup script check is kept for `ENABLE_LDAP=1` but adjusted to emit an error instead. It can be dropped in a future release. Just a precaution for those who mistakenly update (_possibly via automation_) without checking the release notes, an error log is somewhat helpful, although it could alternatively panic?
- Docs updated to remove the `ENABLE_LDAP=1` usage
- ENV docs updated to reference a maintained LDAP image.
- Changelog includes the breaking change, and slight revision to prior release mention of deprecation.
- The `uniqueIdentifier` attribute is not appropriate and was relying on `objectClass: extensibleObject` as a workaround to allow it. A more appropriate attribute to use instead is `userID` (_short name: `uid`_).
- Removing `extensibleObject` now requires switching the user accounts to use `inetOrgPerson` class (_which inherits from `organizationalPerson`_). which allows the attributes `givenName`, `userID` and `mail` (_also provided via the `PostfixBookMailAccount` class_).
- The LDAP root object now uses `dc` attributes for `example.test` instead of `localhost.localdomain`. This has nothing to do with DMS or LDAP containers networking config, nor the users mail addresses.
- Users are now grouped under the organizational unit of `users` instead of `people`. Purely a naming change out of preference, no functional difference.
The LDAP test ENV has been updated to accommodate the above changes. An additional ENV override was required for SASLAuthd to switch an attribute set for `ldap_filter` in `/etc/saslauthd.conf` from the implicit default of `uniqueIdentifier` (_that we set during startup as an ENV default for fallback_) to the `userID` attribute.
* chore: Adjust default DKIM size (`open-dkim`) from 4096-bit to 2048-bit
4096-bit is excessive in size for DKIM key. 2048-bit is plenty.
* chore: Additional revisions to `open-dkim` command help output
- The examples use `keysize 2048`, but as that's the new default it makes sense to change that.
- Other help text was also revised.
- Last example for domains did not need to demonstrate the other options. Changed example domains to more appropriate values.
* docs: Revise DKIM docs
Primarily for the change in default key size, but does revise some text to better communicate to the user.
- While the referenced RFC advises 512-bit to 2048-bit key size, we now explicitly discourage `512-bit` as it's not secure. `1024-bit` is still likely safe for most, but `2048-bit` is a good default for those not rotating their keys.
- Adjusted the domains example to match the new `setup config dkim domain` domains example.
- Tip for changing default key size changed to "info" with added clarity of lowering security or increasing it (excessively).
- Rspamd section is minor formatting changes, with the exception of clarifying the "main domain" for the mail accounts is assumed as the DMS FQDN with any subdomain (like `mail.`) stripped away. This is not great, but a legacy issue that needs to be addressed in future.
- `docs-rspamd-override-d` ref removed, and usage replaced with equivalent ref `docs-rspamd-config-dropin`, while `docs-rspamd-config-declarative` ref was not in use and also removed.
- Revised the `<selector>.txt` DNS formatting info section to better communicate with the reader. Additionally it had mixed usage of default `mail` and custom `dkim-rsa` selectors (_file content and output_).
* docs: Sync DKIM commands help messages and update DKIM docs for LDAP
- Adopt the help options format style from the `rspamd-dkim` into `open-dkim` command. And convert `./setup.sh` to `setup`. `selector` option has been implemented. for a while now.
- Update `rspamd-dkim` examples help output to align with `open-dkim` command examples.
- Give both DKIM command tools a consistent description. The two tools differ in support for the `domain` option (_implicit domain sourcing for default account provisioner, and support for multiple domains as input_).
- DKIM docs for LDAP domain support revised to better communicate when explicit domain config is necessary.
* tests: Adjust test-cases for `setup config dkim` change
`rspamd_dkim.bats`:
- Update assert for command help output.
- Don't bother creating a DKIM key at 512-bit size.
`setup_cli.bats`:
- Update assert for command help output of the `setup config dkim` (OpenDKIM) command.
* docs: Update DKIM section for large keys to newer RFC
The linked discussion from 2021 does mention this updated RFC over the original. That removes outdated advice about `512-bit` key length support.
The discussion link is still kept to reference a comment for the reader to better understand the security strength of 2048-bit RSA keys and why larger keys are not worthwhile, especially for DKIM.
* docs: Extract out common DKIM generation command from content tabs
Should be fine to be DRY here, not specific to `open-dkim` or `rspamd` generation/support. Previously rspamd lacked support of an equivalent command in DMS.
* docs: DKIM refactoring
- Shifted out the info admonition on key size advice out of the content tabs as it's now generic information.
- Indented the 4096-bit warning into this, which is less of a concern as the default for our DKIM generation tools is consistently 2048-bit now.
- Reworked the LDAP and Rspamd multi-domain advice. To avoid causing a bad diff, these sections haven't been moved/merged yet.
* docs: Revise DKIM docs
Advice for managing domains individually with LDAP and Rspamd extracted out of the content tabs. Default domain behaviour explained with extra info about OpenDKIM + FILE provisioner sourcing extra domains implicitly.
* ci: Revise `question.yml` to better clarify the issue tracker is not for support queries
Users have been making low effort reports (_bypassing the dedicated form_) through this alternative that is intended for addressing other concerns related to the project - not troubleshooting user problems.
When a user does not want to put the effort in of a full bug report (_and following our debug docs tips that it refers them to_), they should be using the Github Discussions page which provides the same free-form input, but should not require attention of project devs (contributors / maintainers).
---
The markdown rendered field above the "Description" input field didn't seem too relevant for this template. I've opted for a markdown comment (so it won't render if kept) into the input field with hopes that'll be more visible to the readers attention.
* chore: Fix typo
**TL;DR:**
- New image is actively maintained vs existing one that is over 5 years old.
- Slight improvement to LDAP tree config via `.ldif` files.
- No more `Dockerfile` required to build, we can just rely on `docker run`.
`osixia/openldap` has not seen any activity since Feb 2021, while our `Dockerfile` was fixed to v1.1.6` (Feb 2018).
Startup time for this new image is around 5 seconds? (_The LDAP test uses a standard 20 second timeout check to wait until the server is ready before continuing with starting the DMS image_).
This commit migrates to `bitnami/openldap` which required modifying the `01_mail-tree.ldif` to also include adding the root object to start successfully. This image is actively maintained and one of the most popular OpenLDAP images on DockerHub.
The user account `.ldif` files have minimal changes:
- Lines moved around for better organization
- Additional comments for context
- Removal of inherited `objectClass` attributes (`person`, `top`) from the `orgnizationalPerson` class. Attribute `sn` changed to long form `surname` and values corrected with `givenName`. `changetype: add` was also not necessary.
Additionally the image does not support the `.schema` format, they must be converted to `.ldif` which has been done for `postfix-book.schema`.
See PR for more details.
The new function can
1. update/append
2. update/prepend
3. initialize if non-existent
options in `/etc/postfix/main.cf` in a safe and secure manner. When the
container is improperly restarted, the option is not applied twice.
---
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* tests: Switch to setup helper and conventions
* tests: Adapt run command to new conventions
- We have two helper methods with implicit `CONTAINER_NAME` reference, which is a bit more DRY and improves readability.
- `wc -l` + `assert_output 1` converted to use helper `_should_output_number_of_lines 1`
- `DOMAIN` var changed from `my-domain.com` to local testing domain `example.test`.
* tests: Refactor `setup_file()`
- Test wide ENV defined at the top
- OpenLDAP build and run logic grouped together. Added notes, network alias and tty not required.
- Extracted out special LDAP Postfix/Dovecot ENV into separate array. LDAP specific provisioning / auth ENV also included, with comments + linebreak to better group related ENV.
- Likewise additional ENV to support test cases has been extracted to a separate array with additional comments for context.
- Those two arrays are expanded back into the main `CUSTOM_SETUP_ARGUMENTS` that configure hostname and network for the DMS container.
* tests: Refactor the LDAP account table query testcase
- Covers 3 accounts to test from LDAP.
- 2 are the same query against users/aliases/groups tables in Postfix, only differing by account queried (and expected as returned result).
- 1 separate test to ensure a difference in config is supported correctly.
- Extracted repeated test logic into a helper method.
- Added additional context in comments about the creation source of these LDAP accounts and their related Postfix config / interaction. Direct reference to special case PR (since `git blame` will be less useful).
* tests: Use iteration for `grep` setting checks
More DRY approach. With a bit more helpful failure context via `assert_output` (_and only grepping the key_). Simpler to grok what's being covered.
* tests: DRY test email delivery
A bit more verbose with the new helper method. `test-email.txt` template is only used by the LDAP test, as is the `sendmail` command.
Helper will take two args to support the testcases, but at a later date should be refactored to be consistent with the `_send_email()` helper (_which presently uses `nc` that is required for plain-text mail/auth, otherwise we'd have used `openssl`, bigger refactor required_).
* tests: Slight revisions and relocating testcases
- Dovecot quota plugin testcase revised to check files exist instead of rely on `ls` failure.
- Moved Postfix quota plugin testcase into prior dovecot testcase for quota plugin check. Better error output by only querying the `smtpd_recipient_restrictions` setting (_which should be the only one configured for the service_).
- Moved the saslauthd and pflogsumm testcases (_no changes beyond revised comments_) above the `ATTENTION` comment, and one testcase below the comment that belonged to that group.
* tests: Simplify openldap `docker build` command
- `--no-cache` was creating a new image on the Docker host each time the test is run. Shouldn't be any need to build without cache.
- No need to use `pushd` + `popd`, can just provide the path context directly, and the `./Dockerfile` is an implicit default thus `-f` not required either.
Additionally removed the old `checking` prefix from testcase names.
* tests: Move LDAP specific config into `test/config/ldap/`
- No changes to any of these config files, just better isolation as not relevant to any other tests.
- Section heading in `setup_file()` added to distinguish the remainder of the function is dedicated to the DMS container setup.
- Comment providing some context about the `mv` to maintainers, this should be done after defaults are initialized but before starting up the container.
* chore: Appease the lint gods
* Apply suggestions from code review
Adding `internet.nl` mail tester, this testing services gives users in-depth analysis of their mail server, connectivity, DKIM/SPF/DMARC records and DNS.
For added clarity, a user requested we document the example config snippets instead of only linking external references to them. Revised section and adjusted to presenting via the content tabs feature.
docs: Add compatibility section to debugging page
ci: Adjust bug report template
Reduce some text + compress the preliminary checks down to single check item.
* Fix issue with concatenating $dmarc_milter and $dkim_milter in main.cf
Upon each start the `smtpd_milters` and `non_smtpd_milters` would be extended with the following:
```
smtpd_milters = $dmarc_milter $dkim_milter
non_smtpd_milters = $dkim_milter
```
In my case they became long enough that mail delivery stopped. I think this was because of the extra headers that are added by these steps. (which in turn would cause the mail to be dropped)
* fix sed to work when the variables are there and when they are not.
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Simplify the bug report form further by dropping / merging form sections.
Change Overview:
- Minor revisions and formatting changes (_multi-line pipe operator, emoji, fix typos, etc_).
- Collapsed OS + Arch into single input field (_not much benefit from the two additional dropdown items_).
- Description/reproduction and expectation sections revised (_expectation intent is typically inferred by the issue description, while detailed reproduction steps can belong a separate optional section_).
- Removed platform dropdown (_Windows and macOS are mentioned in description as unsupported_).
- Removed experience checkboxes (_context doesn't really change responses_).
- Removed the orchestrator dropdown (_we don't seem to use this information, it's just noise_)
- Relocate the DMS version + OS/Arch sections to come after the Reproduction steps.
Previously it was assumed the sed operation was applying the sed expressions as a sequence, but it did not seem to filter entries being looked up correctly.
Instead any line that matched either sed expression pattern was output (_value without matching key, values split by the delimiter_), then grep would match any of that causing false-positives.
Resolved by piping the first sed expression into the next.
* add new home dir for Dovecot
I tried changing the mail dir, but this is a _very_ disruptive change,
so I took approach 3 on
<https://doc.dovecot.org/configuration_manual/home_directories_for_virtual_users/>,
whereby the home directory is now inside the mail directory.
The MDBOX/SDBOX formats are not touched by this change. The change
itself could be considered breaking though.
* adjust Sieve tests accordingly
* Update target/dovecot/10-mail.conf
* Update target/dovecot/auth-passwdfile.inc
---------
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* adjust learning of ham
See #3333
When moving a mail from the Junk folder to the Trash folder, the mail
previously classified as ham due to the wildcard match of `*`. Because
the syntax does not allow for negation, we can only change the behavior
in a way that mails are learned as ham when they are moved into `INBOX`
from `Junk`. This is reasonable though.
* adjust tests accordingly
* adjust docs accordingly
* add sanity check for Clam size & adjusted MaxScanSize
The second part is of special importance! See
<https://askubuntu.com/a/1448525>, which explains that the maximum scan
size is important as well. We previously just set the maximum file size,
which actually is pretty insecure as we silently not scan mile bigger
than `MaxScanSize`. This is corrected now.
* add SlamAV size configuration to Rspamd
For an upcoming PR, these changes are required, because the script that
is using the helpers uses `set -eE`. This leads to situations where
errors are not properly handled in our helpers (yet; I plan on changing
that in the future).
* move modules adjustment file to new location
Because we link `/tmp/docker-mailserver/rspamd/override.d` to
`/etc/rspamd/override.d`, I think it makes sense to move the modules
adjustment file into `/tmp/docker-mailserver/rspamd/` as well.
I write the code in a way that it is backwards compatible for now, so
this is NOT a breaking change.
* minor improvement to `__rspamd__handle_user_modules_adjustments`
The expansion of `ARGUMENT3` is now done in a way that only adds the
whitespace in case the variable is set and not null.
* move test file structure to respect latest changes
Because we're now linking `rspamd/override.d/`, we can simplify the
setup a bit. But this requires a change in directory structure.
The current Rspamd test will be renamed to `rspamd_full.bats`, because I
plan on adding more tests in different files for different feature sets.
This is done to make this feature well-tested!
* improved and added tests to Rspamd-full
FYI: The line
```bats
_run_in_container grep 'sieve_global_extensions.*\+vnd\.dovecot\.pipe'
"${SIEVE_CONFIG_FILE}"
```
was testing a condition that should actually not be met, but when I
started working on this feature, I thought this was the correct
configuration. Adding the `assert_success` statements revealed this
wrong line.
I also added tests to check whether `override.d` is linked correctly.
* renamed: `rspamd.bats` => `rspamd_full.bats`
* added new tests for incomplete Rspamd feature set
We now test that warnings are emitted & features are disabled correctly.
* update documentation
* added checks whether OpenDKIM/OpenDMARC/policyd-spf are enabled
* added functions to check if VAR is 0/0 or an int
and also added tests.
I also adjusted the test file to not run in a container, because there
is no need. This also decreases test time, which, in turn, increases
maintainers' happiness.
* added more checks to Rspamd setup
I added the helpers from the previous commit to the Rspamd setup to make
the whole setup more robust, and indicate to the user that an ENV
variable's value is incorrect.
While we did not issues for this in the past, I believe it to be
worthwhile for the future.
* added canonical directory for users to place files in
This dir is canonical with DMS's optional configuration dirs, as it
lives in well-known volume mounts. Hence, users will not need to adjust
`/etc/rspamd/override.d` manually anymore, or mount a volume to this
place.
The docs explain this now, but the DKIM page needs a slight update on
this too I guess. I will follow-up here.
* misc minor improvements
* use variables for common directories
When a new version of docker-mailserver is available the account used in this
tests also gets the postmaster notification for the new version. The mailbox
then may contain 2 mails but only one with 'This is a test mail.'.
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
The user management docs are now one page, because the division between
accounts and aliases is useless because there simply isn't enough
content to justify the split. I improved and updated the text a bit.
Misc spelling fixes and resolved imprecise statements. Shortened the bug
report introduction a bit further and added a statement about being
precise to all templates.
* docs: change some absolute links to relative links
* docs: change most hard-coded links to `edge` to point to `latest`
* Apply suggestions from code review
* docs: revert 404 page to edge and change canonical link to `latest
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* fixed special bits for maildrop and public dir
After changing the group, special bits are lost, but they should be set for the directories `/var/spool/postfix/{maildrop,public}`, otherwise you see the following error:
```
postfix/postdrop[17400]: warning: mail_queue_enter: create file maildrop/729504.17400: Permission denied
```
* fix: Match octal permissions originally provided
Officially Postfix source seems to imply:
- `730` for `maildrop/` (_but has mentioned a sticky bit in the past, set-gid bit only for the postdrop binary involved_)
- `710` for `public/`
Both folders are assigned the same group that `postdrop` belongs to which has the SGID permission for it's executable. SGID special bit on`public/` doesn't seem necessary, but left as-is to match the default from Debian.
---------
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
If permissions are specified at the workflow level, any that are not explicitly set became `none` and jobs cannot request that to change.
Permissions are therefore scoped to the job itself.
* remove PostSRSD wrapper
The setup is now completely done during _actual_ setup phase. The
wrapper did not even catch signals (SIGINT, etc.), which I think is
strange.
I also added all the ENVs the wrapper relied on (which previously could
have been unset) to the variables script.
* forgot adjusting the `Dockerfile`
`postfix start-fg` was not properly responding to signals received to stop. This caused `supervisorctl restart postfix` and `supervisor stop postfix` to not work as expected (_stopping the Postfix master process, before attempting to start the service again_).
Supervisor does not support custom commands for restarting or stopping a service, relying only on managing the process via a signal. In the past we used a wrapper script to TRAP the signals and trigger commands that way.
However there is a feature which allows us to proxy signals to a different process by referencing a PID file. As Postfix master process creates a pid file when started, we can avoid a wrapper script and the `supervisorctl` functionality works as intended 👍
* postfix: remove smtpd_sasl_auth_enable global setting
* tests: disable auth on 25 port
* tests: revert ldap-smtp-auth-spoofed-sender-with-filter-exception.txt
* Skip failing test
The test seems to have been broken from the beginning.
Sadly, no LDAP maintainers can verify. Added a TODO item if ever a LDAP maintainer comes around.
* Apply PR feedback
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* chore: Only replace `CHKSUM_FILE` when a change has been processed
* chore: Change Detection service should be the last daemon started
* chore: Remove 10 second startup delay for change detector
There should be no concern with conflicts as any writes should have already been done by the time this daemon service is started.
* tests(fix): `smtp_delivery.bats` must wait for Amavis
The change event for adding a user can be processed much sooner now, which means Amavis may not yet be ready.
Added extra condition to wait on at least the Amavis port being reachable, and some failure asserts with the mail queue to better catch / debug when this problem occurs.
* chore: Add some minor delay to avoid Amavis failing to connect
* tests(refactor): Make test cases for opendkim keysizes DRY
- These all do roughly the same logic that can be split into two separate methods.
- `_should_generate_dkim_key()` covers a bit more logic as it can be leveraged to handle other test cases that also perform the same logic.
- The `config/opendkim/` doesn't seem necessary for tests. Only the first few test cases here are testing against it, so we can conditionally make that available. `process_check_restart.bats` also depended on it to run OpenDKIM successfully, but this was due to the `setup-stack.sh` config defaults failing to find an "empty" file forcing `supervisord` to constantly restart the process..
- With this, there we inverse the default opendkim config, so we don't have to mount unique / empty subfolders for each test case, followed by copying over the two extra configs.
* tests(refactor): DRY up more test cases
All the remaining test cases but the last one were refactored here for a clean commit diff. The last test case will be refactored in the following commit.
Plenty of repeated logic spread across these test cases, now condensed into shared methods.
* tests(refactor): Make final test case DRY
* chore: Migrate to new testing helpers
* chore: Revise test case descriptions
* tests(refactor): Improve and simplify assertions
* tests(refactor): Use common container setup instead of `docker run`
- As the majority of test cases are only running `open-dkim` helper, we don't actually have to wait for a full container setup. So an alternative container start is called.
- Also improves assertions a bit more instead of just counting lines.
- Some test cases don't bind mount all of `/tmp/docker-mailserver` contents, thus don't raise permission errors on subsequent test runs.
- Instead of `rm -f` on some config files, have opted to mount them read-only instead, or alternatively mount an anonymous empty volume instead.
- Collapsed the first three test cases into one, thus no `setup_file()` necessary.
- Shift the `_wait_for_finished_setup_in_container()` method into `_common_container_setup()` instead since nothing else is using `_common_container_start()` yet, this allows for avoiding the wait.
* tests(refactor): Collapse dkim key size test cases into single test case
This makes these tests a bit more DRY, and enhances the raised quality issue with these tests. Now not only is the domain checked in the generated DNS dkim record, but we also verify the key size is corrected in the public and private keys via openssl.
* chore: Revise container names
* chore: Swap order of test case 1 and 2
* tests(refactor): Assert generated log output
- `__should_have_tables_trustedhosts_for_domain` shifted in each test case to just after generating the domains keys.
- Asserts `open-dkim` logs instead of just counting them.
- Added checks for domains that should not be present in a test case.
- Additional coverage and notes about the alias from vhost `@localdomain.com`
- Single assert statement with switch statement as all are using common args.
* chore: Minor changes
* tests(refactor): Share `find` logic in helpers and tests
* tests(fix): Listing file content does not need to match line order
The order printed from local system vs CI differed causing the CI to fail. The order of lines is irrelevant so `--index` is not required.
Additionally correct the prefix of the called method to be only one `_` now that it's a `common.bash` helper method.
* chore: Collapse custom DKIM selector test into custom DKIM domain test
These cover the same test logic for the most part, the first domain could also be testing the custom selector.
`special_use_folders.bats` + `mailbox_format_dbox` can assert lines instead, removing the need for `--partial`.
* Apply suggestions from code review
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* chore: Split switch statement method into wrapper methods
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* ci(fix): Temporarily avoid specifying `provenance`
As the test workflow does not use the `docker-container` buildx driver, it uses the Docker Engine bundled BuildKit version which until v23 release does not support attestations.
Likewise the current buildx version in CI is `0.10.0` which does not respect `--provenance false`, the presence of the option appears to trigger a BuildKit version compatibility check and fail early before it considers the value of the option.
* chore: Use buildx `docker-container` driver instead
An alternative solution to omitting `provenance: false` (_not supported by buildx 0.10.0 with default `docker` driver when Docker Engine bundles BuildKit less than 0.10.0, which is the case prior to the Docker Engine v23 release_).
This approach provides more consistency with the build and publish workflows by using the same buildx `docker-container` driver (_and thus newer BuildKit, enabling support for `provenance: false`_).
* chore: Revise test workflow inline docs
Buildx `docker-container` driver is not needed here, but it does seem like it improves cache-hit ratio when building from the retrieved build cache (from the earlier build workflow). Possibly due to building with the same BuildKit version.
* refactor `mail_pop3.bats`
* refactor `mail_with_imap.bats`
* refactor `mail_with_relays.bats`
* moved test that that did not belong into POP3 test
* slightly clean up `no_container.bats`
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Extract out accounts test cases from `tests.bats`
Standard test file format, the test cases have been copied over unmodified.
* chore: Revise test case descriptions
* tests(refactor): `accounts.bats`
Revised test cases:
- Some common test case logic extracted to test methods.
- Update direct user management commands to use the `setup email ...` variants.
- Improved assertions.
- Removed `sleep 2` lines as the need for that is ambiguous (may no longer be relevant?)
- Additional commentary for maintaining
- Two test cases for missing `postfix-accounts.cf` opted to just run the image without any volumes instead, as the `without-accounts/` folder was empty anyway. 2nd test case can instead use a single `docker run` to check the newly created`postfix-accounts.cf` content.
- `test/config/without-accounts/` remains as `open_dkim.bats` presently uses it.
* chore: Remove unnecessary account removal assert
Traced this back to the original PR where it appears to have been a typo and was probably intended as a cleanup on the `user4` account. Not necessary, removing.
* chore: Rename `accounts.bat` -> `account_management.bats`
---------
* feedback: Avoid `ls` for detecting directories
Replace `ls -d` approach from original test cases
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* feedback: Remove asserting empty output on failure
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* only add Amavis configuration to Postfix when enabled
Since I am running Rspamd nowadays, I noticed there still are ports open
that belong to Amavis. This is because the Amavis configuration is a
fixed part of Postfix's `master.cf`. I changed that. Now, the Amavis
section is added when Amavis really is enabled.
I took the chance and added proper indentation to `master.cf`; hence the
diff is a bit fuzzy. **But**, only the Amavis part was adjusted, the
rest is just styling.
* tests: Migrate and combine ENV tests for `*_INET_PROTOCOLS`
These two features + tests were introduced years apart but serve the same purpose for both Postfix and Dovecot.
Using `--openssl` uses the native `openssl` package within the image instead of the older `1.0.2` bundled from `testssl.sh`.
The test is only testing cipher suite compatibility is what we expect it to be, thus we do not need to run `testssl.sh` with a broader range of ciphers.
* add functionality for filtering mail log by ID
This was not planned, but as @polarthene mentioned in
https://github.com/docker-mailserver/docker-mailserver/pull/3033#issuecomment-1407169569
, filtering the mail log by email ID would be (the only) correct
approach for the Rspamd test (to eliminate race conditions).
I asserted the currect state, and came to the conclusion that this might
(or actually is) something we want in more than one place. So I went
ahead and implemented a solution.
The solution for acquiring the ID is a bit slower because it ensures the
mail queue is empty _before_ and _after_ the mail is sent. This is the
tradeoff one has to make if they want to send multiple emails in one
test file and get their IDs.
I hope you like this approach. I will provide another PR that adjusts
our current tests to use these new functions.
* added note about our helper functions in the docs
I think our work for our custom test framework should be noted in the
docs for newcomers to better understand what they should do.
* adjust Rspamd test to use new helpers for sending
* improve filter helpers further
* add sanity check when acquiring mail ID
* re-add `refute_output` to test which should now work well
This doesn't make any difference to the tests performed here (_partly due to `--preference`_).
It would make a difference if performing a test for receiving a grade, which would otherwise fail due to chain of trust not being verifiable for a self-signed certificate (_or a signed certificate without a CA public key to verify against_)
This appears to have been added to replace the `fam` package in an early version of DMS with Courier for IMAP instead of Dovecot on an Ubuntu 14.04 base image.
It does not appear to serve a purpose anymore.
* chore: Remove the wrapper script for `fail2ban`
- This does not appear necessary. The server can be run with foreground mode.
- `daemons-stack.sh` removal of the socket can be handled by the fail2ban server when using the `-x` option.
* chore: Remove `touch /var/log/auth.log`
These were both added as supposed fixes in 2016 for the then Ubuntu 2014 base image.
Removing them causes no failures in tests.
* fix: Install optional python packages for `fail2ban`
These have barely any overhead in layer weight. The DNS package may provide some QoL improvements, while the `pyinotify` is a better alternative than polling logs to check for updates.
We have `gamin` package installed but `fail2ban` would complain in the log that it was not able to initialize the module for it. There only appears to be a `python-gamin` dependent on EOL python 2, no longer available from Debian Bullseye.
* chore: Use a common method to check domain and fqdn config
* chore: Shift other test cases into shared test methods
* chore: Add another shared method for checking mail headers
* chore: Add another shared method for checking hostname
* refactor: Improve quality of shared test methods
Based on changes from an earlier closed hostname PR from Oct 2021 with additional revision to use `assert_output` and more thorough checking of values expected in output.
* chore: Move clean shutdown test to `process-check-restart.bats`
This was originally a single test case in `tests.bats` intended for `supervisord` testing.
It seems at some point it got reassigned to a hostname override test container, and then migrated to separate test file for hostname override test by accident.
It now belongs in the correct place again, as hostname config should have nothing to do with a graceful shutdown?
* chore: Prepare for migrating to use `test/helper/setup.bash`
* chore: Rename containers and configured FQDN settings
* chore: Convert to using common container setup helpers
Wait for SMTP port is left at the end to avoid additional start-up delays.
* chore: Use `_run_in_container_bash` helper
* chore: Be more specific on matching mail headers
- I could do multiple container grep calls instead, but opted to match by lines in file. This better ensures values are being matched to the correct lines.
- Renamed the test case descriptions.
- Expanded test coverage of the 4th container as it represents another DNS config, while the 3rd is just the 4th container with the `SRS_DOMAINNAME` env added, no value in more coverage there.
* chore: Remove redundant test coverage in `tests.bats`
These checks are performed in `mail_hostname.bats` with better coverage.
* chore: Move each containers setup into it's own test-case instead
* chore: Re-arrange container name IDs
The original `fqdn-with-subdomain` is now `with-nis-domain` which is more accurate. A new test case will properly cover the default `--hostname` only config that is not a bare domain.
* chore: Re-arrange test cases to align with new ID ordering
This commit just shifts the test cases, no new changes to any content beyond that.
* chore: Add new test case for default config
* chore: Review feedback `_run_in_container_bash` to `_run_in_container`
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* chore: Additional review feedback
- Fix a suggested change bug with quote wrapping an interpolated variable.
- Convert two other `_bash` methods that were missed from review.
- Apply the last two suggested changes from review.
* chore: `_exec_in_container_bash` to `_exec_in_container`
The `| head -n 1` can be dropped if we know for sure it's only one line, which is what we expect. Quotes can then be dropped too.
---------
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* tests(fix): `spam_junk_folder.bats` Wait on Amavis port to be ready
Postfix can potentially be ready before Amavis is. This caused test failures as mail was sent before Amavis was ready to process it.
Both test cases shared the same test logic, except for the expected location to deliver the spam to. Extracted into a shared test method, and moved the port conditions into there.
* tests(chore): `spam_junk_folder.bats` minor revisions
Test case descriptions, container names and test prefix are now more descriptive of what is under test here (an ENV for Amavis).
* tests(chore): Move Amavis bounce test into `spam_junk_folder.bats`
These two tests seem to be related to the same feature. Grouping them into a single test file instead.
* tests(refactor): Split shared method into smaller methods
Now it can be better shared with the bounce test case.
* tests(chore): Shift test cases to match their CONTAINER_NAME order
No changes to code, just cut + paste of the `CONTAINER3_NAME` test case to shift it to the last test case position.
* added options to toggle OpenDKIM & OpenDMARC
rspamd can provide DKIM signing and DMARC checking itself, so users
should be able to disable OpenDKIM & OpenDMARC. The default is left at
1, so users have to to opt-in when the want to disable the features.
* misc small enhancements
* adjusted start of rspamd
The order of starting redis + rspamd was reversed (now correct) and
rspamd now starts with the correct user.
* adjusted rspamd core configuration
The main configuration was revised. This includes AV configuration as
well as worker/proxy/controller configuration used to control the main
rspamd processes.
The configuration is not tested extensively, but well enough that I am
confident to go forward with it until we declare rspamd support as
stable.
* update & improve the documentation
* add tests
These are some initial tests which test the most basic functionality.
* tests(refactor): Improve consistency and documentation for test helpers (#3012)
* added `ALWAYS_RUN` target `Makefile` recipies (#3013)
This ensures the recipies are always run.
Co-authored-by: georglauterbach <44545919+georglauterbach@users.noreply.github.com>
* adjusted rspamd test to refactored test helper functions
* improve documentation
* apply suggestions from code review (no. 1 by @polarthene)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* streamline heredoc (EOM -> EOF)
* adjust rspamd test (remove unnecessary run arguments)
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* fix: RSPAM ENV should only add to array if ENV enabled
* fix: Correctly match ownership for Postfix content
- `/var/lib/postfix` dir and content is `postfix:postfix`, not `postfix:root`.
- `/var/spool/postfix` is `root:root` not `postfix:root` like it's content.
- Add additional comments, including ownership changes by Postfix to `/var/spool/postfix` when process starts / restarts.
* fix: Ensure correct `chown -R` user and groups applied
These were all fine except for clamav not using the correct clamav group. `fetchmail` group is `nogroup` as per the group set by the debian package.
Additionally formatted the `-eq 1 ]]` content to align on the same columns, and added additional comment about the purpose of this `chown -R` usage so that it's clear what bug / breakage it's attempting to prevent / fix.
* refactor: `misc-stack.sh` conditional handling
The last condition doesn't get triggered at all AFAIK. Nor does it make sense to make a folder path with `mkdir -p` to symlink to when the container does not have anything to copy over?
- If that was for files, the `mkdir -p` approach seems invalid?
- If it was for a directory that could come up later, it should instead be created in advance? None of the current values for `FILES` seem to hit this path.
Removing as it doesn't seem relevant to current support.
Symlinking was done for each case, I've opted to just perform that after the conditional instead.
Additional inline docs added for additional context.
* chore: Move amavis `chown -R` fix into `misc-stack.sh`
This was handled separately for some reason. It belongs with the other services handling this fix in `misc-stack.sh`.
The `-h` option isn't relevant, when paired with `-R` it has no effect.
* fix: Dockerfile should preserve `clamav` ownership with `COPY --link`
The UID and GID were copied over but would not match `clamav` user and group due to numeric ID mismatch between containers. `--chown=clamav` fixes that.
* chore: Workaround `buildx` bug with separate `chown -R`
Avoids increasing the image weight from this change by leveraging `COPY` in the final stage.
* chore: `COPY --link` from a separate stage instead of relying on scratch
The `scratch` approach wasn't great. A single layer invalidation in the previous stage would result in a new 600MB layer to store.
`make build` with this change seems to barely be affected by such if a change came before copying over the linked stage, although with `buildx` and the `docker-container` driver with `--load` it would take much longer to import and seemed to keep adding storage. Possibly because I was testing with a minimal `buildx` command, that wasn't leveraging proper cache options?
* lint: Appease the linting gods
* chore: Align `misc-stack.sh` paths for `chown -R` operations
Review feedback
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* fix: Reduce one extra cache layer copy
No apparent advantage of a `COPY --link` initially in separate stage.
Just `COPY --chown` in the separate stage and `COPY --link` the stage content. 230MB less in build cache used.
* fix: Remove separate ClamAV stage by adding `clamav` user explicitly
Creating the user before the package is installed allows to ensure a fixed numeric ID that we can provide to `--chown` that is compatible with `--link`.
This keeps the build cache minimal for CI, without being anymore complex as a workaround than the separate stage was for the most part.
* chore: Add reference link regarding users to `misc-stack.sh`
* Fix#3007: Changed description of explicit TLS to indicate that insecure connections are rejected
* Further clarification that description only applies to authentication
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* chore: Co-locate process checking and process restart verification
Extract the test cases for checking a process is running and properly restarts from various test files into a single one:
Core (always running):
opendkim, opendmarc, master (postfix)
ENV dependent:
amavi (amavisd-new), clamd, dovecot, fail2ban-server (fail2ban), fetchmail, postgrey, postsrsd, saslauthd
These now run off a single container with the required ENV and call a common function (the revised version in parallel test cases).
* fix(saslauthd): Quote wrap supervisor config vars
`saslauth.conf` calls `-O` option for most commands defined with an ENV that may be empty/null. This would cause the process to silently fail / die.
This doesn't happen if quote wrapping the ENV, which calls `-O` with an empty string.
Not necessary, but since one of `postgrey` ENV were quote wrapped in `supervisor-app.conf`, I've also done the same there.
* fix(postsrsd): Change supervisor `autorestart` policy to `true`
The PR that introduced the config switched from `true` to `unexpected` without any context. That prevents restart working when the process is killed. Setting to `true` instead will correctly restart the service.
* chore: Remove disabled postgrey test file
`mail_with_postgrey_disabled_by_default.bats` only checked the migrated test cases, removed as no longer serving a purpose.
* tests(refactor): Make `_should_restart_when_killed()` more reliable
The previous version did not ensure that the last checks process was actually restarted, only that it was running.
It turns out that `pkill` is only sending the signal, there can be some delay before the original process is actually killed and restarted.
This can be identified with `pgrep --older <seconds>`. First ensure the process is at a specified age, then after killing check that the process is not running that is at least that old, finally check that there is a younger process actually running.. (_could fail if a process doesn't restart, or there is a delay such as imposed by `sleep` in wrapper scripts for postfix and fail2ban_)
The helper method is not used anywhere else now, move it into this test instead. It has been refactored to accomodate the needs for `--older`, and `--list-full` provides some output that can be matched (similar for `pkill --echo`).
* test(docs): Add inline notes about processes
* chore: Compress test cases into single case with loop
Moves the list of processes into array vars to iterate through instead.
If a failure occurs, the process name is visible along with line number in `_should_restart_when_killed()` to identify what went wrong.
* chore: Handle `FETCHMAIL_PARALLEL=1` process checks as well
* tests: Add test case for disabled ENV
Additional coverage to match what other test files were doing before, ensuring that these ENV can prevent their respective service from running.
* chore: Move `clamd` enabled check to it's own test case
Not sure about this.
It reduces the time of CPU activity (sustained full load on a thread) and increase in memory usage (1GB+ loading signatures database), but as a separate test case it also adds 10 seconds without reducing the time of the test case it was extracted from.
* chore: Make `disabled` variant the 1st test case
* fix: Adjust test cases to pass when using slower wrapper scripts
* tests(refactor): `mail_fetchmail.bats` updated to new format
Additionally merges in the parallel test file.
* chore: Move `config/fetchmail.cf` into separate sub-directory
Keep out of the default base config for tests.
* chore: Change `fetchmail.cf` FQDNs to `.test` TLD
Changed the first configs remote and local user values to more clearly document what their values should represent (_and that they don't need to be a full mail address, that's just what our Dovecot is configured with for login_).
Shifted the `here` to the end of the `is` line. It's optional syntax, only intended to contrast with the remote `there` for readability.
Additionally configured imap protocol. Not tested or verified if that's correct configuration for usage with imap protocol instead. The fetchmail feature tests are currently lacking.
Added an inline doc into the fetchmail test to reference a PR about the importance of the trailing `.` in the config. Updated the partial matching to ensure it matches for that in the value as well.
* chore: Finalize `process-check-restart.bats`
Few minor adjustments. The other ENV for clamd doesn't seem to provide any benefit, trim out the noise. Added a note about why it's been split out.
Fetchmail parallel configs are matching the config file path in the process command that is returned. The `.rc` suffix is just to add further clarity to that.
* tests(refactor): `mail_changedetector.bats` - Leverage DRY methods
`supervisorctl tail` is not the most reliably way to get logs for the latest change detection and has been known to be fragile in the past.
We can instead read the full log for the service directly with `tac` and `sed` to extract all log content since the last change detection.
Common asserts have also been extracted out into separate methods.
* tests(chore): Remove sleep and redundant change event
Container 1 is still blocked at this point from an existing lock and change event.
Make the lock stale immediately and no extra sleep is required when paired with the helper method to wait until the event is processed (which should remove the stale lock).
* tests(refactor): Add more DRY methods
- Simplify the test case so it's easier to grok.
- 2nd test case (blocking) extracts out initial setup into a separate method and merges the later service restart logic which is redundant.
- Additional comments for improved context of what is going on / expected.
* tests(chore): Revise the change detection helper method
- Add explicit counting arg to change detection support.
- Extract revised logic into it's own generic helper method.
- Utilize this for a separate method that monitors for a change event having started, but not waiting for completion.
This allows dropping the 40 sec of remaining `sleep` in `mail_changedetector` test. It was also required due to potentially missing the timing of a change event completing concurrently in a 2nd container that needed to be waited on and then checked.
* tests(chore): Migrate to current test conventions
- Switch to common container setup helpers
- Update container name and change usage to variables instead.
- Adopt the new convention of prefix variable for test cases (revised test case descriptions).
* tests(chore): Remove legacy change detection
This has since been replaced with the new helper watches the `changedetector` service logs directly instead of only detecting a change has occurred via checksum comparison.
No tests use this method anymore, it was originally for `tests.bats`. Thus the tests in `test_helper.bats` are being dropped too. The new helper has test coverage in `changedetector` tests.
* chore: Lock removal should not incur `sleep 5` afterwards
- A new lock should be created by this script after removal. The sleep doesn't help avoid a race condition with lock file creation after removal.
- Reduces test time as a bonus.
- Added some additional comments to test.
* tests(chore): `tls_letsencrypt.bats` leverage improved change detection
- No need to wait on the change detection service anymore during container startup.
- No need to count change events processed either as waiting a fixed duration is no longer relied on.
- This makes the reload count method redundant, dropped.
* tests(chore): Convert `setup-cli.bats` to new test conventions
This test file was already adapted to the original common setup helpers.
- `TEST_NAME` replaced with `CONTAINER_NAME`.
- Prefix var added, test case descriptions drop explicit prefix.
- No other changes.
* tests(chore): Extract out helpers related to change-detection
- New helper file for sharing these helpers to tests.
- Includes the helpful log method from changedetector tests.
- No longer need to maintain duplicate copies of these methods during the test migration. All tests that use them are now importing the separate helper file.
- `tls_letsencrypt.bats` has switched to using the log helper.
- Generic log count helper is removed from `test_helper/common.bash` as any test that needs it in future can adapt to `helper/common.bash`.
* tests(refactor): `tls_letsencrypt.bats` remove `_get_service_logs()`
This helper does not seem useful as moving away from `supervisorctl tail` and no other tests had a need for it.
* tests(chore): Remove common setup methods from `test_helper/common.bash`
No other tests depend on this. Future tests will adopt the revised versions from `helper/setup.bash`.
Additionally updates `helper/setup.bash` comments that are no longer applicable to `TEST_TMP_CONFIG` and `CONTAINER_NAME`.
* chore: Use `|| true` to simplify setting `EXPECTED_COUNT` correctly
* chore: Drop ENV `ENABLE_POSTFIX_VIRTUAL_TRANSPORT`
* tests(chore): Remove redundant `dovecot-lmtp` config
None of this is needed. Only relevant change is changing the LMTP service listener for Dovecot and that can be delegated to `user-patches.sh`.
* tests(refactor): Use `user-patches.sh` instead of replacing config file
The only relevant changes in `test/config/dovecot-lmtp` regarding LMTP was:
- `/etc/dovecot/dovecot.conf` (`protocols = imap lmtp`) and `/etc/dovecot/protocols.d/` (`protocols = $protocols lmtp`).
- `conf.d/10-master.conf` only changed the LMTP service listener from a unix socket to TCP on port 24 (_this was the only change required for the test to pass_).
None of those configs are required as:
- `protocols = imap pop3 lmtp` [is the upstream default](https://doc.dovecot.org/settings/core/#core_setting-protocols), no need to add `lmtp`.
- The LMTP service listener is now configured for the test with `user-patches.sh`.
* tests(refactor): `mail_lmtp_ip.bats`
- Converted to new testing conventions and common container helpers.
- `ENABLE_POSTFIX_VIRTUAL_TRANSPORT` was not relevant, dropped.
- Revised test cases, logic remains the same.
- Large custom config used was not documented and doesn't appear to serve any purpose. Simplified by replacing with a single modification with `user-patches.sh`.
- Added some additional comments for context of test and improvements that could be made.
* tests(chore): Adjust comments
The comment from `mail_hostname` provides no valid context, it was likely copied over from `tests.bats` in Oct 2020 by accident.
The email sent is just for testing, nothing relevant to LMTP.
---
Added additional comment for test to reference extra information from.
* tests(chore): Update similar log line matching
Extracts out the match pattern and formatting commands into separate vars (reduces horizontal scrolling), and includes extra docs about what the matched line should be expected to look like.
* chore: Remove `backup` target from Makefile
- The `backup` target is no longer serving any value to us. It was made redundant with changes added in Oct 2020.
- `clean` target inline docs revised.
- `.gitignore` remove test lines that are no longer valid.
* chore: Parallel test target split to multi-line
* tests(fix): Test `setup.sh` with temporary config dir
The `no_containers.bats` test has many redundant test cases already covered by `setup-cli`. They're basically identical. Removed all but one.
This removes some config dirs that were being explicitly created instead of using the test helper to generate a directory that can be used to test the `-p` option instead.
* ci: Ensure tests are run when `Makefile` is modified
* fix: Workaround `postconf` write settle logic
After updating `main.cf`, to avoid an enforced delay from reading the config by postfix tools, we can ensure the modified time is at least 2 seconds in the past as a workaround. This should be ok with our usage AFAIK.
Shaves off 2+ seconds roughly off each container startup, reduces roughly 2+ minutes off tests.
* chore: Only modify `mtime` if less than 2 seconds ago
- Slight improvement by avoiding unnecessary writes with a conditional check on the util method.
- Can more comfortably call this during `postfix reload` in the change detection cycle now.
- Identified other tests that'd benefit from this, created a helper method to call instead of copy/paste.
- The `setup email restrict` command also did a modification and reload. Added util method here too.
* tests(fix): `mail_smtponly.bats` should wait for Postfix
- `postfix reload` fails if the service is not ready yet.
- `service postfix reload` and `/etc/init.d/postfix reload` presumably wait until it is ready? (as these work regardless)
* chore: Review feedback - Move reload method into utilities
* tests(chore): `tls-dh-params.bats` - Drop `ONE_DIR` ENV variants
There is no longer special handling for this ENV with this feature, these variant test cases serve no value.
* tests(refactor): `tls-dh-params.bats`
Converted to new common setup helper methods and testing structure.
No `setup_file` needed. Only two test cases used now, the Mozilla check is bundled into the default params test case where it's relevant.
Refactored some logic into common functions. Should be easier to grok intention.
* chore: Apply review feedback
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* chore: Inline functions into test cases
As per review feedback
These two links have remained broken for over 6 months. Removing them.
* chore(housekeeping): Broken links
* chore: Remove broken links from `mailserver.env`
* docs: Certbot cloudflare
Add docs for implement certbot-dns-cloudflare to generate certificate for mail server
* Apply suggestions from code review
* fix: certbot-cloudflare docs
Fix the docker-compose command according to the advice
* feat: DNS-Cloudflare certificate renew
Add docs for implementing renewing certificate with crontab
* Apply suggestions from code review
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
While working on tests, I noticed that some of the configs being mounted were adding a few seconds to the start-up time of each container. Notably `postfix-*` and `dovecot.conf` config files, which have been extracted out into their own tests with those files moved into a separate config folder.
`tests.bats` has been adapted to the common setup helper, and removed ENV no longer required to run those tests. Future PRs will extract out more tests.
Review may be easier via individual commit diffs and their associated commit messages describing relevant changes.
<details>
<summary>Commit message history for reference</summary>
```markdown
tests(chore): `tests.bats` - Remove redundant config
===
- ONEDIR volume support no longer relevant, this should have been dropped.
- ClamAV ENV no longer relevant as related tests have been extracted already.
- Same with the some of the SpamAssassin ENV config.
- `VIRUSMAILS_DELETE_DELAY` is tested in the file, but doesn't use this ENV at all? (runs a separate instance to test the ENV instead)
- Hostname updated in preparation for migrating to new test helpers. Relevant test lines referencing the hostname have likewise been updated.
```
```markdown
tests(chore): `tests.bats` - Convert to common setup
===
ENV remains the same, but required adding `ENABLE_AMAVIS=1` to bring that back, while the following became redundant as they're now defaulting to explicitly disabled in the helper method:
- `ENABLE_CLAMAV=0`
- `LOG_LEVEL=debug`
- `ENABLE_UPDATE_CHECK=0`
- `--hostname` + `--tty` + standard `--volume` lines
- `-e` option expanded to long-name `--env`, and all `\` dropped as no longer necessary.
`wait_for_finished_setup_in_container` is now redundant thanks to `common_container_setup`.
```
```markdown
tests(refactor): `tests.bats` - Extract out Dovecot Sieve tests
===
Sieve test files relocated into `test/config/dovecot-sieve/` for better isolation.
`dovecot.sieve` was not using the `reject` import, and we should not encourage it? (docs still do):
https://support.tigertech.net/sieve#the-sieve-reject-jmp
```
```markdown
tests: `tests.bats` - Extract out `checking smtp` tests
===
Migrated to the standard template and copied over the original test cases with `_run_in_container` adjustment only.
Identified minimum required ENV along with which mail is required for each test case.
```
```markdown
tests(refactor): `smtp-delivery.bats`
===
- Disabled `ENABLE_SRS=1`, not necessary for these tests.
- Added a SpamAssassin related test (X-SPAM headers) which requires `SA_TAG` to properly pass (or `ENABLE_SRS=1` to deliver into inbox).
- Many lines with double quotes changed to single quote wrapping, and moving out `grep` filters into `assert_output --partial` lines instead.
- Instead of `wc -l` making failures less helpful, switch to the helper method `_should_output_number_of_lines`
- x2 `assert_output` with different EOF style of usage was not actually failing on tests when it should. Changed to assert partial output of each expected line, and count the number of lines instead.
- Added additional comments related to the test cases with a `TODO` note about `SPAMASSASSIN_SPAM_TO_INBOX=1`.
- Revised test case names, including using the common prefix var.
- `tests.bats` no longer needs to send all these emails, no other test cases require them. This affects a test checking a `/mail` folder exists which has been corrected, and a quotas test case adjusted to expect an empty quota size output.
```
```markdown
tests: `tests.bats` - Extract out test cases for config overrides
===
Slight improvement by additionally matching `postconf` output to verify the setting is properly applied.
```
```markdown
tests: `tests.bats` - Extract out Amavis SpamAssassin test case
===
Removes the need for SpamAssassin ENV in `tests.bats`.
```
</details>
tests: Convert more serial tests into parallel ready
<details>
<summary>Commit message history for reference</summary>
```markdown
tests(chore): Move some serial tests into parallel sets
===
Additionally with the `tls.bash` helper for the `letsencrypt` tests.
```
```markdown
tests: Adjusted files not directly related to tests
===
`tls.bash` helper was adapted to the new helper scripts location. The `setup.bash` helper saw a bugfix (expanding the array properly) and updates the container default config to configure for IPv4 explicitly.
The IPv4 default was added after recent Docker pushes and I saw weird IPv6 related errors in the logs.. now we're sure IPv4 is the default during tests.
Added functionality to check if a process is running:
- This change adds a helper function to check whether a program is running inside a container or not.
- This added the need for a function like `_run_in_container` but allowing for providing an explicit container name.
- Future PRs can use this helper function now to check whether a process is running or not. This was done for the tests of Fail2Ban, but can be used for other tests in the future as well.
---
chore: Restructured BATS flags in `Makefile`
The `Makefile` has seen a bit of a restructuring when it comes to flags:
1. The `MAKEFLAGS` variables is used by `make`, and allows for adding additional flags that can be used within in recursive calls (via `$(MAKE)`) too, thus DRY approach.
2. The flags for calling BATS were adjusted. `--no-parallelize-within-files` has been added as well to ensure tests _inside_ a single file are run sequentially.
`dms-test` prefix matching changed to expect a `_` suffix as a delimiter.
---
docs: Add a note regarding output from running tests in parallel
```
```markdown
tests: Adjust parallel tests
===
- The usual serial to parallel test conversion to utilize the `setup.bash` common setup structure, and adding a `TEST_PREFIX` var for each test case to leverage.
- Standardize on parallel test naming conventions for variables / values.
- More consistent use of `bash -c` instead of `/bin/bash -c` or `/bin/sh -c`.
- Using the `_run_in_container` helper instead of `run docker exec ${CONTAINER_NAME}`.
- Updates tests to use the `check_if_process_is_running` helper.
---
chore: Revise inline docs for the `ssl_letsencrypt` test
- Moves the override to be in closer proximity to the `initial_setup` call, and better communicates the intent to override.
- Removes top comment block that is no longer providing value or correct information to maintainers.
- Revised `acme.json` test case inline doc comments.
```
```markdown
refactor: Parallel Tests
===
- `disabled_clamav_spamassassin`:
- Just shuffling the test order around, and removing the restart test at the end which doesn't make sense.
- `postscreen`:
- Now uses common helper for getting container IP
- Does not appear to need the `NET_ADMIN` capability?
- Reduced startup time for the 2nd container + additional context about it's relevance.
- Test cases are largely the same, but refactored the `nc` alternative that properly waits it's turn. This only needs to run once. Added additional commentary and made into a generic method if needed in other tests.
- `fail2ban`:
- Use the common container IP helper method.
- Postscreen isn't affecting this test, it's not required to do the much slower exchange with the mail server when sending a login failure.
- IP being passed into ENV is no longer necessary.
- `sleep 5` in the related test cases doesn't seem necessary, can better rely on polling with timeout.
- `sleep 10` for `setup.sh` also doesn't appear to be necessary.
- `postgrey`:
- Reduced POSTGREY_DELAY to 3, which shaves a fair amount of wasted time while still verifying the delay works.
- One of the checks in `main.cf` doesn't seem to need to know about the earlier spamhaus portion of the line to work, removed.
- Better test case descriptions.
- Improved log matching via standard method that better documents the expected triplet under test.
- Removed a redundant whitelist file and test that didn't seem to have any relevance. Added a TODO with additional notes about a concern with these tests.
- Reduced test time as 8 second timeouts from `-w 8` don't appear to be required, better to poll with grep instead.
- Replaced `wc -l` commands with a new method to assert expected line count, better enabling assertions on the actual output.
- `undef_spam_subject`:
- Split to two separate test cases, and initialize each container in their case instead of `setup_file()`, allowing for using the default `teardown()` method (and slight benefit if running in parallel).
- `permit_docker`:
- Not a parallel test, but I realized that the repeat helper methods don't necessarily play well with `run` as the command (can cause false positive of what was successful).
```
```markdown
docs: Revise contributing advice for tests
```
</details>
- `disabled_clamav_spamassassin`:
- Just shuffling the test order around, and removing the restart test at the end which doesn't make sense.
- `postscreen`:
- Now uses common helper for getting container IP
- Does not appear to need the `NET_ADMIN` capability?
- Reduced startup time for the 2nd container + additional context about it's relevance.
- Test cases are largely the same, but refactored the `nc` alternative that properly waits it's turn. This only needs to run once. Added additional commentary and made into a generic method if needed in other tests.
- `fail2ban`:
- Use the common container IP helper method.
- Postscreen isn't affecting this test, it's not required to do the much slower exchange with the mail server when sending a login failure.
- IP being passed into ENV is no longer necessary.
- `sleep 5` in the related test cases doesn't seem necessary, can better rely on polling with timeout.
- `sleep 10` for `setup.sh` also doesn't appear to be necessary.
- `postgrey`:
- Reduced POSTGREY_DELAY to 3, which shaves a fair amount of wasted time while still verifying the delay works.
- One of the checks in `main.cf` doesn't seem to need to know about the earlier spamhaus portion of the line to work, removed.
- Better test case descriptions.
- Improved log matching via standard method that better documents the expected triplet under test.
- Removed a redundant whitelist file and test that didn't seem to have any relevance. Added a TODO with additional notes about a concern with these tests.
- Reduced test time as 8 second timeouts from `-w 8` don't appear to be required, better to poll with grep instead.
- Replaced `wc -l` commands with a new method to assert expected line count, better enabling assertions on the actual output.
- `undef_spam_subject`:
- Split to two separate test cases, and initialize each container in their case instead of `setup_file()`, allowing for using the default `teardown()` method (and slight benefit if running in parallel).
- `permit_docker`:
- Not a parallel test, but I realized that the repeat helper methods don't necessarily play well with `run` as the command (can cause false positive of what was successful).
- The usual serial to parallel test conversion to utilize the `setup.bash` common setup structure, and adding a `TEST_PREFIX` var for each test case to leverage.
- Standardize on parallel test naming conventions for variables / values.
- More consistent use of `bash -c` instead of `/bin/bash -c` or `/bin/sh -c`.
- Using the `_run_in_container` helper instead of `run docker exec ${CONTAINER_NAME}`.
- Updates tests to use the `check_if_process_is_running` helper.
---
chore: Revise inline docs for the `ssl_letsencrypt` test
- Moves the override to be in closer proximity to the `initial_setup` call, and better communicates the intent to override.
- Removes top comment block that is no longer providing value or correct information to maintainers.
- Revised `acme.json` test case inline doc comments.
`tls.bash` helper was adapted to the new helper scripts location. The `setup.bash` helper saw a bugfix (expanding the array properly) and updates the container default config to configure for IPv4 explicitly.
The IPv4 default was added after recent Docker pushes and I saw weird IPv6 related errors in the logs.. now we're sure IPv4 is the default during tests.
Added functionality to check if a process is running:
- This change adds a helper function to check whether a program is running inside a container or not.
- This added the need for a function like `_run_in_container` but allowing for providing an explicit container name.
- Future PRs can use this helper function now to check whether a process is running or not. This was done for the tests of Fail2Ban, but can be used for other tests in the future as well.
---
chore: Restructured BATS flags in `Makefile`
The `Makefile` has seen a bit of a restructuring when it comes to flags:
1. The `MAKEFLAGS` variables is used by `make`, and allows for adding additional flags that can be used within in recursive calls (via `$(MAKE)`) too, thus DRY approach.
2. The flags for calling BATS were adjusted. `--no-parallelize-within-files` has been added as well to ensure tests _inside_ a single file are run sequentially.
`dms-test` prefix matching changed to expect a `_` suffix as a delimiter.
---
docs: Add a note regarding output from running tests in parallel
* chore: Set `TLS_INTERMEDIATE_SUITE` to only use TLS 1.2 ciphersuites
Removes support of the following cipher suites that are only valid for TLS 1.0 + 1.1:
- `ECDHE-ECDSA-AES128-SHA`
- `ECDHE-RSA-AES128-SHA`
- `ECDHE-ECDSA-AES256-SHA`
- `ECDHE-RSA-AES256-SHA`
- `DHE-RSA-AES128-SHA`
- `DHE-RSA-AES256-SHA`
* chore: Update TLS version min and ignore settings
These are now the same as modern settings.
* fix: Remove min TLS support workaround
No longer required now that outdated TLS versions have been dropped.
* tests: Remove support for TLS 1.0 and 1.1 ciphersuites
* tests: Remove support for TLS 1.0 and 1.1 ciphersuites (Port 25)
The removed SHA1 cipher suites are still supported in TLS 1.2, thus they've been excluded for port 25 via the `SHA1` exclusion pattern in `main.cf`.
With `reload` a change detection event during local testing can be processed in less than a second according to logs. Previously this was 5+ seconds (_plus additional downtime for Postfix/Dovecot to become available again_).
In the past it was apparently an issue to use `<service> reload` due to a concern with the PID for wrapper scripts that `supervisorctl` managed, thus `supervisorctl <service> restart` had been used. Past discussions with maintainers suggest this is not likely an issue anymore, and `reload` should be fine to switch to now 👍
---
**NOTE:** It may not be an issue in the CI, but on _**local systems running tests may risk failure in `setup-cli.bats` from a false positive**_ due to 1 second polling window of the test helper method, and a change event being possible to occur entirely between the two checks undetected by the current approach.
If this is a problem, we may need to think of a better way to catch the change. The `letsencrypt` test counts how many change events are expected to have been processed, and this could technically be leveraged by the test helper too.
---
**NOTE:** These two lines (_with regex pattern for postfix_) are output in the terminal when using the services respective `reload` commands:
```
postfix/master.*: reload -- version .*, configuration /etc/postfix
dovecot: master: Warning: SIGHUP received - reloading configuration
```
I wasn't sure how to match them as they did not appear in the `changedetector` log (_**EDIT:** they appear in the main log output, eg `docker logs <container name>`_).
Instead I've just monitored the `changedetector` log messages, which should be ok for logic that previously needed to ensure Dovecot / Postfix was back up after the `restart` was issued.
---
Commit history:
* chore: Change events `reload` Dovecot and Postfix instead of `restart`
Reloading is faster than restarting the processes.
Restarting is a bit heavy handed here and may no longer be necessary for general usage?
* tests: Adapt tests to support service `reload` instead of `restart`
* chore: Additional logging for debugging change event logs
* fix: Wait on change detection, then verify directory created
Change detection is too fast now (0-1 seconds vs 5+).
Directory being waited on here was created near the end of a change event, reducing that time to detect a change by the utility method further.
We can instead check that the directory exists after the change detection event is completed.
* chore: Keep using the maildir polling check
We don't presently use remote storage in tests, but it might be relevant in future when testing NFS.
This at least avoids any confusing failure happening when that scenario is tested.
As per deprecation notice from v11.3 release notes, and a related prior PR; this ENV is to be removed.
It's no longer considered useful, and none of the tests that configured it were actually using it for relaying anything.
## Pull Request
### ci: Run tests in parallel (part 1)
- Tests can now be arbitrarily grouped into sub-directories.
- Some tests are now run in parallel.
- CI will now spawn 4 jobs to run the whole test suite in parallel.
## Individual Commits
### tests(chore): Rename test files to serial and parallel types
- `test_helper.bats` needs more work than this PR provides to be compatible with parallel tests, so must remain as a serial test for now.
- `spam_bounced.bats` had failures as a serial test, but works well converted to a parallel test in a future commit.
### tests(CI): Adjust Makefile & GHA workflow to support new test layout
These updates support running tests that have been relocated into `serial` and `parallel/set*` directories.
- `make tests` now calls the two make targets beneath it. The only difference is that `serial` continues the "1 test at a time" approach used prior to this PR, while the `parallel` target increases the `--jobs` arg to run multiple tests concurrently (_configured by `PARALLEL_JOBS`_).
- The `test/%` target leverages Bash syntax magic to ease running single tests without providing the exact path.
- This syntax also supports providing multiple test names (eg: `make test/clamav,template`) to run.
- `**` (globstar) allows for future improvements that can group multiple test files into sub-directories by their scope (eg: anti-spam, ssl, etc).
---
chore: Add `shopt -s globstar` to other targets
I realized that other targets should have this as well in case it is not set.
It is better to be more explicit here than to have weird errors due to `**` not expanding properly.
---
fix(Makefile): Add back `.PHONY` targets
I encountered `make` telling me the target was already up-to-date, which of course is nonsense.
I therefore added back the `.PHONY` targets to ensure tests are always run.
---
docs: Added instructions for running a single test
See https://github.com/docker-mailserver/docker-mailserver/pull/2857/files#r1008582760
### tests(chore): Use `REPOSITORY_ROOT` export var from Makefile
Allows for using `load` with an absolute path instead of a relative one, which makes it possible to group tests into different directories.
Parallel tests differ slightly, loading the newer `helper/common.bash` and `helper/setup.bash` files instead of the older `test_helper/common.bash` which serial tests continue to use.
### tests(refactor): `common.bash` helper split into two files
The current `test/test_helper/common.bash` was getting large. Setup logic has been extracted out into a new file.
`common.bash` resides in a directory named `test_helper/`, the `test_` prefix is redundant.
As an interim solution this provides a new approach for the updated tests, while the "old" tests can use the "old" `common.bash`. Eventually all tests should migrate to the new approach in `helper/` instead of the older `test_helper/`.
The new helper files are located under `test/helper/` (_which drops the `test_` prefix_). The new and updated helpers apply the new naming convention for ENV variables (_such as `CONTAINER_NAME` or `IMAGE_NAME`_).
---
Some refactoring occurred, including new methods like `_run_in_container()` and `_default_teardown()`.
---
I encountered a situation before in which the updated tests would fail because there were collisions of ENV names in the tests (_for example with `CONTAINER_NAME`_).
### tests(refactor): Conversion to parallel tests and use revised helpers
- Introduced `CONTAINER_NAME` and `TEST_NAME_PREFIX` as new vars for better managing test consistency (DRY).
- `CONTAINER_NAME` replaces any repeated container name with the variable. The value will differ slightly as the prior prefix (`mail_`) has been changed to `dms-test-`.
- `TEST_NAME_PREFIX` provides a prefix value for each `@test` description string.
---
chore: Add a reference template for tests
- Introduced `CONTAINER_NAME` and `TEST_NAME_PREFIX` as new vars for better managing test consistency (DRY).
- `CONTAINER_NAME` replaces any repeated container name with the variable. The value will differ slightly as the prior prefix (`mail_`) has been changed to `dms-test-`.
- `TEST_NAME_PREFIX` provides a prefix value for each `@test` description string.
---
chore: Add a reference template for tests
The current `test/test_helper/common.bash` was getting large. Setup logic has been extracted out into a new file.
`common.bash` resides in a directory named `test_helper/`, the `test_` prefix is redundant.
As an interim solution this provides a new approach for the updated tests, while the "old" tests can use the "old" `common.bash`. Eventually all tests should migrate to the new approach in `helper/` instead of the older `test_helper/`.
The new helper files are located under `test/helper/` (_which drops the `test_` prefix_). The new and updated helpers apply the new naming convention for ENV variables (_such as `CONTAINER_NAME` or `IMAGE_NAME`_).
---
Some refactoring occurred, including new methods like `_run_in_container()` and `_default_teardown()`.
---
I encountered a situation before in which the updated tests would fail because there were collisions of ENV names in the tests (_for example with `CONTAINER_NAME`_).
Allows for using `load` with an absolute path instead of a relative one, which makes it possible to group tests into different directories.
Parallel tests differ slightly, loading the newer `helper/common.bash` and `helper/setup.bash` files instead of the older `test_helper/common.bash` which serial tests continue to use.
These updates support running tests that have been relocated into `serial` and `parallel/set*` directories.
- `make tests` now calls the two make targets beneath it. The only difference is that `serial` continues the "1 test at a time" approach used prior to this PR, while the `parallel` target increases the `--jobs` arg to run multiple tests concurrently (_configured by `PARALLEL_JOBS`_).
- The `test/%` target leverages Bash syntax magic to ease running single tests without providing the exact path.
- This syntax also supports providing multiple test names (eg: `make test/clamav,template`) to run.
- `**` (globstar) allows for future improvements that can group multiple test files into sub-directories by their scope (eg: anti-spam, ssl, etc).
---
chore: Add `shopt -s globstar` to other targets
I realized that other targets should have this as well in case it is not set.
It is better to be more explicit here than to have weird errors due to `**` not expanding properly.
---
fix(Makefile): Add back `.PHONY` targets
I encountered `make` telling me the target was already up-to-date, which of course is nonsense.
I therefore added back the `.PHONY` targets to ensure tests are always run.
---
docs: Added instructions for running a single test
See https://github.com/docker-mailserver/docker-mailserver/pull/2857/files#r1008582760
- `test_helper.bats` needs more work than this PR provides to be compatible with parallel tests, so must remain as a serial test for now.
- `spam_bounced.bats` had failures as a serial test, but works well converted to a parallel test in a future commit.
Currently a change detection would be triggered and during processing, a CRLF is converted to LF, which updates the `postfix-accounts.cf` file and triggers another change event.
No need for the first approach to add an account, and it is the culprit for causing the CRLF to appear.
This new script is a clean way of handling the installation of packages.
I think the huge `RUN` command in `Dockerfile` was hard to read and
maintain.
Using a script is a non-issue, as the image is rebuilt whenever the
script is touched.
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
The build arguments `VCS_REF` and `VCS_VER` were renamed and given
proper values according to their names.
1. `VCS_REVISION` holds the current SHA sum of the (git) HEAD pointer
2. `VCS_VERSION` now holds the contents of the `VERSION` file, i.e. a
semver version tag (one can now inspect the image and find a proper
version tag in the `org.opencontainers.image.version` label)
The build arguments were given defaults in order to allow the
`generic_build` and `generic_test` workflows to omit them (as they are
not need there anyways). When publishing images, this is fina as the
cache will rebuild almost all of the image except the last few layers
which are `LABEL`s anyways.
Mew re-usable workflows are introduced to handle building, testing and publishing the container
image in a uniform and easy way. Now, the `scheduled_builds`, `default_on_push`
and a part of the `test_merge_requests` workflow can use the same code
for building, testing and publishing the container images. This is DRY.
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
This PR prepares for other PRs that use the newly introduced helper
functions. The `_log` function itself was adjusted a bit to be shorter
and more concise.
* ci: Cache builds by splitting into two jobs
For the cache to work properly, we need to derive a cache key from the build context (files that affect the Dockerfile build) instead of the cache key changing by commit SHA.
We also need to avoid a test suite failure from preventing the caching of a build, thus splitting into separate jobs.
This first attempt used `upload-artifact` and `download-artifact` to transfer the built image, but it has quite a bit of overhead and prevented multi-platform build (without complicating the workflow further).
* ci: Transfer to dependent job via cache only
While `download-artifact` + `docker load` is a little faster than rebuilding the image from cached layers, `upload-artifact` takes about 2 minutes to upload the AMD64 (330MB) tar image export (likely due to compression during upload?).
The `actions/cache` approach however does not incur that hit and is very quick (<10 secs) to complete it's post upload work. The dependent job still gets a cache-hit, and the build job is able to properly support multi-platform builds.
Added additional notes about timing and size of including ARM builds.
* ci: Move Dockerfile ARG to end of build
When the ARG changes due to commit SHA, it invalidates all cache due to the LABEL layers at the start. Then any RUN layers implicitly invalidate, even when the ARG is not used.
Introduced basic multi-stage build, and relocated the container config / metadata to the end of the build. This avoids invalidating expensive caching layers (size and build time) needlessly.
* tests: Ensure excessive FD limits are avoided
Processes that run as daemons (`postsrsd` and `fail2ban-server`) initialize by closing all FDs (File Descriptors).
This behaviour queries that maximum limit and iterates through the entire range even if only a few FDs are open. In some environments (Docker, limit configured by distro) this can be a range exceeding 1 billion (from kernel default of 1024 soft, 4096 hard), causing an 8 minute delay with heavy CPU activity.
`postsrsd` has since been updated to use `close_range()` syscall, and `fail2ban` will now iterate through `/proc/self/fd` (open FDs) which should resolve the performance hit. Until those updates reach our Docker image, we need to workaround it with `--ulimit` option.
NOTE: If `docker.service` on a distro sets `LimitNOFILE=` to approx 1 million or lower, it should not be an issue. On distros such as Fedora 36, it is `LimitNOFILE=infinity` (approx 1 billion) that causes excessive delays.
* chore: Use Docker host limits instead
Typically on modern distros with systemd, this should equate to 1024 (soft) and 512K (hard) limits. A distro may override the built-in global defaults systemd sets via setting `DefaultLimitNOFILE=` in `/etc/systemd/user.conf` and `/etc/systemd/system.conf`.
* tests(fix): Better prevent non-deterministic failures
- `no_containers.bats` tests the external script `setup.sh` (without `-c`). It's expected that no existing DMS container is running - otherwise it may attempt to use that container and fail. Detect this and fail early via `setup_file()` step.
- `mail_hostname.bats` had a odd timing failure with teardown due to the last tests bringing the containers down earlier (`docker stop` paired with the `docker run --rm`). Adding a moment of delay via `sleep` helps avoid that false positive scenario.
## Quick Summary
Resolves a `TODO` task with `addmailuser`.
## Overview
The main change is adding three new methods in `common.bash`, which replace the completion delay in `addmailuser` / `setup email add` command.
Other than that:
- I swapped `sh -c 'addmailuser ...'` to `setup email add ...`.
- Improved three tests in `setup-cli.bats` for `setup email add|update|del` (_logic remains effectively the same still_).
- Rewrote the `TODO` comment for `setup-cli.bats` test on `setup email del` to better clarify the concern, but the test itself was no longer affected due to changes prior to this PR, so I enabled the commented out assertion.
- Removed unnecessary waits. The two `skip` tests in `test/tests.bats` could be enabled again after this PR.
- Additional fixes to tests were made during the PR (see discussion comments for details), resolving race conditions.
Individual commit messages of the PR provide additional details if helpful.
---
## Relevant commit messages
* chore: Remove creation delay in `addmailuser`
This was apparently only for supporting tests that need to wait on account creation being ready to test against.
As per the removed inline docs, it should be fine to remove once tests are updated to work correctly without it.
* tests(feat): Add two new common helper methods
`wait_until_account_maildir_exists()` provides the same logic `addmailuser` command was carrying, to wait upon the account dir creation in `/var/mail`.
As this was specifically to support tests, it makes more sense as a test method.
`add_mail_account_then_wait_until_ready()` was added to handle the common pattern of creating account and waiting on it. An internal assert will ensure the account was successfully created first during the test before attempting to wait.
* tests(feat): Add common helper for waiting on change event to be processed
The current helper is more complicated for no real benefit, it only detects when a change is made that would trigger a change event in the `changedetector` service. Our usage of this in tests however is only interested in waiting out the completion of the change event.
Remove unnecessary change event waits. These waits should not be necessary if handled correctly.
* tests: `addmailuser` to `add_mail_account_then_wait_until_ready mail()`
This helper method is used where appropriate.
- A password is not relevant (optional).
- We need to wait on the creation on the account (Dovecot and `/var/mail` directory).
* tests: `setup-cli` revise `add`, `update`, `del` tests
The delete test was failing as the `/var/mail` directory did not yet exist.
There is now a proper delay imposed in the `add` test now shares the same account for both `update` and `del` tests resolving that failure.
Additionally tests use better asserts where appropriate and the wait + sleep logic in `add` has been improved (now takes 10 seconds to complete, approx half the time than before).
The `del` test TODO while not technically addressed is no longer relevant due to the tests being switched to `-c` option (there is a separate `no container` test file, but it doesn't provide a `del` test).
* tests(fix): Ensure Postfix is reachable after waiting on ClamAV
There is not much reason to check before waiting on ClamAV.
It is more helpful to debug failures from `nc` mail send commands if we know that nothing went wrong inbetween the ClamAV wait time.
Additionally added an assertion which should provide more information if this part of the test setup fails again.
* tests(fix): Move health check to the top
This test is a bit fragile. It relies on defaults for the healthcheck with intervals of 30 seconds.
If the check occurs while Postfix is down due a change event from earlier tests and the healthcheck kicks in at that point, then if there is not enough time to refresh the health status from `unhealthy`, the test will fail with a false-positive as Postfix is actually working and up again..
* tests(fix): Wait on directory to be removed
Workaround that tries not to introduce heavier delays by waiting on a full change event to complete in the previous `email update` if possible.
There is a chance that the account has the folder deleted, but restored from an active change event (for password update, then the account delete).
The new version uses our `log.sh` helper to simplify logging
significantly. Moreover, the script was adjusted to the current style
and the GitHub workflow was streamlined. The workflow is ot providing
the version anymore (which was useless anyway), and has been compacted.
* outsourcing env variable setup
This commit contains major parts of the work of refactoring the setup
and usage of environment variables. It outsources the setup into its own
script and provides dedicated functions to be executed at a later point in time.
A **new** env variable was added: `USER_PROVISIONG` which provides a
better way of defining which method / protocol to use when it comes to
setting up users. This way, the `ENABLE_LDAP` variable is deprecated,
but all of this is backwards compatible due to a "compatibility layer", a function provided by the new variables script.
This is not a breaking change. It mostly refators internal scripts. The
only change facing the user-side is the deprecation of `ENABLE_LDAP`. We
can prolong the period of deprecation for this variable as long as we
want, because the new function that ensures backwards compatibility
provides a clean interface for the future.
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
When waiting on an account to be added to `postfix-accounts.cf`, Dovecot account creation during the startup process had already run.
Startup continued without properly creating the mail account for Dovecot. Methods like `setup email list` (with `ENABLE_QUOTAS=1`) would fail. `changedetector` service was required to be triggered to re-create Dovecot users.
- Wrapped the logic for wait + shutdown into a function call.
- Moved `_create_accounts()` to bottom of the setup function.
* tests: Update testing submodules (bats-assert, bats-support)
These two submodules were migrated to the `bats-core` organization, where they continued to receive updates.
* tests: Use tagged release of `bats-core/bats-support`
This is technically one commit backwards, but no relevant difference has been made since, other than moving the submodule to the `bats-core` organization.
* tests: Bump `bats-assert` to August 2022 (master)
No official release tag since Nov 2018, but a fair amount of changes since then.
* tests: Bump `bats-core` to `v1.7.0` release
* tests(fix): Correctly use assertions
Some tests were updated as the upgrade of bats submodules had `assert` methods raise awareness of incorrect usage.
This additionally revealed some existing tests that weren't meant to be using `run`, which swallowed failures from surfacing.
See the associated PR for more detailed commentary on specific changes.
### Commands refactored:
- User (**All:** add / list / update / del + _dovecot-master variants_)
- Quota (**All:** set / del)
- Virtual Alias (**All:** add / list /del)
- Relay (**All:** add-relayhost / add-sasl / exclude-domain)
### Overall changes involve:
- **Fairly common structure:**
- `_main` method at the top provides an overview of logical steps:
- After all methods are declared beneath it (_and imported from the new `helpers/database/db.sh`_), the `_main` is called at the bottom of the file.
- `delmailuser` additionally processes option support for `-y` prior to calling `_main`.
- `__usage` is now consistent with each of these commands, along with the `help` command.
- Most logic delegated to new helper scripts. Some duplicate content remains on the basis that it's low-risk to maintenance and avoids less hassle to jump between files to check a single line, usually this is arg validation.
- Error handling should be more consistent, along with var names (_no more `USER`/`EMAIL`/`FULL_EMAIL` to refer to the same expected value_).
- **Three new management scripts** (in `helpers/database/manage/`) using a common structure for managing changes to their respective "Database" config file.
- `postfix-accounts.sh` unified not only add and update commands, but also all the dovecot-master versions, a single password call for all 4 of them, with a 5th consumer of the password prompt from the relay command `addsaslpassword`.
- These scripts delegate actual writes to `helpers/database/db.sh` which provides a common API to support the changes made.
- This is more verbose/complex vs the current inline operations each command currently has, as it provides generic support instead of slightly different variations being maintained, along with handling some edge cases that existed and would lead to bugs (notably substring matches).
- Centralizing changes here seems wiser than scattered about. I've tried to make it easy to grok, hopefully it's not worse than the current situation.
- List operations were kept in their respective commands, `db.sh` is only really managing writes. I didn't see a nice way for removing the code duplication for list commands as the duplication was fairly minimal, especially for `listalias` and `listdovecotmasteruser` which were quite simple in their differences in the loop body.
- `listmailuser` and `delmailuser` also retain methods exclusive to respective commands, I wasn't sure if there was any benefit to move those, but they were refactored.
* fix: Reload `amavisd-new` when vhost config is updated
Amavis was not aware of new domains in `/etc/postfix/vhost` as it did not refresh it's sources upon change detection.
- Inline docs for `check-for-changes.sh` have been shuffled around and revised a bit.
- Change processing extracted from the main change detection loop method to their own methods:
- `_get_changed_files()` - Clarifies what is going on (and how) without having to look it up. To reduce noise in the main logic loop, extracted to a separate method.
- `_postfix_dovecot_changes()` - The bulk of change processing was moved to this method. I've added conditionals to only run relevant logic.
- `_ssl_changes()` - Just shifted, no logic changed. `REGEX_NEVER_MATCH` and `ACME_CERT_DIR` vars scope set to `local`.
* chore: Split vhost helper method and use filepath vars
- Helpers `accounts.sh` and `aliases.sh` can move their vhost code into this helper.
- They share duplicate code with `bin/open-dkim` which will also leverage this vhost helper going forward.
* chore: Sync vhost generation logic into helper
- Chunky commit, but mostly copy/paste of logic into a common method.
- `bin/open-dkim` additionally wrapped relevant logic in a function call and revised inline docs.
* chore: Include LDAP vhost support
- Revises notes for LDAP vhost support.
- This now ensures LDAP users get vhost rebuilt to match the startup script for when change detection support is enabled.
- `bin/open-dkim` will additionally be able to support the default `DOMAINNAME` var (set via `helpers/dns.sh`) for LDAP users instead of requiring them to provide one explicitly.
* chore(`bin/open-dkim`): Ensure `DOMAINNAME` is properly set
- This will ensure LDAP users insert the same `DOMAINNAME` value as used during container startup.
- The container itself should panic at startup (during `helpers/dns.sh`) if this isn't configured correctly already, thus it should not introduce any breaking change to users of this utility?
* chore: Set the 2nd value as blank `_`
Line is split by a delimiter such as white-space (or via IFS: `|`), the blank `_` var is to indicate we're not interested in that value, but still leverage how `read -r` works, instead of splitting the var ourselves first thing.
* chore: Remove shellcheck disable lines
No longer applicable with the switch to `_`
* chore: Remove requirement for `postfix-accounts.cf`
This is an old requirement from when the change detector service was first introduced. It's no longer relevant.
* chore: Do not needlessly create `postfix-aliases.cf`
The config was created regardless to workaround early change detection support. No longer necessary to require the file to exist.
* chore: Drop guards requiring `/tmp/docker-mailserver` to exist
Legacy guards when this was the only location change detection location supported.
There does not appear to be any need for changing into this directory at the start of `check-for-changes.sh` as we use absolute filepaths (originally monitored files were checked with relative paths to this config dir).
* chore: Revise inline docs
* chore: Add change detection monitoring for extra configs
These are also handled at run-time in the current change detection support, so it makes sense to allows these config updates to also trigger change events.
* chore: Create bare new test file `setup-cli.bats`
Bare minimum to setup a new test.
* chore: Transfer over relevant tests
* chore: `mail` container name to dynamic `${TEST_NAME}`
Only applied where it's relevant. Next commit will handle the config path correction.
* chore: Use `TEST_TMP_CONFIG` for referencing local config directory
Could technically use the existing function call. Some paths were using a hard-coded config location.
Both have been converted to `TEST_TMP_CONFIG` and related `grep` calls normalizing the quote mark usage, escaping doesn't seem necessary.
* tests(fix): Create container without providing extra args reference var
If a variable name (of an array) was not provided to reference, this would fail trying to reference `'`.
* chore: Remove `SYS_PTRACE` capability from docs and configs
* chore: Remove `SYS_PTRACE` capability from tests
Doesn't seem to be required. It was originally added when the original change detection feature PR apparently needed it to function.
* chore(`aliases.sh`): Filepath to local var `DATABASE_VIRTUAL`
* chore(`accounts.sh`): Filepath to local var `DATABASE_ACCOUNTS`
* chore(`accounts.sh`): Filepath to local var `DATABASE_VIRTUAL`
* chore(`accounts.sh`): Filepath to local var `DATABASE_DOVECOT_MASTERS`
* chore(`bin/open-dkim`): Filepaths to local vars (accounts,virtual,vhost)
* chore(`relay.sh`): Filepath to local var `DATABASE_SASL_PASSWD`
* chore: Rename method
Prior PR feedback suggested a better helper method name.
* chore: Normalize filtering config lines as input for iterating
* chore: Remove `_is_comment` helper method
No longer serving a purpose with more appropriate filter method for pre-processing the entire config file.
* fix(listmailuser): Don't parse comments
Avoids passing comments to `dovecot_quota_to_hr()` which fails to handle it and would throws errors.
* chore: Move config filter method to `helpers/utils.sh`
This helper was to support an earlier ENV for SASL auth support. When extracting logic into individual helpers, it was assumed this was separate from relay support, which it appears was not the case.
---
The `SASL_PASSWD` ENV is specified in tests but no longer used. There is no `external-domain.com` relay configured or tested against anywhere in the project.
The ENV was likely used in tests prior to improved relay support that allowed for adding more than a single set of relay credentials.
---
It likewise has no real relevance anywhere else outside of `relay.sh` as it's the only portion of code to operate with it.
It's only relevant for SASL auth as an SMTP client, not the SMTP server (`smtpd`) SASL support that is delegated to Dovecot. Functionality has been completely migrated into `relay.sh` as a result.
Documentation is poor for this ENV, it is unlikely in wide use? Should consider for removal.
---
The ENV has been dependent upon `RELAY_HOST` to actually enable postfix to use `/etc/postfix/sasl_passwd`, thus not likely relevant in existing setups?
---
Migrate `/etc/postfix/sasl_passwd` check from `tests.bats` as it belongs to relay tests.
* chore: Fix typo
* chore: Apply explicit chroot default for `sender-cleanup`
The implicit default is set to `y` as a compatibility fallback, but otherwise it is [advised to set to `n` going forward](http://www.postfix.org/COMPATIBILITY_README.html#chroot).
Test was changed to catch any backwards-compatibility logs, not just those for `chroot=y`. `using` added as a prefix to avoid catching log message whenever a setting is changed that the default compatibility level is active.
* chore: Set `compatibility_level` in `main.cf`
We retain the level`2` value previously set via scripts. This avoids log noise that isn't helpful.
Applied review feedback to give maintainers some context with this setting and why we have it presently set to `2`.
Presently relay-host support modifies `main.cf` settings directly. This adjusts the default transport (`smtp`) which other transports in `master.cf` inherit from.
When configuring for implicit TLS to a `relay-host` this would set `main.cf:smtp_tls_wrappermode = yes` and affect the transport `master.cf:smtp-amavis` which does not set an override like it does for `smtp_tls_security_level`. This causes Amavis to fail working which the default transport relies on due to `main.cf:content_filter`.
Easy fix, by explicitly adding the override `smtp_tls_wrappermode=no`.`no` is default in `main.cf` so inheriting this setting hasn't been a problem in the past for other relay-hosts using StartTLS.
* fix: Conditionally add service state
These services will no longer copy over state unless they are enabled.
The biggest offender here was ClamAV as it's database that is baked into the docker image is over 200MB and would copy over to every container instance with a volume mounted state directory.
* chore: Add Dovecot to conditional support
* chore: Extract change-detection method to it's own helper
This doesn't really belong in `helpers/ssl.sh`. Moving to it's own helper script.
* chore: Co-locate related change-detection method from container startup
It seems relevant to migrate the related support during startup for the change detection feature into this helper.
I opted to move the call from `start-mailserver.sh` into the `_setup` call at the end for a more explicit/visible location.
* chore: Move `CHKSUM_FILE` into `helpers/change-detection.sh`
It belongs there, not in `helpers/index.sh`.
* chore: Revise inline documentation
* tests(fix): Ensure correct functionality
Presently `test/test_helper.bats` is using it's own `CHKSUM_FILE` instead of sourcing the var for the filepath.
`test_helper/common.bash` was calling a method to check for changes, but this helper may not correctly detect letsencrypt related changes as these are not ENV rely on, but global vars handled by `helpers/dns.sh`, so that should be run first like it is for `check-for-changes.sh`.
* tests(chore): Use `CHKSUM_FILE` var from helper
* chore: `addmailuser` should use `CHKSUM_FILE` var
* chore: Update `check-for-changes.sh` log message with correct path
* chore: Make `_populate_relayhost_map` easier to grok
Changes to `sed` handling that made it quicker to grok, and thus easier for maintainers like myself:
- Switched regex to [extended regex](https://www.gnu.org/software/sed/manual/html_node/Extended-regexps.html).
- Extracted `sed` patterns to be self-descriptive local vars.
- Used a function to reduce noise from intent of loop input (each line as `DOMAIN_PART`).
Input for the loop is filtered through `sort -u` to drop duplicates, reducing iterations.
`DOMAIN` loop var renamed to less vague `DOMAIN_PART`. Additional comment in the containing method clarifies what the domain part refers to.
---
`|` regexp syntax needed to be escaped due to switch. Not documented in the earlier link. `-r`/`-E` (ERE) aka extended regexp syntax is [detailed here](https://learnbyexample.github.io/learn_gnused/breere-regular-expressions.html#cheatsheet-and-summary).
* chore: Drop unnecessary postfix parameters
`smtp_tls_note_starttls_offer = yes` - Only adds a log entry to let you know when an unencrypted connection was made, but STARTTLS was offered:
https://www.postfix.org/postconf.5.html#smtp_tls_note_starttls_offer
`smtp_tls_CAfile` is unnecessary. This was added before `smtp_tls_CApath = /etc/ssl/certs` was several months later via a separate PR.
* chore: Move `smtp_` parameters to relevant sections
These have been shifted to relevant logic for now.
---
NOTE: `SASL_PASSWD` previously needed to define `RELAY_HOST` to set `smtp_sasl_password_maps` to enable the `/etc/postfix/sasl_passwd` table. This change now additionally blocks early on in `_relayhost_sasl`. Not likely important due to `RELAY_HOST` logic, user should be using the `RELAY_USER` + `RELAY_PASSWORD` ENV or `postfix-sasl-password.cf` instead.
Especially the sender dependent parameters which are only relevant with user provided configs really.
`SASL_PASSWD` is the oldest ENV for relay support before any other relay feature arrived. It is poorly documented and should not be used.
Potential breakage risk considered acceptable.
* chore: Revise inline docs
Further clarifying current processing logic and adding some additional notes for future work.
* chore: Use a common ENV relay-host getter
The mapping should be in sync between the two configs.
I also wanted to raise awareness of current state of support, which will likely need some refactoring.
This also removes the need for the `RELAY_PORT` fallback method.
The log message was adjusted as configuration is potentially for more than one relay host beyond the currently required ENV config to enable support.
---
NOTE: The ENV `DEFAULT_RELAY_HOST` skips modifying the default transport for an authenticated relay (locked behind `RELAY_HOST` to activate). It presently will only relay mail through a relay host on port 25 instead of delivering directly to the destination. A separate use-case.
* chore: Revise config examples
More verbose example configs with expanded documentation.
Additional doc references for SASL support and cautioning maintainers that may reference popular relay service providers docs. May later be migrated to a "maintainers" section in official docs and link to that.
Brief overview description of what `_populate_relayhost_map` is doing.
* chore: Add notes pertaining to future work
`_populate_relayhost_map` will get some refactoring in future and likely introduce some breaking changes for a future major release.
* chore: Better document relay support inline
This helper now includes a description of it's purpose, links to relevant user docs and supported `setup.sh` commands.
Intent is to keep a maintainer of the feature aware of anything relevant to this feature.
* tests(fix): Increase some timeouts
Running tests locally via a VM these tests would fail sometimes due to the time from being queued and Amavis actually processing being roughly around 30 seconds.
There should be no harm in raising this to 60 seconds, other than delaying a failure case which will ripple through other time sensitive tests.
It's better to pass when functionality is actually correct but just needs a bit longer to complete.
* tests(fix): Don't setup an invalid hostname
During container startup `helpers/dns.sh` would panic with `hostname -f` failing.
Dropping `--domainname` for this container is fine and does not affect the point of it's test.
---
It's unclear why this does not occur in CI. Possibly changes within the docker daemon since as CI runs docker on Ubuntu 20.04? (2020).
For clarity, this may be equivalent to setting a hostname of `domain.com.domain.com`, or `--hostname` value truncated the NIS domain (`--domainname`) of the same value.
IIRC, it would still fail with both options using different values if `--hostname` was multi-label. I believe I've documented how non-deterministic these options can be across different environments.
`--hostname` should be preferred. There doesn't seem to be any reason to actually need `--domainname` (which is NIS domain name, unrelated to the DNS domain name). We still need to properly investigate reworking our ENV support that `dns.sh` manages.
---
Containers were also not removing themselves after failures either (missing teardown). Which would cause problems when running tests again.
* chore: Normalize white-space
Sets a consistent indent size of 2 spaces. Previously this varied a fair bit, sometimes with tabs or mixed tabs and spaces.
Some formatting with blank lines.
Easier to review with white-space in diff ignored. Some minor edits besides blank lines, but no change in functionality.
* fix: `setup.sh` target container under test
Some of the `setup.sh` commands did not specify the container which was problematic if another `docker-mailserver` container was running, causing test failures.
This probably doesn't help with `test/no_container.bats`, but at least prevents `test/tests.bats` failing at this point.
* consistently name functions (starting with `_`) in `start-mailserver.sh`
Most of the functions that execute the different stacks during startup
were not prefixed with `_`, but all our other functions are. This has
now been fixed.
* cleanup in `start-mailserver.sh`
I adjusted the comments for all sections in the start script so they are
properly displayed again.
Dovecot master accounts can now be configured in DMS via `setup.sh`.
A master account is useful for administration purposes, or to perform mailbox backups of every user account over IMAP.
Upstream Docs: https://doc.dovecot.org/configuration_manual/authentication/master_users/
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Removed typo "logs".
Removed warning that Docker volumes are strongly recommended, as this can lead new users in the wrong direction (bind mounts are now the default).
Altering line 8, mentioning the default of bind mounts.
* chore: Remove `DATABASE` fallback ENV
This was introduced without any mention or need, thus removing until a real use-case requires it.
* chore: Remove `USER_DATABASE` fallback ENV
Likewise, nothing requires this to be customizable.
* chore: Consistently use single quote strings
* chore: Extract letsencrypt logic into methods
This allows other scripts to share the functionality to discover the correct letsencrypt folder from the 3 possible locations (where specific order is important).
As these methods should now return a string value, the `return 1` after a panic is now dropped.
* chore: Update comments
The todo is resolved with this PR, `_setup_ssl` will be called by both cert conditional statements with purpose for each better documented to maintainers at the start of the logic block.
* refactor: Defer most logic to helper/ssl.sh
The loop is no longer required, extraction is delegated to `_setup_ssl` now.
For the change event prevention, we retrieve the relevant FQDN via the new helper method, beyond that it's just indentation diff.
`check-for-changes.sh` adjusted to allow locally scoped var declarations by wrapping a function. Presently no loop control flow is needed so this seems fine. Made it clear that `CHANGED` is local and `CHKSUM_FILE` is not.
Panic scope doesn't require `SSL_TYPE` for context, it's clearly`letsencrypt`.
* fix: Correctly match wildcard results
Now that the service configs are properly updated, when the services restart they will return a cert with the SAN `DNS:*.example.test`, which is valid for `mail.example.test`, however the test function did not properly account for this in the regexp query.
Resolved by truncating the left-most DNS label from FQDN and adding a third check to match a returned wildcard DNS result.
Extracted out the common logic to create the regexp query and renamed the methods to communicate more clearly that they check the FQDN is supported, not necessarily explicitly listed by the cert.
* tests(letsencrypt): Enable remaining tests
These will now pass. Adjusted comments accordingly.
Added an additional test on a fake FQDN that should still be valid to a wildcard cert (SNI validation in a proper setup would reject the connection afterwards).
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
This PR does two small things:
1. The log level, in case it is unset, will now be "calculated" from
`/etc/dms-settings` and not always default to `info`. This way, we
can ensure that more often than not, the log level the user chose
when starting DMS is used everywhere.
2. I noticed that the way I obtained the log level could be used to
obtain any env variable's log level. I therefore added a function to
`utils.sh` in case we use it in the future.
* first adjustments to use Fail2Ban with nftables
* replace `iptables` -> `nftables` and adjust tests
nftables lists IPs a bit differently , so the order was adjusted for the
tests to be more flexible.
* line correction in mailserver.env
* change from `.conf` -> `.local` and remove redundant config
* revert HEREDOC to `echo`
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* refactored `check-for-changes.sh`
I refactored `check-for-changes.sh` and used the new log. `_notify` can
therefore be deleted as it is used no more.
I opted to source `/etc/dms-settings` as a whole to
future-proof the script. When the DNS adjustments PRs (that do not exist
by now but will exit in the future) are done, we can then remove
`_obtain_hostname_and_domainname` because we're already writing the
variables to `/etc/dms-settings`. I left instructions in the script in
the form of TODO comments.
Because we now log the date for all messages of the changedetector, we
need to `tail` a bit more log than before.
* disabled unreliable test
The "quota exceeded" test is unreliable and failed too often lately for
my taste. Therefore, I'd like to disable it because there is no use in
having such a test.
* corrected PR id in URL
* refactored `daemon-stack.sh`
A new method was introduced to uniformaly start daemons and log output
accordingly. The methods for daemon start were renamed (plural ->
singular), therefore the adjustments in `start-mailserver.sh`.
* cleaned Fetchmail setup from `daemon-stack.sh`
Not sure why, but the Fetchmail setup was somehow happening in
`daemon-stack.sh` - this is not supposed to be the case. I relocated the
setup into `setup-stack.sh`, where it belong.
* delete old, unnecessary script in `target/bin/`
These are unused leftovers from the last commit, that relocated the
setup of Fetchmail into `setup.stack.sh`.
* corrected changedetector function name
* Apply suggestions from code review
* adjusted `debug-fetchmail` script
It is absolutely fine to source `setup-stack.sh` because sourcing the
script does not execute a single function (by desing of the script).
This way, we retain functionality.
* praise be ShellCheck
* added `log.sh` to `debug-fetchmail` as a dependency
* final cleanup
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* `update-check.sh` now uses the new log
* refactored `setup-stack.sh`
The changes are:
1. Replaced `""` wiht `''` where possible (reasoning: Bash is very
implicit and I'd like to use `''` where possible to indicate no
variables are expanded here)
2. `> /file` -> `>/file` according to our style guide
3. Some log adjustments for messages where I deemed it appropriate
4. Then, an error message from a Dovecot setup was also prevented (by
adding a check whether the directory is present before a `: >...`
command would create a file in this directory).
These are all small, miscellaneous changes that I wanted to combine into
one commit and ultimately one PR because I see no point in opening a PR
for every small change here. I hope this is fine.
* added a small `sleep` to the `_shutdown` function
This ensure the last log message is actually logged before Supervisor
logs the message that it received a SIGTERM. This makes reading the log
easier because now the causal relationship is shown (we are terminating
Supervisor, and not someone else and we're just logging it).
I forgot to replace `""` with `''` in `update-check.sh`, so I included
it here because this is the last commit before PR review.
* re-add exit on successful update (only)
* re-added date information to update-check log messages
* added `_log_with_date` function
The new function will log a message with a proper timestamp. This is all
handled in `log.sh`, we therefore not need to source other files too.
This will be used in the future by `check-for-changes.sh` as well :)
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* refactored scripts located under `target/bin/`
The scripts under `target/bin/` now use the new log and I replaced some
`""` with `''` on the way. The functionality stays the same, this mostly
style and log.
* corrected fail2ban (script and tests)
* corrected OpenDKIM log output in tests
* reverted (some) changes to `sedfile`
Moreover, a few messages for BATS were streamlined and a regression in
the linting script reverted.
* apple PR feedback
* improve log output from `fail2ban` script
The new output has a single, clear message with the '[ ERROR ] '
prefix, and then output that explains the error afterwards. This is
coherent with the logging style which should be used while providing
more information than just a single line about IPTables not functioning.
* simplified `setquota` script
* consistently named the `__usage` function
Before, scripts located under `target/bin/` were using `usage` or
`__usage`. Now, they're using `__usage` as they should.
* improved `sedfile`
With `sedfile`, we cannot use the helper functions in a nice way because
it is used early in the Dockerfile at a stage where the helper scripts
are not yet copied. The script has been adjusted to be canonical with
all the other scripts under `target/bin/`.
* fixed tests
* removed `__usage` from places where it does not belong
`__usage` is to be used on wrong user input, not on other failures as
well. This was fixed in `delquota` and `setquota`.
* apply PR review feedback
* added new `_log` function
With `_log`, the `_notify` method wa rendered obsolete. `_notify` was
not completely removed due to test failures in `check-for-changes.sh`.
The new `_log` function properly uses log levels such as `trace`,
`debug`, `info`, `warn` and `error`. It provides a cleaner solution
and renders `DMS_DEBUG` obsolete too (as only `_notify` depends on it).
* converted all helper script to new `_log` function
* converted all startup stacks to new `log` function
* `start-mailserver.sh` now uses new `_log` function
* final test and misc small script adjustments
* updated documentation
The new setup will now set env variables on one place and on one place
only. The old setup used two separate places wich is not DRY and
confusing.
Some default values changed:
1. PFLOGSUMM_TRIGGER: logrotate => none
2. REPORT_SENDER: mailserver-report@HOSTNAME => mailserver-report@DOMAIN
3. REPORT_RECIPIENT: "0" => POSTMASTER_ADDRESS
One env variable was renamed: REPORT_INTERVAL => LOGROTATE_INTERVAL
I believe these defaults to be more sensible, especially the REPORT_RECIPIENT
address. The PFLOGSUMM_TRIGGER value was changed to `none` because otherwise
people would start getting daily Postfix log summary reports automatically.
Now, this is opt-in, and reports are sent only when enabled properly.
Some of the variables changed were marked as deprecated. I removed the note,
as the variables now bear some (sane) defaults again for other variables
(i.e.) REPORT_RECIPIENT is now default for other recipient addresses.
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* docs(deps): bump mkdocs-material to v8.2.1
* feat(docs): enable mermaid integration
Configuration based on https://squidfunk.github.io/mkdocs-material/reference/diagrams/?h=mermaid#configuration
* fix: allow yaml value mapping
* chore: Adopt mkdocs-material mermaid integration support
Supported by the docs generator now, we no longer need to rely on external image generator or live editor link (both relied on large base64 encoding of mermaid markup). SVG will be rendered by docs now, although a little different style (can be fixed with custom CSS).
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* get rid of subshell + exec
The new way of executing `sha512sum` should work as well as the old way
but without the clutter and possible problems the usage of subshells +
exec incurs.
Moreover, there was a misconception about array expansion. Using `""`
around an expanding array (`${ARRAY[@]}`) is quite fine (and actually
the preffered way), not because it makes the expansion _one_ string
(this would be `${ARRAY[*]}`), but it makes sure when elements are
expanded, each element has `""` around them so to speak, i.e. there is
no re-splitting of these elements.
* removed old concerns in comments
* increase test and check for changes sleep duration
* follow up on #2383
Fixes a documentation error by which a list would not be rendered
correctly. This has been taken care of.
* update the `README.md`
I felt the need to update the README for several reasons:
1. LDAP issues that the core maintainers team cannot really resolve
2. Cleaning up the somewhat messy structure near the end
The first point goes without explanantion. The second points includes:
2.1. The tagging convention is now easier to read and understand
2.2. Some bullut points or notes have been inlined to "stick" more to
the content that it actually belongs to
2.3. The note about the "old" `setup.sh` for DMS `10.1.0` has been
removed as it is obsolete now. We encourage users to upgrade to
`10.4.0` anyways.
2.4. The markdown code highlighting is now using `CONSOLE` instead of
`BASH` because `CONSOLE` is more appropriate.
2.5. Capitalized headings
2.6. Updated the section about `./setup.sh help` to be in one place now
instead of two
2.7. DKIM key generation does now not interfere with user account
creation.
* adjusted content to PR suggestions
The prepare workflow runs in an untrusted context already and thus should not have anything worthwhile to exploit.
However care should still be taken to avoid interpolating expressions into shell scripts directly that is data a user can control the value of. Especially to avoid any maintainer referencing an existing workflow from copying a risky snippet unaware of different security contexts for workflows.
In this case, as per Github Documentation and referenced issue comment, the PR title is user controllable data, which if directly interpolated into the shell script being run (as it previously was), allows for injecting commands to execute.
The `DYNAMIC_FILES` var was quote wrapped, treating all filepaths to create checksums for as a single string that would be ignored instead of processed individually.
Removed the quotes, and changed the for loop to an array which accomplishes the same goal.
* fix: Prevent unnecessary change detection event
`acme.json` change would extract new cert files, which would then be hashed after restarting services and considered a change event, running through the logic again and restarting services once more when that was not required.
The checksum entries for those cert files are now replaced with new entries containing updated checksum hashes, after `acme.json` extraction.
* docs(deps): bump mkdocs-material to 8.0.2
* docs(deps): bump mkdocs-material to 8.0.3
* chore: add default version of docs
* feat: add version warning
* fix: remove version warning
* docs(deps): bump mkdocs-material to 8.0.5
* added code annotation feature
We can introduce new annotation with new PRs in the future. I'd advise against overhauling all code blocks with this feature in this PR - this PR should just introduce the feature.
* docs(deps): bump mkdocs-material to 8.1.0
* fix: remove unnecessary default value
re-add if version warning gets a thing in the future. See https://github.com/docker-mailserver/docker-mailserver/pull/2311#issuecomment-991805830
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Removes duplicate logic from `check-for-changes.sh` that is used/maintained elsewhere to avoid risk of problems, as this code is already starting to diverge / rot.
---
Previously the change detection support has had code added for rebuilding config upon change detection which is the same as code run during startup scripts. Unfortunately over time this has fallen out of sync. Mostly the startup scripts would get maintenance and the contributor and reviewers may not have been aware of the duplicate code handled by `check-for-changes.sh`.
That code was starting to diverge in addition to some changes in structure (_eg: relay host logic seems interleaved here vs separated out in startup scripts_). I wanted to address this before it risks becoming a much bigger headache.
Rather than bloat `helper-functions.sh` further, I've added a `helpers/` folder extracting relevant common logic between startup scripts and `changedetector`. If you want to follow that process I've kept scoped commits to make those diffs easier. Some minor changes/improvements were added but nothing significant.
---
- chore: Extract relay host logic to new `relay.sh` helper
- chore: Extract `/etc/postfix/sasl_passwd` logic to new `sasl.sh` helper
- chore: Extract `postfix-accounts.cf` logic to new `accounts.sh` helper
- chore: Extract `/etc/aliases` logic to new `aliases.sh` helper
- chore: Extract `/etc/postfix/vhost` logic to new `postfix.sh` helper
- chore: Add inline docs for Postfix configs
> These are possibly more verbose than needed and can be reduced at a later stage.
> They are helpful during this refactor process while investigating that everything is handled correctly.
`accounts.sh`:
- Add note regarding potential bug for bare domain setups with `/etc/postfix/vhost` and `mydestination` sharing same domain value.
`relay.sh`:
- Remove the tabs for a single space delimiter, revised associated comment.
- Add PR reference for original `_populate_relayhost_map` implementation which has some useful details.
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
* chore: Normalize container setup
Easier to grok what is different between configurations.
- Container name usage replaced with variable
- Volumes defined earlier and redeclared when relevant (only real difference is `VOLUME_LETSENCRYPT`)
- Contextual comment about the `acme.json` copy.
- Quoting `SSL_TYPE`, `SSL_DOMAIN` and `-h` values for syntax highlighting.
- Moved `-t` and `${NAME}` to separate line.
- Consistent indentation.
* chore: DRY test logic
Extracts out repeated test logic into methods
* chore: Scope configs to individual test cases (1/3)
- Preparation step for shifting out the container configs to their own scoped test cases. Split into multiple commits to ease reviewing by diffs for this change.
- Re-arrange the hostname and domain configs to match the expected order of the new test cases.
- Shuffle the hostname and domainname grouped tests into tests per container config scope.
- Collapse the `acme.json` test cases into single test case.
* chore: Scope configs to individual test cases (2/3)
- Shifts the hostname and domainname container configs into their respective scoped test cases.
- Moving the `acme.json` container config produces a less favorable diff, so is deferred to a follow-up commit.
- Test cases updated to refer to their `${CONTAINER_NAME}` var instead of the hard-coded string name.
* chore: Scope configs to individual test cases (3/3)
Final commit to shift out the container configs.
- Common vars are exported in `setup_file()` for the test cases to use without needing to repeat the declaration in each test case.
- `teardown_file()` shifts container removal at end of scoped test case.
* chore: Adapt to `common_container_setup` template
- `CONTAINER_NAME` becomes `TEST_NAME` (`common.bash` helper via `init_with_defaults`).
- `docker run ...` and related configuration is now outsourced to the `common.bash` helper, only extra args that the default template does not cover are defined in the test case.
- `TARGET_DOMAIN`establishes the domain folder name for `/etc/letsencrypt/live`.
- `_should*` methods no longer manage a `CONTAINER_NAME` arg, instead using the `TEST_NAME` global that should be valid as test is run as a sequence of test cases.
- `PRIVATE_CONFIG` and the `private_config_path ...` are now using the global `TEST_TMP_CONFIG` initialized at the start of each test case, slightly different as not locally defined/scoped like `PRIVATE_CONFIG` would be within the test case, hence the explicit choice of a different name for context.
* chore: Minor tweaks
- Test case comment descriptions.
- DRY: `docker rm -f` lines moved to `teardown()`
- Use `wait_for_service` helper instead of checking the `changedetector` script itself is running.
- There is a startup delay before the `changedetector` begins monitoring, wait until it ready event is logged.
- Added a helper to query logs for a service (useful later).
- `/bin/sh` commands reduced to `sh`.
- Change the config check to match and compare output, not number of lines returned. Provides better failure output by bats to debug against.
* chore: Add more test functions for `acme.json`
This just extracts out existing logic from the test case to functions to make the test case itself more readable/terse.
* chore: Housekeeping
No changes, just moving logic around and grouping into inline functions, with some added comments.
* chore: Switch to `example.test` certs
This also required copying the source files to match the expected letsencrypt file structure expected in the test/container usage.
* chore: Delete `test/config/letsencrypt/`
No longer necessary, using the `example.test/` certs instead.
These letsencrypt certs weren't for the domains they were used for, and of course long expired.
* chore: Housekeeping
Add more maintainer comments, rename some functions.
* tests: Expand `acme.json` extraction coverage
Finally able to add more test coverage! :)
- Two new methods to validate expected success/failure of extraction for a given FQDN.
- Added an RSA test prior to the wildcard to test a renewal simulation (just with different cert type).
- Added extra method to make sure we're detecting multiple successful change events, not just a previous logged success (false positive).
* tests: Refactor the negotiate_tls functionality
Covers all ports (except POP) and correctly tests against expected verification status with new `example.test` certs.
The `FQDN` var will be put to use in a follow-up commit.
* tests: Verify the certs contain the expected FQDNs
* chore: Extract TLS test methods into a separate helper script
Can be useful for other TLS tests to utilize.
* chore: Housekeeping
* chore: Fix test typo
There was a mismatch between the output and expected output between these two files "find key for" and "find key & cert for". Changed to "find key and/or cert for" to make the warning more clear that it's issued for either or both failure conditions.
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Better logical flow, handling and inline documentation.
Despite the verbosity, it's better to make this visible here for maintenance and debugging purposes than trying to dig through issue/PR or commit history for it.
* fix: Panic when HOSTNAME is misconfigured
* chore: Add more comment docs for maintainers
* tests(fix): Use `--domainname` not ENV `DOMAINNAME`
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Previously we only monitored for `$HOSTNAME` in `/etc/letsencrypt/live` and only for hard-coded `.pem` filenames.
This ensures we check the locations of other locations that may not match `$HOSTNAME`, which we also support. Ideally in future at least the directory to look in would be better known in advance..
These are improvements for better supporting the requirements of other tests.
- Opted for passing an array reference instead of an ENV file. This seems to be a better approach and supports more than just ENV changes.
- Likewise, shifted to a `create` + `start` approach, instead of `docker run` for added flexibility.
- Using `TEST_TMP_CONFIG` instead of `PRIVATE_CONFIG` to make the difference in usage with config volume in tests more clear.
- Changed the config volume from read-only volume mount to be read-write instead, which seems required for other tests.
- Added notes about logged failures from a read-only config volume during container startup.
- Added `TEST_CA_CERT` as a default CA cert path for the test files volume. This can be used by default by openssl methods.
Split into scoped commits with messages if further details are needed, view those via the associated PR :)
**Commit Summary:**
**`check-for-changes.sh`**
- Prevent `SSL_DOMAIN` silently skipping when value has wildcard prefix `*.` (_at least this was known as a bugfix when originally committed in linked PR_).
- Improved inlined docs for maintainers.
- Additional logging for debugging.
**`helper-functions.sh:_extract_certs_from_acme`**:
- Fail if the input arg (_`$CERT_DOMAIN`, aka the FQDN_) provided for extraction is empty.
- Use `$CERT_DOMAIN` in place of `$HOSTNAME` and `$1` for a consistent value (_previously could mismatch, eg with `SSL_DOMAIN` defined_).
- The conditional is now only for handling extraction failure (_key or cert value is missing from extraction_).
- Log an actual warning or success (debug) based on outcome.
- Don't use `SSL_DOMAIN` with wildcard value for the `mkdir` letsencrypt directory name (_wildcard prefix `*.` is first stripped instead_).
**`acme_extract`** (_new python utility for `acme.json` handling_):
- Extracted out into a python script that can be treated as a utility in the `$PATH` like other helper scripts. It can now be used and optionally tested directly instead of via `helper-functions.sh`.
-Made compatible with Python 3, as Python 2 is EOL and no longer in newer versions of Debian.
These files will replace the existing `test/config/letsencrypt` content which has some random provisioned FQDN for letsencrypt that doesn't match the FQDN tested, `acme.json` files with FQDNs that don't match those certs FQDNs and changes to certs that won't expire until 2031. `test/config/letsencrypt` will be removed with the associated test update PR.
The changes amount to:
- Re-configuring the FQDN values that some certs were created for (_needed for flexibility in testing_).
- Adding an `*.example.test` wildcard (_both RSA and ECDSA_).
- Adding `acme.json` encoded versions (_traefik extraction support will use these instead_).
- Updated / new internal docs for maintainers of this content.
For more detailed information on those changes, please see the associated commit messages via the PR.
Mostly cleans up the code and documents it better, although there are some minor fixes for handling `SSL_DOMAIN` ENV and additional logging added for spotting issues related to it in future when troubleshooting.
Commits are scoped with context messages for easing review if necessary. Overview of changes:
Traefik specific:
- Logic extracted out into it's own function.
- Conditional reworked to assist with debugging.
- `SSL_DOMAIN` must not be empty when attempting to extract.
- Added additional notes.
`SSL_TYPE=letsencrypt` case:
- Revised top note block.
- Correct handling for `SSL_DOMAIN`.
- Removed some unnecessary nesting.
- Less repetitive error message for `LETSENCRYPT_DOMAIN`.
- Added use of panics where appropriate (kept `return 1` so failures still exit functionality early).
- Improved inline docs.
Dovecot quota support would log auth failures when Postfix validated incoming mail to accept/reject and the `check_policy_service` for `quota-status` was queried with a recipient that was an account alias.
When Dovecot is not aware of the user account, it will not be able to check a quota and inform Postfix that everything is fine, Postfix will accept the mail and send it to Dovecot, where if the quota is exceeded will result in a bounce back to the sender. This is considered "backscatter" and can be abused by spammers forging the sender address which can get your server blacklisted.
The solution is to either disable quota support `ENABLE_QUOTAS=0`, or as a workaround, add dummy accounts to Dovecot userdb for aliases in `postfix-virtual.cf` (not `postfix-aliases.cf`), these dummy accounts will map to the real user account mailbox (real users are defined in `postfix-accounts.cf`).
The workaround is naive, in that we only check for basic 1-to-1 alias mapping to real accounts. This will still be an issue for aliases that map to another alias or multiple addresses (real or alias). Unfortunately Postfix will not expand aliases until accepting mail where this would be too late.
A better solution is to proxy the `check_policy_service` from Dovecot `quota-status` that Postfix queries in `main.cf:smtpd_recipient_restrictions`, however this requires a fair amount more of additional work and still requires an implementation to recursively query aliases for nested or multiple address mappings, which can then be forwarded to the `quota-status` service configured by Dovecot in `/etc/dovecot/conf.d/90-quota.conf`.
LDAP users are unaffected as quota support is not supported/implemented with `docker-mailserver` at this time, it is always considered disabled when using LDAP.
---
Additionally Dovecot configuration for `passdb` has been fixed to use the correct password hash scheme of `SHA512-CRYPT`.
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
This was all introduced by the original project author early on, no explanation for it.
- None of the paths they use as `argv` values exist.
- `uucp` doesn't seem relevant to the project. No justification for it, no issues or PRs in project history or codebase.
See associated PR for further details and linked resources.
Using `set -ex` will exit the script as soon as a non-zero exit code is returned, such as when the docker image fails building the docs due to `build --strict` catching broken links. This also removes the need for `|| exit` when changing directory.
This seems fine for a small script, but AFAIK an alternative fix is just adding `|| exit` to the end of the `docker run` command too? There appears to be advice [against adopting `-e` carelessly](http://mywiki.wooledge.org/BashFAQ/105), while others [encourage `-e`](http://redsymbol.net/articles/unofficial-bash-strict-mode/). I know that several maintainers here have preference towards `set -e` so I've kept the original PR solution.
Additionally:
- `-x` is used to improve command visibility when reviewing the workflow log output.
- `--name` isn't necessary, but was part of the original PR.
- I've chosen not to include `-o pipefail`, only because no pipes are used in this script.
* docs(fix): Fix broken links
* ci(docs): Added inline docs
Extra documentation context for maintainers to quickly grok what's going on.
* chore(docs): Minor typo fix by wernerfred
Added from their related PR by request.
* docs(ssl): Adjust heading levels for provisioning sections
- Group provisioning sections under one heading level.
- Use `attr_list` syntax for headings to make the ToC sidebar entry less verbose.
* docs(ssl): Minor fixes
Typos, formatting.
* docs(ssl): Rephrase Traefik wildcard support
Split the line out into multiple with better phrasing.
* docs(ssl): Add FQDN section
We briefly mention the same info twice on the docs page, but as it applies to all provisioners in general, it's been given it's own detailed section with examples.
Single section to inform users about an FQDN, how it's configured and understood by `docker-mailserver` for both Docker CLI and `docker-compose.yml` variations.
Adds note about wildcard support and bare domains to clear up any confusion configuring FQDN for these two.
Additional note about Certbot using symlinks for it's cert storage.
* chore: Add FQDN comment for `docker-compose.yml` example config
* ci: Fix lint check status update
The lint workflow is not important for this PR, but a fixed requirement to pass for merging.
As this workflow is triggered by `schedule` or `workflow_dispatch`, it will not trigger other events such as `pull_request` for other workflows to respond to.
Since the linting workflow is not important for this type of PR, we can pretend it was "skipped" and set the check status to "success". This is simpler than running the actual Lint workflow redundantly.
* ci: Remove workflow_run approach
This didn't work out, reverting.
This should resolve the issue of the lint workflow not being triggered by PRs opened via another workflow (`contributors.yml`).
This workflow will be triggered after the dependent workflow completes (regardless of status).
This was a community contributed guide from the Github Wiki prior to docs migration. I've rewritten it by restructuring the content, introducing numerical steps and revising some of the content, while removing third-party software that was unnecessary (the original authors and content related to their use-case, Moodle).
See the PR for further details and reference links regarding the original documents history.
The PR provides improved diff via separate commits scoping changes at the correct change bounds, unfortunately the full diff doesn't align to those boundaries well making it more difficult to review vs individual commits.
Below commit messages are roughly equivalent to what is listed on the PR. The PR provides additional linked resources for reference to support commit message statements.
---
* docs: Add CT log warning
- Added a warning to make users aware that using a public CA like _Let's Encrypt_ will publicly log information that may be somewhat sensitive, or undesirable to have historic records made public which cannot be redacted.
* docs: Revise the manual `certbot` guide
- The `letsencrypt` repo that was linked early in this guide now redirects to the [Certbot repo](https://github.com/certbot/certbot).
- More explicit volume mount instruction for CertBot; the local location was a tad vague.
- Better clarified `/etc/letsencrypt/live` contents structure, as well as FQDN info. Removed the misleading `fqdn:` from `docker-compose.yml` example snippet.
* docs: Revise certbot with Docker guide
- General rewrite of the Docker Certbot section with additional tips (_renewals with automation, and using a alternative CA_).
- Generalized tone and paths in content.
- Update volume mount paths to be consistent with recent normalization effort.
- Moved some instructions into inline-comments for script examples instead.
* docs: Revise Docker with `nginx-proxy` and `acme-companion`
- Break apart into individual steps, indenting content into the step as appropriate.
- Use normalized volume paths (`docker-data/<service>/` prefix).
- `letsencrypt-nginx-proxy-companion` has _changed project name to `acme-companion`_, and _transferred to new maintainers and the `nginx-proxy` organization_. This also affects the DockerHub image references.
- `acme-companion` has _switched from using `simp_le` to `acme.sh`_ for provisioning certificates. This requires mounting an additional volume for persisting provisioner state.
- The dummy container (_webmail_) is no longer `library/nginx`, just [`nginx`](https://hub.docker.com/_/nginx). This container also doesn't appear to be required. I've verified that the ENV can be given to the `mailserver` service container directly. Retained for now.
* docs: Revise Docker Compose with `nginx-proxy` and `acme-companion`
Heavy rewrite of this section. Like the previous commit mentions, this content was outdated. It has been simplified with improved documentation and reference links.
It also looks like there was a mistake in the existing config example as it uses the regular `nginx` image instead of `nginx-proxy`.
- The bulk of the `mailserver` service has been removed, users are advised to have an existing `docker-compose.yml` config for `docker-mailserver` and update only what is relevant to integrate with the cert provisioner.
- `DEBUG` is _false_ by default.
- The `networks:` portion of the example appears to be taken from upstream, _which that has since dropped it_. While we could continue to document this, I consider it more of an advanced config detail that we don't need to touch on in our docs.
- The `htpasswd` volume is unnecessary, only relevant if using _"Basic Authentication"_ to protect access to web service endpoints. `conf.d/` is also not required by default, it can be useful for the `standalone` mode (_documented as a `tip`_). Remaining volumes have inline-comments to document their purpose.
- `volumes_from:` is _not supported in v3 Compose format_, _only v2_ and the Docker CLI. I did not want to advise v2, so I've duplicated the volumes between the two containers instead. Internally `acme-companion` would rely on `volumes_from:` to identify the `nginx-proxy` container, it _provides alternative discovery methods_, the label is outdated and refers the legacy label (_their script logic is the same_); using the ENV `NGINX_PROXY_CONTAINER` seemed most appropriate and has been added.
- Upstream `acme-companion` docs only cover support for v2 Compose format. _There is a note regarding `nginx-proxy`_ having _volumes configured in it's Dockerfile_. Providing a volume for `/etc/nginx/dhparam` is required to avoid creating anonymous volumes each run of `nginx-proxy`. I've used a named data volume here to make it stick out more, it's not desirable and upstream should fix this, then we can drop it.
- I've also opted to only demonstrate the _Two Container (Basic) setup_ that upstream documents. Previously our docs have been showing _`docker-gen` with the Three Container (Advanced) setup_, which allows for not having the Docker API socket attached as a volume to a container exposed to the web. This reduces the security a bit, and I have not mentioned that on our docs. I could caution the reader with a link to upstream about the risk, but I don't think we should maintain the `docker-gen` setup.
* docs(fix): Update anchor links
These mismatched the current section headers they were meant to link to.
* function _defunc removed
* _shutdown is better than just notify in that cases
* PANIC_TYPE 'fail-init' introduced
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
"Brief" summary/overview of changes. See the PR discussion or individual commits from the PR for more details.
---
Only applies to the `docs/content/**` content (_and `setup` command_). `target/` and `test/` can be normalized at a later date.
* Normalize to `example.com`
- Domains normalized to `example.com`: `mywebserver.com`, `myserver.tld`, `domain.com`, `domain.tld`, `mydomain.net`, `my-domain.tld`, `my-domain.com`, `example.org`, `whoami.com`.
- Alternative domains normalized to `not-example.com`: `otherdomain.com`, `otherdomain.tld`, `domain2.tld`, `mybackupmx.com`, `whoareyou.org`.
- Email addresses normalized to `admin@example.com` (in `ssl.md`): `foo@bar.com`, `yourcurrentemail@gmail.com`, `email@email.com`, `admin@domain.tld`.
- Email addresses normalized to `external-account@gmail.com`: `bill@gates321boom.com`, `external@gmail.com`, `myemail@gmail.com`, `real-email-address@external-domain.com`.
- **`faq.md`:** A FAQ entry title with `sample.domain.com` changed to `subdomain.example.com`.
- **`mail-fetchmail.md`:** Config examples with FQDNs for `imap`/`pop3` used `example.com` domain for a third-party, changed to `gmail.com` as more familiar third-party/external MTA.
* Normalize config volume path
- Normalizing local config path references to `./docker-data/dms/config/`: `./config/`, `config/`, \``config`\`, `/etc/` (_volume mount src path prefix_).
- Normalize DMS volume paths to `docker-data/dms/mail-{data,state,log}`: `./mail`, `./mail-state` `./data/mail`, `./data/state`, `./data/logs`, `./data/maildata`, `./data/mailstate`, `./data/maillogs`, (_dropped/converted data volumes: `maildata`, `mailstate`_).
- Other docker images also adopt the `docker-data/{service name}/` prefix.
* `ssl.md` - Use `dms/custom-certs` where appropriate.
* Apply normalizations to README and example `docker-compose.yml`
---
Common terms, sometimes interchangeably used or now invalid depending on context: `mail`, `mail container`, `mail server`, `mail-server`, `mailserver`,`docker-mailserver`, `Docker Mailserver`.
Rough transformations applied to most matches (_conditionally, depending on context_):
- 'Docker Mailserver' => '`docker-mailserver`'
- 'mail container' => '`docker-mailserver`' (_optionally retaining ' container'_)
- 'mail server' => 'mail-server' / '`docker-mailserver`'
- 'mail-server' => '`docker-mailserver`'
- 'mailserver' => 'mail-server' / '`docker-mailserver`'
Additionally I checked `docker run` (_plus `exec`, `logs`, etc, sub-commands_) and `docker-compose` commands. Often finding usage of `mail` instead of the expected `mailserver`
Additionally changes `mailserver` hostname in k8s to `mail` to align with other non-k8s examples.
---
* drive-by revisions
Mostly minor revisions or improvements to docs that aren't related to normalization effort.
* fix: Spam bounced test copy/paste typo
* tests(docs): Expand inline documentation
Should assist maintainers like myself that are not yet familiar with this functionality, saving some time :)
* Refactor bounced test + Introduce initial container template
DRY'd up the test and extracted a common init pattern for other tests to adopt in future.
The test does not need to run distinct containers at once, so a common name is fine, although the `init_with_defaults()` method could be given an arg to add a suffix: `init_with_defaults "_${BATS_TEST_NUMBER}"` which could be called in `setup()` for tests that can benefit from being run in parallel.
Often it seems the containers only need the bare minimum config such as accounts provided to actually make the container happy to perform a test, so sharing a `:ro` config mount is fine, or in future this could be better addressed.
---
The test would fail if the test cases requiring smtp access ran before postfix was ready (_only a few seconds after setup scripts announce being done_). Added the wait condition for smtp, took a while to track that failure down.
Initial pass for achieving more consistency with docker-compose related configs.
* Set DMS_DEBUG to 0
* align with default docker-compose.yml
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Recent `sedfile` addition moved all scripts section earlier into the Dockerfile so that `sedfile` could be used within the Dockerfile.
However whenever a change is made to scripts which is most of the time for this project, building the Docker image for tests results in all layers after the scripts being invalidated, notably ClamAV, wasting storage of previous instances and increasing build time unnecessarily.
This isn't as noticeable of an issue via the CI as we don't leverage any caching at present there, but for iterating on a local branch and testing, it can be quite the drawback.
- `sedfile` is handled early in the Dockerfile still, while the scripts have been moved as far down as it made sense to.
- `chmod` was split out into it's own RUN command as again it's unnecessary for the rest of it's prior RUN command group to be invalidated.
* chore(refactor): DRY up the `_setup_ssl` method
- `/etc/postfix/ssl` was a bit misleading in usage here. As a maintainer (of my own contribution!) I was confused why only `/etc/postfix/ssl` was referenced and not `/etc/dovecot/ssl`.
- The postfix specific path is unnecessary, dovecot was referencing it via it's config, the same can be done from postfix to a generic DMS specific config location instead.
- This location is defined and created early as `/etc/dms/tls` (with var `DMS_TLS_PATH`). All usage of `/etc/postfix/ssl` has been replaced, making it easier to grok. Several `mkdir` commands related to this have been dropped as a result.
- Likewise, a related `TMP_DMS_TLS_PATH` var provides a reference to the config volume path `/tmp/docker-mailserver` which is used for conditions on presently hard-coded paths.
- Other values that benefit from being DRY have been lifted up into vars. Definitely easier to follow now and makes some further opportunities clearer to tackle in a future refactor.
- `chmod` has been updated where appropriate. Public key/cert is acceptable to have as readable by non-root users (644). The custom type with single fullchain file was not root accessible only, but should as it contains a private key.
- That said, the security benefit can be a bit moot due to source files that were copied remain present, the user would be responsible to ensure similar permissions on their source files.
- I've not touched LetsEncrypt section as I don't have time to investigate into that yet (not familiar with that portion).
---
* chore: Remove mkcert logic and dovecot cert
- No longer serving a purpose.
- Our own TLS startup script handles a variety of cert scenarios, while the dropped code was always generating a self-signed cert and persisting an unused cert regardless with `ONE_DIR=1`.
- To avoid similar issues that DH params had with doveadm validating filepath values in the SSL config, the default dummy values match postfix pointing to "snakeoil" cert. That serves the same purpose as mkcert was covering in the image.
- Bonus, no more hassle with differing mkcert target paths for users replacing our supplied Dovecot with the latest community edition.
---
* Error handling for SSL_TYPE
- Added a panic utility to exit early when SSL_TYPE conditions are misconfigured.
- Some info text had order of key/cert occurrence swapped to be consistent with key then cert.
- Some existing comments moved and rephrased.
- Additional comments added.
- `-f` test for cert files instead of `-e` (true also for directories/devices/symlinks).
- _notify messages lifted out of conditionals so that they always output when the case is hit.
- ~~Empty SSL_TYPE collapsed into catch all panic, while it's contents is now mapped to a new 'disabled' value.~~
---
* Use sedfile + improve sed expressions + update case style
- Uses sedfile when appropriate (file change intentional, not optional match/check).
- sed expressions modified to be DRY and reduce escaping via `-r` flag (acceptable if actual text content contains no `?`,`+`,`()` or `{}` characters, [otherwise they must be escaped](https://www.gnu.org/software/sed/manual/html_node/Extended-regexps.html)).
- sed captures anything matched between the parenthesis`()` and inserts it via `\1` as part of the replacement.
- case statements adopt the `(` prefix, adopting recent shell style for consistency.
---
* Refactor SSL_TYPE=disabled
- Postfix is also disabled now.
- Included heavy inline documentation reference for maintainers.
- Dropped an obsolete postfix config option 'use_tls' on the relayhost function, it was replaced by 'security_level'.
---
* I'm a friggin' sed wizard now
- The `modern` TLS_LEVEL is the default values for the configs they modify. As such, `sedfile` outputs an "Error" which isn't an actual concern, back to regular `sed`.
- I realized that multiple edits for the same file can all be done at once via `-e` (assuming other sed options are the same for each operation), and that `g` suffix is global scope for single line match, not whole file (default as sed iterates through individual lines).
- Some postfix replacements have `smtp` and `smtpd` lines, collapsed into a single `smtpd?` instead now that I know sed better.
---
* tests(fix): Tests that require SSL/TLS to pass
- SSL_TYPE=snakeoil added as temporary workaround.
- nmap tests are being dropped. These were added about 4-5 years ago, I have since made these redundant with the `testssl.sh` tests.
- Additionally the `--link` option is deprecated and IIRC these grades were a bit misleading when I initially used nmap in my own TLS cipher suite update PRs in the past.
- The removed SSL test is already handled in mail_ssl_manual.bats
ldap test:
- Replace `--link` alias option with `--network` and alias assignment.
- Parameterized some values and added the `SSL_TYPE` to resolve the starttls test failure.
privacy test:
- Also needed `SSL_TYPE` to pass the starttls test.
`tests.bats` had another starttls test for imap:
- Workaround for now is to give the main test container `SSL_TYPE=snakeoil`.
---
* Remove the expired lets-encrypt cert
This expired in March 2021. It was originally required when first added back in 2016 as LetsEncrypt was fairly new and not as broadly accepted into OS trust stores.
No longer the case today.
---
* chore: Housekeeping
Not required for this PR branch, little bit of tidying up while working on these two test files.
- privacy test copied over content when extracted from `tests.bats` that isn't relevant.
- ldap test was not as easy to identify the source of DOVECOT_TLS. Added comment to make the prefix connection to `configomat.sh` and `.ext` files more easier to find.
- Additionally converted the two localhost FQDN to vars.
---
* Default SSL_TYPE becomes `''` (aka equivalent to desired `disabled` case)
- This is to prevent other tests from failing by hitting the panic catchall case.
- More ideal would be adjusting tests to default to `disabled`, rather than treating `disabled` as an empty / unset SSL_TYPE value.
---
* Add inline documentation for `dms_panic`
- This could later be better formatted and placed into contributor docs.
Panic with kill (shutdown) not exit (errex):
- `kill 1` from `_shutdown` will send SIGTERM signal to PID 1 (init process).
- `exit 1` within the `start-mailserver.sh` init scripts context, will just exit the initialization script leaving the container running when it shouldn't.
The two previous `_shutdown` methods can benefit from using `dms_panic` wrapper instead to standardize on panic messages.
Although these two config lines have not changed since `debian:buster-slim` image, Dovecot seems to now be affected by it which results in rejecting cipher suites below TLS v1.2.
To continue supporting the `intermediate` TLS_LEVEL, we now need to relax the global config. Dovecot could alternatively be given a modified openssl config to only affect it's interaction with openssl.
Postfix is unaffected and continues to support TLS <1.2 cipher suites when configured to.
This feature was originally introduced by the PR: https://github.com/docker-mailserver/docker-mailserver/pull/1463
- Assign default DH params to use via Dockerfile build instead of copy and update at runtime.
- Parameterized service names and paths.
- Refactor postfix and dovecot dh methods to wrap shared dh logic
- I don't see any value in checking the alternative service for dh params file to copy over, so that's now dropped too.
- Another conditional check is dropped and the default fallback message for existing DH params file is no longer relevant.
- Improved the remaining `_notify` messages. Collapsing the warning into a single logged message also seemed relevant.
- There is no apparent need for special handling with `ONE_DIR=1`. Dropped it.
- Refactor DH params tests
- Combine custom and default DH param tests into single test file
- docs: Add instructions to use custom DH params
There is no official documented support for custom DH parameters. As no guarantee is provided, this is considered an internal change, not a breaking one.
* docs: Improve FAQ entry for `mail-state` folder
- Links to relevant script logic.
- Better list of services data moved to `mail-state`.
* Update docs/content/faq.md
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* docs(fix): ONE_DIR env default is now `1`
This was missed during the `ONE_DIR` default change in https://github.com/docker-mailserver/docker-mailserver/pull/2148
* fix relative filepath
* fix: use new URI anchor
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* changed the locking function to better support multiple servers running at once and sharing the same config
* helper function testing now runs inside of container
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Decoupling setup process from `setup.sh` script by introducing a setup script _inside_ the container that coordinates the setup process.
**This is not a breaking change**. This way, we do not have to keep track of versions of `setup.sh`.
This change brings the additional benefit for Kubernetes users to be able to make use of `setup` now, without the need for `setup.sh`.
---
* move setup process into container; setup.sh versioning not needed anymore
* add tilde functionality to docs
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Centralize the collection of the HOSTNAME and DOMAINAME so that it's predictable and uniform across the various scripts (using the helper). Ensure it supports the various configurations users can have (both subdomain and without subdomain, override and no override).
---
* using _obtain_hostname_and_domainname helper + covers when not a subdomain
doc: OVERRIDE_HOSTNAME takes priority
* added tests for non-subdomain hostname + further improvements
* moved SRS DOMAINANME tests into hostname test file + Allowing DOMAINNAME ENV to override what would be automatically set
---
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* docker_container first, then fall back to docker_image
+ test changes to support
+ test change to wait for smtp port to fix flakey tests since https://github.com/docker-mailserver/docker-mailserver/pull/2104
* quick fix
* Update setup.sh
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
Co-authored-by: Casper <casperklein@users.noreply.github.com>
Adds a new blog post that covers setting up docker-mailserver on a VPS, including but not limited to
* Considerations when selecting a VPS
* Initial configuration of docker-mailserver
* DNS setup and verification of settings
* Multiple domains
Co-authored-by: Frederic Werner <20406381+wernerfred@users.noreply.github.com>
* Add logwatch maillog.conf file to support /var/log/mail/
* Simpliied after reviewing logwatch doc
Co-authored-by: Georg Lauterbach <44545919+georglauterbach@users.noreply.github.com>
* add dovecot-fts-xapian
update Docker to build from debian bullseye slim, as it contains
packages for fts-xapian.
update Docker to install dovecot-fts-xapian.
update docs with instructions on how to enable fts-xapian or fts-solr
and what considerations to take into when deciding.
* address review feedback
* update backport method to previously proposed approach (which was lost in a forced push)
Debian opendmarc package has a dependency on dbconfig-mysql, which it
will pull, together with its dependencies, during image build.
Explicitly listing an alternative dbconfig-no-thanks package prevents
installation of unnecessary packages and reduces the image size.
* docs: Add example for customizing IMAP folders (mailboxes)
* chore: Update `15-mailboxes.conf` to sync with upstream
This config has not been updated since 2016 (ignoring the Junk autosubscribe addition).
Synced to upstream equivalent at https://github.com/dovecot/core/blob/master/doc/example-config/conf.d/15-mailboxes.conf
Retains the `Archive` example definition from this PR and prior `auto = subscribe` additions.
---
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* docs: SSL - Deprecate internal self-signed cert tool
We no longer support this method with `setup.sh` from v10 onwards, `SSL_TYPE=self-signed` remains supported however. Advice has been revised for users to provide their own self-signed cert or use an external tool with an example provided.
* chore: typo fix
* chore: fix docker cmd
* chore: fix link syntax
* fix: make sure the SASLAUTHD_LDAP_HOST/PROTO logic makes sense and use LDAP_SERVER_HOST as a fallback (#1983)
* chore(docs): document changes to LDAP/SASLAUTHD as of #1983
* fix!: apply default value modifications suggested in #1983https://github.com/docker-mailserver/docker-mailserver/issues/1983#issuecomment-844848224
* chore(test): Test SASLAUTHD_LDAP_SERVER with protocol and ..._SSL=0, as well as with default bind credentials
Note that there are currently no regression tests for this as there's only one setup_file, so that would require big changes to the testing methodology.
* refactor!: completely remove SASLAUTHD_LDAP_SSL and SASLAUTHD_LDAP_PROTO
Co-authored-by: Georg Lauterbach <44545919+aendeavor@users.noreply.github.com>
Co-authored-by: Frederic Werner <20406381+wernerfred@users.noreply.github.com>
* docs(ldap): Make DOVECOT_PASS_FILTER clearer and add a small DOVECOT_AUTH_BIND section
* docs(ldap): Remove superfluous environment variables as of #1989
* docs(ldap): Document defaults for DOVECOT_*_ATTRS/FILTER
* docs(ldap): Add documentation for LDAP with TLS and StartTLS
Co-authored-by: Frederic Werner <20406381+wernerfred@users.noreply.github.com>
Co-authored-by: Brennan Kinney <5098581+polarathene@users.noreply.github.com>
* docs(ci): Support deploy previews for documentation
Each PR that contributes to docs will generate a unique (to that PR) URL to preview the PR live for review.
* docs(ci): Split workflow
To support previews from non-collaborators PR contributions, we cannot rely on secrets access from workflows triggered by the `pull_request` event.
To do so securely, according to official advice from Github, we must run the third-party contribution in the restricted `pull_request` context, and then use a 2nd workflow to deploy the build (which requires secrets access).
* docs(ci): Rename doc workflows + add commit status
Better naming convention for documentation workflows.
Split workflow only indicated status on PR of the 1st stage (building the preview to deploy), not the deployment progress/result. This needs to be managed more directly until the action better supports split-workflow scenario.
* docs(ci): Add concurrency limit to preview deploy workflow
This would be more ideal on the 2nd phase workflow (`workflow_run`), however keeping it simple for now.
Limits the concurrency of the initial pull request workflow for documentation contributions that have PRs with multiple event triggers in a small time span (before the workflow triggered would complete). The main benefit is to avoid redundant deploys if the initial workflow has been triggered again to build the PR once more. It only will work against concurrent workflows for that PR in the 1st stage, if an existing `workflow_run` (2nd stage) is active for that PR it will not be cancelled.
* docs(ci): Add sponsor branding for deploy preview service
A requirement from Netlify for the [sponsored OSS organization plan](https://www.netlify.com/legal/open-source-policy).
* docs(ci): Use a shared build script
Production and Deploy Preview builds are now maintained via the same shell command, so version updates of docker image is in one place.
Additionally deletes unnecessary build output which upstream provides no support to exclude.
* docs: Add a custom 404 page
This is used by the preview deploys on Netlify. Production deploys on Github Pages require a top-level 404 page manually deployed (since all are deployed to a version subpath).
This 404 page was custom built and optimized by me. This is the final minified output, separate source to build is available if needed.
---
Likewise the `favicon.ico` is a fallback for browsers that implicitly check the domain root for this file if the SVG isn't supported/preferred. Browsers check for this file without it being present in the HTML head meta elements.
On Github Pages the `favicon.ico` isn't likely to be picked up by even top-level as typical deployment has the project name as a subpath. The docs however reference a PNG favicon which should be widely supported.
The `favicon.ico` was generated by RealFaviconGenerator online tool with SVG source input. It contains 16px, 32px and 48px sizes. Quality is better than the `favicon.io` generator.
* chore: Optimized logo
SVG source cleaned up and optimized with SVGO 2.3.
Minified versions (`.min.svg` extension) remove unnecessary data and white-space to reduce size further for production use. This extension better differentiates by filename that it's different from the `src` version.
* mail binary
* initial work
* make env vars available
* typo
* some fixes
* make script ugly, to satisfy linter..
* mailserver.env updated
* Version to welcome message added
* remove VERSION file references
* VERSION --> DMS_VERSION
* fetch remote version
* variable usage
* Quoting added
* edge test & docu
* dash removed
* subject changed
* re-add VERSION
* VERSION added
* new file: VERSION
* rewrite
* unnecessary additions from fail2ban PR removed
* UPDATE_CHECK_INTERVAL added
* syntax check & _log function
* comment added
* final commit
* chore(deps): bump docker/login-action from 1 to 1.9.0
Bumps [docker/login-action](https://github.com/docker/login-action) from 1 to 1.9.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v1...v1.9.0)
Signed-off-by: dependabot[bot] <support@github.com>
* chore(security): switch from PAT to GITHUB_TOKEN
* chore(security): switch from PAT to GITHUB_TOKEN
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Frederic Werner <20406381+wernerfred@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+aendeavor@users.noreply.github.com>
* Use dovecot's LDAP uris option instead of hosts (fixes#1510)
* Clean up variables & environment documentation for #1901
Co-authored-by: Frederic Werner <20406381+wernerfred@users.noreply.github.com>
Co-authored-by: Georg Lauterbach <44545919+aendeavor@users.noreply.github.com>
* mail_crypt plugin + quick improvement to dovecot override defaults doc
* quick change for <your-container-name> to use mailserver
Co-authored-by: Georg Lauterbach <44545919+aendeavor@users.noreply.github.com>
description:Submit a bug report to help us improve
title: 'bug report:'
labels:
- kind/bug/report
- meta/needs triage
body:
- type:checkboxes
id:preliminary-checks
attributes:
label:📝 Preliminary Checks
description:|
By submitting this issue, you agree to our [Code of Conduct](https://github.com/docker-mailserver/docker-mailserver/blob/master/CODE_OF_CONDUCT.md).
options:
- label:I tried searching for an existing issue and followed the [debugging docs](https://docker-mailserver.github.io/docker-mailserver/latest/config/debugging/) advice, but still need assistance.
required:true
- type:textarea
id:what-happened
attributes:
label:👀 What Happened?
description:How did this differ from your expectations?
placeholder:Although `LOG_LEVEL=debug` is set, the logs are missing debug output.
validations:
required:true
- type:textarea
id:steps-to-reproduce
attributes:
label:👟 Reproduction Steps
description:|
How did you trigger this bug? Please walk us through it step by step.
Please use [fenced code blocks](https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks) when pasting lots of text!
placeholder:The easier it is for us to reproduce your issue, the sooner we can help resolve it 😉
- type:input
id:mailserver-version
attributes:
label:🐋 DMS Version
description:Onwhich version (image tag) did you encounter this bug?
<!-- Select one, remove the other and do not shorten the sentence -->
Yes, because I know the probability of someone else doing it is low and I can learn from it.
No, and I understand that it is highly likely no one will implement it. Furthermore, I understand that this issue will likely become stale and will be closed.
Markdown formatting can be used in almost all text fields. The description will tell you if this is not the case for a specific field.
Be as precise as possible, and if in doubt, it's best to add more information that too few.
---
- type:textarea
id:context
attributes:
label:Context
description:Tell us how your request is related to DMS, one of its components or another issue / PR. Also **link all conected issues and PRs here**!
validations:
required:true
- type:textarea
id:description
attributes:
label:Description
description:Describe the solution you would like to have implemented. Be as precise as possible!
validations:
required:true
- type:textarea
id:alternatives
attributes:
label:Alternatives
description:Which alternatives have you considered?
validations:
required:true
- type:textarea
id:applicable-users
attributes:
label:Applicable Users
description:Who will that feature be useful to?
validations:
required:true
- type:dropdown
id:implementer
attributes:
label:Are you going to implement it?
options:
- Yes,because I know the probability of someone else doing it is low and I can learn from it.
- No,and I understand that it is highly likely no one will implement it. Furthermore, I understand that this issue will likely become stale and will be closed.
validations:
required:true
- type:textarea
id:contribution
attributes:
label:What are you going to contribute?
description:You may also tell us what you have already done.
description:Miscellaneous questions and reports for the project (not support)
title: 'other:'
labels:
- meta/help wanted
body:
- type:dropdown
id:subject
attributes:
label:Subject
options:
- I would like to contribute to the project
- I would like to configure a not documented mail server use case
- I would like some feedback concerning a use case
- Something else that requires developers attention
validations:
required:true
- type:textarea
id:description
validations:
required:true
attributes:
label:Description
value:|
<!---
Please do not use this form to bypass doing a proper bug report.
The issue tracker is for anything relevant to the project itself, not individual support queries.
If you don't want to fill out a bug report, please ask support questions at our community discussions page:https://github.com/orgs/docker-mailserver/discussions
# This input does not fallback to the GITHUB_TOKEN taken from context, nor log that it will skip extra features of the action when this input is not set:
This project is Open Source. That means that you can contribute on enhancements, bug fixing or improving the [documentation](https://docker-mailserver.github.io/docker-mailserver/edge).
1. [Issues & PRs](#issues--prs)
1. [Opening an Issue](#opening-an-issue)
2. [Pull Request](#pull-requests)
2. [Coding Style](#coding-style)
1. [Bash and Shell](#bash-and-shell)
2. [YAML](#yaml)
## Issues & PRs
### Opening an Issue
**Before opening an issue**, read the [`README`](./README.md) carefully, use the [Documentation](https://docker-mailserver.github.io/docker-mailserver/edge), the Postfix/Dovecot documentation and your search engine you trust. The issue tracker is not meant to be used for unrelated questions! When opening an issue, please provide details use case to let the community reproduce your problem. Please start the mail server with env `DMS_DEBUG=1` and paste the output into the issue. **Use the issue templates** to provide the necessary information. Issues which do not use these templates are not worked on and closed. By raising issues, I agree to these terms and I understand, that the rules set for the issue tracker will help both maintainers as well as everyone to find a solution.
Maintainers take the time to improve on this project and help by solving issues together. It is therefore expected from others to make an effort and **comply with the rules**.
### Pull Requests
#### Submit a Pull-Request
You want to add a feature? Feel free to start creating an issue explaining what you want to do and how you're thinking doing it. Other users may have the same need and collaboration may lead to better results.
The development workflow is the following:
1. Fork the project and clone your fork
1. Create a new branch to work on
2. Run `git submodule update --init --recursive`
2. Write the code that is needed :D
3. Add integration tests if necessary
4. Get the linters with `make install_linters` and install `jq` with the package manager of your OS
5. Use `make clean all` to build image locally and run tests (note that tests work on Linux **only**)
6. Document your improvements if necessary (e.g. if you introduced new environment variables, write the description in [`ENVIRONMENT.md`](./ENVIRONMENT.md))
7. [Commit][commit] and [sign your commit][gpg], push and create a pull-request to merge into `master`. Please **use the pull-request template** to provide a minimum of contextual information and make sure to meet the requirements of the checklist.
1. Pull requests are automatically tested against the CI and will be reviewed when tests pass
2. When your changes are validated, your branch is merged
3. CI builds the new `:edge` image immediately and your changes will be includes in the next version release.
## Coding Style
### Bash and Shell
When refactoring, writing or altering scripts, that is Shell and Bash scripts, in any way, adhere to these rules:
1. **Adjust your style of coding to the style that is already present**! Even if you do not like it, this is due to consistency. There was a lot of work involved in making all scripts consistent.
2. **Use `shellcheck` to check your scripts**! Your contributions are checked by TravisCI too, so you will need to do this. You can **lint your work with `make lint`** to check against all targets.
3. **Use the provided `.editorconfig`** file.
4. Use `/bin/bash` or `/usr/bin/env bash` instead of `/bin/sh`. Adjust the style accordingly.
5. `setup.sh` provides a good starting point to look for.
6. When appropriate, use the `set` builtin. We recommend `set -euEo pipefail` or `set -uE`.
#### Styling rules
##### If-Else-Statements
``` BASH
# when using braces, use double braces
# remember you do not need "" when using [[ ]]
if [[ <CONDITION1> ]] && [[ -f ${FILE} ]]
then
<CODETORUN>
# when running commands, you don't need braces
elif <COMMANDTORUN>
<CODETOTUN>
else
<CODETOTUN>
fi
# equality checks with numbers are done
# with -eq/-ne/-lt/-ge, not != or ==
if [[ ${VAR} -ne 42 ]] || [[ ${SOME_VAR} -eq 6 ]]
then
<CODETORUN>
fi
```
##### Variables & Braces
Variables are always uppercase. We always use braces.
If you forgot this and want to change it later, you can use [this link][regex]. The used regex is `\$([^{("\\'\/])([a-zA-Z0-9_]*)([^}\/ \t'"\n.\]:(=\\-]*)`, where you should in practice be able to replace all variable occurrences without braces with occurrences with braces.
``` BASH
# good
local VAR="good"
local NEW="${VAR}"
# bad -> TravisCI will fail
var="bad"
new=$var
```
##### Loops
Like `if-else`, loops look like this
``` BASH
for / while <LOOPCONDITION>
do
<CODETORUN>
done
```
##### Functions
It's always nice to see the use of functions as it also provides a clear structure. If scripts are small, this is unnecessary, but if they become larger, please consider using functions. When doing so, provide `function _main`.
``` BASH
function _<name_underscored_and_lowercase>
{
<CODETORUN>
# variables that can be local should be local
local <LOCAL_VARIABLE_NAME>
}
```
##### Error Tracing
A construct to trace error in your scripts looks like this. Remember: Remove `set -x` in the end. This is for debugging purposes only.
Comments should only describe non-obvious matters. Comments should start lowercase when they aren't sentences. Make the code **self-descriptive** by using meaningful names! Make comments not longer than approximately 80 columns, then wrap the line.
A positive example, which is taken from `start-mailserver.sh`, would be
``` BASH
function _setup_postfix_aliases
{
_notify 'task' 'Setting up Postfix Aliases'
: >/etc/postfix/virtual
: >/etc/postfix/regexp
if [[ -f /tmp/docker-mailserver/postfix-virtual.cf ]]
then
# fixing old virtual user file
if grep -q ",$" /tmp/docker-mailserver/postfix-virtual.cf
then
sed -i -e "s/, /,/g" -e "s/,$//g" /tmp/docker-mailserver/postfix-virtual.cf
LABEL org.opencontainers.image.authors="The Docker Mailserver Organization on GitHub"
LABEL org.opencontainers.image.licenses="MIT"
LABEL org.opencontainers.image.description="A fullstack but simple mail server (SMTP, IMAP, LDAP, Antispam, Antivirus, etc.). Only configuration files, no SQL database."
LABEL org.opencontainers.image.authors="The Docker Mailserver Organization on GitHub"
LABEL org.opencontainers.image.licenses="MIT"
LABEL org.opencontainers.image.description="A fullstack but simple mail server (SMTP, IMAP, LDAP, Anti-spam, Anti-virus, etc.). Only configuration files, no SQL database."
1. If an option doesn't work as documented here, check if you are running the latest image!
2. Values in **bold** are the default values.
3. Since `docker-mailserver v7.1.0`, comparisons for environment variables are executed differently. If you previously used `VARIABLE=''` as the `empty` value, **update** to now use `VARIABLE=`.
### Assignments
#### General
##### DMS_DEBUG
- **0** => Debug disabled
- 1 => Enables debug on startup
##### SUPERVISOR_LOGLEVEL
Here you can adjust the [log-level for Supervisor](http://supervisord.org/logging.html#activity-log-levels). Possible values are
- critical => Only show critical messages
- error => Only show erroneous output
- **warn** => Show warnings
- info => Normal informational output
- debug => Also show debug messages
The log-level will show everything in its class and above.
##### ENABLE_AMAVIS
Amavis content filter (used for ClamAV & SpamAssassin)
- 0 => Amavis is disabled
- **1** => Amavis is enabled
##### ENABLE_CLAMAV
- **0** => Clamav is disabled
- 1 => Clamav is enabled
##### ONE_DIR
- **0** => state in default directories
- 1 => consolidate all states into a single directory (`/var/mail-state`) to allow persistence using docker volumes
##### ENABLE_POP3
- **empty** => POP3 service disabled
- 1 => Enables POP3 service
##### ENABLE_FAIL2BAN
- **0** => fail2ban service disabled
- 1 => Enables fail2ban service
If you enable Fail2Ban, don't forget to add the following lines to your `docker-compose.yml`:
- manual => Let you manually specify locations of your SSL certificates for non-standard cases
- Requires: `SSL_CERT_PATH` and `SSL_KEY_PATH` ENV vars to be set to the location of the files within the container.
- Optional: `SSL_ALT_CERT_PATH` and `SSL_ALT_KEY_PATH` allow providing a 2nd certificate as a fallback for dual (aka hybrid) certificate support. Useful for ECDSA with an RSA fallback. Presently only `manual` mode supports this feature.
Please read [the SSL page in the documentation](https://docker-mailserver.github.io/docker-mailserver/edge/config/security/ssl) for more information.
##### TLS_LEVEL
- **empty** => modern
- modern => Enables TLSv1.2 and modern ciphers only. (default)
- intermediate => Enables TLSv1, TLSv1.1 and TLSv1.2 and broad compatibility ciphers.
##### SPOOF_PROTECTION
Configures the handling of creating mails with forged sender addresses.
- **empty** => Mail address spoofing allowed. Any logged in user may create email messages with a forged sender address. See also [Wikipedia](https://en.wikipedia.org/wiki/Email_spoofing)(not recommended, but default for backwards compatibility reasons)
- 1 => (recommended) Mail spoofing denied. Each user may only send with his own or his alias addresses. Addresses with [extension delimiters](http://www.postfix.org/postconf.5.html#recipient_delimiter) are not able to send messages.
##### ENABLE_SRS
Enables the Sender Rewriting Scheme. SRS is needed if your mail server acts as forwarder. See [postsrsd](https://github.com/roehling/postsrsd/blob/master/README.md#sender-rewriting-scheme-crash-course) for further explanation.
- **0** => Disabled
- 1 => Enabled
##### PERMIT_DOCKER
Set different options for mynetworks option (can be overwrite in postfix-main.cf) **WARNING**: Adding the docker network's gateway to the list of trusted hosts, e.g. using the `network` or `connected-networks` option, can create an [**open relay**](https://en.wikipedia.org/wiki/Open_mail_relay), for instance if IPv6 is enabled on the host machine but not in Docker.
- **empty** => localhost only
- host => Add docker host (ipv4 only)
- network => Add the docker default bridge network (172.16.0.0/12); **WARNING**: `docker-compose` might use others (e.g. 192.168.0.0/16) use `PERMIT_DOCKER=connected-networks` in this case
- connected-networks => Add all connected docker networks (ipv4 only)
Note: you probably want to [set `POSTFIX_INET_PROTOCOLS=ipv4`](#postfix_inet_protocols) to make it work fine with Docker.
##### NETWORK_INTERFACE
In case your network interface differs from `eth0`, e.g. when you are using HostNetworking in Kubernetes, you can set this to whatever interface you want. This interface will then be used.
- **empty** => `eth0`
##### VIRUSMAILS_DELETE_DELAY
Set how many days a virusmail will stay on the server before being deleted
- **empty** => 7 days
##### ENABLE_POSTFIX_VIRTUAL_TRANSPORT
This Option is activating the Usage of POSTFIX_DAGENT to specify a ltmp client different from default dovecot socket.
- **empty** => disabled
- 1 => enabled
##### POSTFIX_DAGENT
Enabled by ENABLE_POSTFIX_VIRTUAL_TRANSPORT. Specify the final delivery of postfix
- **empty**: fail
- `lmtp:unix:private/dovecot-lmtp` (use socket)
- `lmtps:inet:<host>:<port>` (secure lmtp with starttls, take a look at <https://sys4.de/en/blog/2014/11/17/sicheres-lmtp-mit-starttls-in-dovecot/>)
- `lmtp:<kopano-host>:2003` (use kopano as mailstore)
- etc.
##### POSTFIX\_MAILBOX\_SIZE\_LIMIT
Set the mailbox size limit for all users. If set to zero, the size will be unlimited (default).
- **empty** => 0 (no limit)
##### ENABLE_QUOTAS
- **1** => Dovecot quota is enabled
- 0 => Dovecot quota is disabled
See [mailbox quota](https://docker-mailserver.github.io/docker-mailserver/edge/config/user-management/accounts/#notes).
##### POSTFIX\_MESSAGE\_SIZE\_LIMIT
Set the message size limit for all users. If set to zero, the size will be unlimited (not recommended!)
- **empty** => 10240000 (~10 MB)
##### ENABLE_MANAGESIEVE
- **empty** => Managesieve service disabled
- 1 => Enables Managesieve on port 4190
##### OVERRIDE_HOSTNAME
- **empty** => uses the `hostname` command to get the mail server's canonical hostname
- => Specify a fully-qualified domainname to serve mail for. This is used for many of the config features so if you can't set your hostname (e.g. you're in a container platform that doesn't let you) specify it in this environment variable.
##### POSTMASTER_ADDRESS
- **empty** => postmaster@domain.com
- => Specify the postmaster address
##### POSTSCREEN_ACTION
- **enforce** => Allow other tests to complete. Reject attempts to deliver mail with a 550 SMTP reply, and log the helo/sender/recipient information. Repeat this test the next time the client connects.
- drop => Drop the connection immediately with a 521 SMTP reply. Repeat this test the next time the client connects.
- ignore => Ignore the failure of this test. Allow other tests to complete. Repeat this test the next time the client connects. This option is useful for testing and collecting statistics without blocking mail.
##### DOVECOT_MAILBOX_FORMAT
- **maildir** => uses very common Maildir format, one file contains one message
- sdbox => (experimental) uses Dovecot high-performance mailbox format, one file contains one message
- mdbox ==> (experimental) uses Dovecot high-performance mailbox format, multiple messages per file and multiple files per box
This option has been added in November 2019. Using other format than Maildir is considered as experimental in docker-mailserver and should only be used for testing purpose. For more details, please refer to [Dovecot Documentation](https://wiki2.dovecot.org/MailboxFormat).
##### POSTFIX_INET_PROTOCOLS
- **all** => All possible protocols.
- ipv4 => Use only IPv4 traffic. Most likely you want this behind Docker.
- ipv6 => Use only IPv6 traffic.
Note: More details in <http://www.postfix.org/postconf.5.html#inet_protocols>
#### Reports
##### PFLOGSUMM_TRIGGER
Enables regular pflogsumm mail reports.
- **not set** => No report
- daily_cron => Daily report for the previous day
- logrotate => Full report based on the mail log when it is rotated
This is a new option. The old REPORT options are still supported for backwards compatibility.
If this is not set and reports are enabled with the old options, logrotate will be used.
##### PFLOGSUMM_RECIPIENT
Recipient address for pflogsumm reports.
- **not set** => Use REPORT_RECIPIENT or POSTMASTER_ADDRESS
- => Specify the recipient address(es)
##### PFLOGSUMM_SENDER
From address for pflogsumm reports.
- **not set** => Use REPORT_SENDER or POSTMASTER_ADDRESS
- => Specify the sender address
##### LOGWATCH_INTERVAL
Interval for logwatch report.
- **none** => No report is generated
- daily => Send a daily report
- weekly => Send a report every week
##### LOGWATCH_RECIPIENT
Recipient address for logwatch reports if they are enabled.
- **not set** => Use REPORT_RECIPIENT or POSTMASTER_ADDRESS
- => Specify the recipient address(es)
##### REPORT_RECIPIENT (deprecated)
Enables a report being sent (created by pflogsumm) on a regular basis.
- **0** => Report emails are disabled unless enabled by other options
- 1 => Using POSTMASTER_ADDRESS as the recipient
- => Specify the recipient address
##### REPORT_SENDER (deprecated)
Change the sending address for mail report
- **empty** => mailserver-report@hostname
- => Specify the report sender (From) address
##### REPORT_INTERVAL (deprecated)
Changes the interval in which logs are rotated and a report is being sent (deprecated).
- **daily** => Send a daily report
- weekly => Send a report every week
- monthly => Send a report every month
Note: This variable used to control logrotate inside the container and sent the pflogsumm report when the logs were rotated.
It is still supported for backwards compatibility, but the new option LOGROTATE_INTERVAL has been added that only rotates
the logs.
##### LOGROTATE_INTERVAL
Defines the interval in which the mail log is being rotated.
- **daily** => Rotate daily.
- weekly => Rotate weekly.
- monthly => Rotate monthly.
Note that only the log inside the container is affected.
The full log output is still available via `docker logs mail` (or your respective container name).
If you want to control logrotation for the docker generated logfile see: [Docker Logging Drivers](https://docs.docker.com/config/containers/logging/configure/).
Also note that by default the logs are lost when the container is recycled. To keep the logs, mount a volume.
Finally the logrotate interval **may** affect the period for generated reports. That is the case when the reports are triggered by log rotation.
#### Spamassassin
##### ENABLE_SPAMASSASSIN
- **0** => Spamassassin is disabled
- 1 => Spamassassin is enabled
**/!\\ Spam delivery:** when Spamassassin is enabled, messages marked as spam WILL NOT BE DELIVERED.
Use `SPAMASSASSIN_SPAM_TO_INBOX=1` for receiving spam messages.
##### SPAMASSASSIN_SPAM_TO_INBOX
- **0** => Spam messages will be bounced (_rejected_) without any notification (_dangerous_).
- 1 => Spam messages will be delivered to the inbox and tagged as spam using `SA_SPAM_SUBJECT`.
##### MOVE_SPAM_TO_JUNK
- **1** => Spam messages will be delivered in the `Junk` folder.
- 0 => Spam messages will be delivered in the mailbox.
Note: this setting needs `SPAMASSASSIN_SPAM_TO_INBOX=1`
##### SA_TAG
- **2.0** => add spam info headers if at, or above that level
Note: this spamassassin setting needs `ENABLE_SPAMASSASSIN=1`
##### SA_TAG2
- **6.31** => add 'spam detected' headers at that level
Note: this spamassassin setting needs `ENABLE_SPAMASSASSIN=1`
##### SA_KILL
- **6.31** => triggers spam evasive actions
Note: this spamassassin setting needs `ENABLE_SPAMASSASSIN=1`. By default, the mailserver is configured to quarantine spam emails. If emails are quarantined, they are compressed and stored in a location dependent on the ONE_DIR setting above. If `ONE_DIR=1` the location is /var/mail-state/lib-amavis/virusmails/. If `ONE_DIR=0` it is /var/lib/amavis/virusmails/. These paths are inside the docker container. To inhibit this behaviour and deliver spam emails, set this to a very high value e.g. 100.0.
##### SA_SPAM_SUBJECT
- **\*\*\*SPAM\*\*\*** => add tag to subject if spam detected
Note: this spamassassin setting needs `ENABLE_SPAMASSASSIN=1`. Add the spamassassin score to the subject line by inserting the keyword \_SCORE\_: **\*\*\*SPAM(\_SCORE\_)\*\*\***.
##### SA_SHORTCIRCUIT_BAYES_SPAM
- **1** => will activate spamassassin short circuiting for bayes spam detection.
This will uncomment the respective line in ```/etc/spamassasin/local.cf```
Note: activate this only if you are confident in your bayes database for identifying spam.
##### SA_SHORTCIRCUIT_BAYES_HAM
- **1** => will activate spamassassin short circuiting for bayes ham detection
This will uncomment the respective line in ```/etc/spamassasin/local.cf```
Note: activate this only if you are confident in your bayes database for identifying ham.
#### Fetchmail
##### ENABLE_FETCHMAIL
- **0** => `fetchmail` disabled
- 1 => `fetchmail` enabled
##### FETCHMAIL_POLL
- **300** => `fetchmail` The number of seconds for the interval
##### FETCHMAIL_PARALLEL
**0** => `fetchmail` runs with a single config file `/etc/fetchmailrc`
**1** => `/etc/fetchmailrc` is split per poll entry. For every poll entry a seperate fetchmail instance is started to allow having multiple imap idle configurations defined.
Note: The defaults of your fetchmailrc file need to be at the top of the file. Otherwise it won't be added correctly to all separate `fetchmail` instances.
#### LDAP
##### ENABLE_LDAP
- **empty** => LDAP authentification is disabled
- 1 => LDAP authentification is enabled
- NOTE:
- A second container for the ldap service is necessary (e.g. [docker-openldap](https://github.com/osixia/docker-openldap))
- For preparing the ldap server to use in combination with this container [this](http://acidx.net/wordpress/2014/06/installing-a-mailserver-with-postfix-dovecot-sasl-ldap-roundcube/) article may be helpful
##### LDAP_START_TLS
- **empty** => no
- yes => LDAP over TLS enabled for Postfix
##### LDAP_SERVER_HOST
- **empty** => mail.domain.com
- => Specify the dns-name/ip-address where the ldap-server
- NOTE: If you going to use the mailserver in combination with docker-compose you can set the service name here
##### LDAP_SEARCH_BASE
- **empty** => ou=people,dc=domain,dc=com
- => e.g. LDAP_SEARCH_BASE=dc=mydomain,dc=local
##### LDAP_BIND_DN
- **empty** => cn=admin,dc=domain,dc=com
- => take a look at examples of SASL_LDAP_BIND_DN
##### LDAP_BIND_PW
- **empty** => admin
- => Specify the password to bind against ldap
##### LDAP_QUERY_FILTER_USER
- e.g. `(&(mail=%s)(mailEnabled=TRUE))`
- => Specify how ldap should be asked for users
##### LDAP_QUERY_FILTER_GROUP
- e.g. `(&(mailGroupMember=%s)(mailEnabled=TRUE))`
- => Specify how ldap should be asked for groups
##### LDAP_QUERY_FILTER_ALIAS
- e.g. `(&(mailAlias=%s)(mailEnabled=TRUE))`
- => Specify how ldap should be asked for aliases
##### LDAP_QUERY_FILTER_DOMAIN
- e.g. `(&(|(mail=*@%s)(mailalias=*@%s)(mailGroupMember=*@%s))(mailEnabled=TRUE))`
- => Specify how ldap should be asked for domains
##### DOVECOT_TLS
- **empty** => no
- yes => LDAP over TLS enabled for Dovecot
#### Dovecot
The following variables overwrite the default values for ```/etc/dovecot/dovecot-ldap.conf.ext```.
##### DOVECOT_BASE
- **empty** => same as `LDAP_SEARCH_BASE`
- => Tell Dovecot to search only below this base entry. (e.g. `ou=people,dc=domain,dc=com`)
##### DOVECOT_DEFAULT_PASS_SCHEME
- **empty** => `SSHA`
- => Select one crypt scheme for password hashing from this list of [password schemes](https://doc.dovecot.org/configuration_manual/authentication/password_schemes/).
##### DOVECOT_DN
- **empty** => same as `LDAP_BIND_DN`
- => Bind dn for LDAP connection. (e.g. `cn=admin,dc=domain,dc=com`)
##### DOVECOT_DNPASS
- **empty** => same as `LDAP_BIND_PW`
- => Password for LDAP dn sepecifified in `DOVECOT_DN`.
##### DOVECOT_HOSTS
- **empty** => same as `LDAP_SERVER_HOST`
- => Specify a space separated list of LDAP hosts.
- **envelope_sender** => Rewrite only envelope sender address
- header_sender => Rewrite only header sender (not recommended)
- envelope_sender,header_sender => Rewrite both senders
##### SRS_EXCLUDE_DOMAINS
- **empty** => Envelope sender will be rewritten for all domains
- provide comma separated list of domains to exclude from rewriting
##### SRS_SECRET
- **empty** => generated when the container is started for the first time
- provide a secret to use in base64
- you may specify multiple keys, comma separated. the first one is used for signing and the remaining will be used for verification. this is how you rotate and expire keys
- if you have a cluster/swarm make sure the same keys are on all nodes
- example command to generate a key: `dd if=/dev/urandom bs=24 count=1 2>/dev/null | base64`
##### SRS_DOMAINNAME
- **empty** => Derived from OVERRIDE_HOSTNAME, DOMAINNAME, or the container's hostname
- Set this if auto-detection fails, isn't what you want, or you wish to have a separate container handle DSNs
#### Default Relay Host
##### DEFAULT_RELAY_HOST
- **empty** => don't set default relayhost setting in main.cf
- default host and port to relay all mail through.
Format: `[example.com]:587` (don't forget the brackets if you need this to
be compatible with `$RELAY_USER` and `$RELAY_PASSWORD`, explained below).
#### Multi-domain Relay Hosts
##### RELAY_HOST
- **empty** => don't configure relay host
- default host to relay mail through
##### RELAY_PORT
- **empty** => 25
- default port to relay mail through
##### RELAY_USER
- **empty** => no default
- default relay username (if no specific entry exists in postfix-sasl-password.cf)
A fullstack but simple mail server (SMTP, IMAP, LDAP, Antispam, Antivirus, etc.). Only configuration files, no SQL database. Keep it simple and versioned. Easy to deploy and upgrade.
## :page_with_curl: About
[Why this image was created.](http://tvi.al/simple-mail-server-with-docker/)
A production-ready fullstack but simple containerized mail server (SMTP, IMAP, LDAP, Anti-spam, Anti-virus, etc.).
- Only configuration files, no SQL database. Keep it simple and versioned. Easy to deploy and upgrade.
- Originally created by [@tomav](https://github.com/tomav), this project is now maintained by volunteers since January 2021.
1. [Included Services](#included-services)
2. [Issues and Contributing](./CONTRIBUTING.md)
3. [Requirements](#requirements)
4. [Usage](#usage)
5. [Examples](#examples)
6. [Environment Variables](./ENVIRONMENT.md)
7. [Release Notes](./CHANGELOG.md)
## <!-- Adds a thin line break separator style -->
## Included Services
> [!TIP]
> Be sure to read [our documentation][documentation::web]. It provides guidance on initial setup of your mail server.
- [Postfix](http://www.postfix.org) with SMTP or LDAP auth
- [Dovecot](https://www.dovecot.org) for SASL, IMAP (or POP3), with LDAP Auth, Sieve and [quotas](https://docker-mailserver.github.io/docker-mailserver/edge/config/user-management/accounts#notes)
> [!IMPORTANT]
> If you have issues, please search through [the documentation][documentation::web] **for your version** before opening an issue.
>
> The issue tracker is for issues, not for personal support.
> Make sure the version of the documentation matches the image version you're using!
- [Postfix](http://www.postfix.org) with SMTP or LDAP authentication and support for [extension delimiters](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#aliases)
- [Dovecot](https://www.dovecot.org) with SASL, IMAP, POP3, LDAP, [basic Sieve support](https://docker-mailserver.github.io/docker-mailserver/latest/config/advanced/mail-sieve) and [quotas](https://docker-mailserver.github.io/docker-mailserver/latest/config/account-management/overview/#quotas)
- [Extension Delimiters](http://www.postfix.org/postconf.5.html#recipient_delimiter) (`you+extension@example.com` go to `you@example.com`)
## Requirements
**Recommended**:
- 1 Core
- 2GB RAM
- Swap enabled for the container
**Minimum**:
- 1 vCore
- 512MB RAM
**Note:** You'll need to deactivate some services like ClamAV to be able to run on a host with 512MB of RAM. Even with 1G RAM you may run into problems without swap, see [FAQ](https://docker-mailserver.github.io/docker-mailserver/edge/faq/#what-system-requirements-are-required-to-run-docker-mailserver-effectively).
## Usage
### Available image sources / tags
The [CI/CD workflows](https://github.com/docker-mailserver/docker-mailserver/actions) automatically build, test and push new images to container registries. Currently, the following registries are supported:
**Make sure to get the `setup.sh` that comes with the release you're using**. Look up the release and the git commit on which this release is based upon by selecting the appropriate tag on GitHub. This can done with the "Switch branches/tags" button on GitHub, choosing the right tag. This is done in order to rule out possible inconsistencies between versions.
### Create a docker-compose environment
- [Install the latest docker-compose](https://docs.docker.com/compose/install/)
- Edit the files `.env` and `mailserver.env` to your liking:
- `.env` contains the configuration for Docker Compose
- `mailserver.env` contains the configuration for the mailserver container
- these files supports [only simple `VAR=VAL`](https://docs.docker.com/compose/env-file/)
- don't quote your values
- variable substitution is *not* supported (e.g. `OVERRIDE_HOSTNAME=$HOSTNAME.$DOMAINNAME`).
- Variables in `.env` are expanded in the `docker-compose.yml` file **only** and **not** in the container. The file `mailserver.env` serves this case where environment variables are used in the container.
- If you want to use a bare domain (host name = domain name), see [FAQ](https://docker-mailserver.github.io/docker-mailserver/edge/faq#can-i-use-nakedbare-domains-no-host-name)
### Get up and running
If you'd like to use SELinux, add `-Z` to the variable `SELINUX_LABEL` in `.env`. If you want the volume bind mount to be shared among other containers switch `-Z` to `-z`
``` BASH
docker-compose up -d mailserver
# without SELinux
./setup.sh email add <user@domain> [<password>]
./setup.sh alias add postmaster@<domain><user@domain>
./setup.sh -Z alias add postmaster@<domain><user@domain>
./setup.sh -Z config dkim
```
If you're seeing error messages about unchecked error, please **verify that you're using the right version of `setup.sh`**. Refer to the [Get the tools](#get-the-tools) section and / or execute `./setup.sh help` and read the `VERSION` section.
In case you're using LDAP, the setup looks a bit different as you do not add user accounts directly. Postfix doesn't know your domain(s) and you need to provide it when configuring DKIM:
If you want to see detailed usage information, run `./setup.sh config dkim help`.
### Miscellaneous
#### DNS - DKIM
When keys are generated, you can configure your DNS server by just pasting the content of `config/opendkim/keys/domain.tld/mail.txt` to [set up DKIM](https://mxtoolbox.com/dmarc/dkim/setup/how-to-setup-dkim).
#### Custom user changes & patches
If you'd like to change, patch or alter files or behavior of `docker-mailserver`, you can use a script. Just place it the `config/` folder that is created on startup and call it `user-patches.sh`. The setup is done like this:
``` BASH
# 1. Either create the config/ directory yourself
# or let docker-mailserver create it on initial
# startup
/where/docker-mailserver/resides/ $ mkdir config && cd config
# 2. Create the user-patches.sh script and make it
And you're done. The user patches script runs right before starting daemons. That means, all the other configuration is in place, so the script can make final adjustments.
#### Supported Operating Systems
We are currently providing support for Linux. Windows is _not_ supported and is known to cause problems. Similarly, macOS is _not officially_ supported - but you may get it to work there. In the end, Linux should be your preferred operating system for this image, especially when using this mailserver in production.
#### Support for Multiple Domains
`docker-mailserver` supports multiple domains out of the box, so you can do this:
You're done! And don't forget to have a look at the remaining functions of the `setup.sh` script with `./setup.sh help`.
#### SPF/Forwarding Problems
If you got any problems with SPF and/or forwarding mails, give [SRS](https://github.com/roehling/postsrsd/blob/master/README.md) a try. You enable SRS by setting `ENABLE_SRS=1`. See the variable description for further information.
1. A connection *may* be secured over TLS when both ends support `STARTTLS`. On ports 110, 143 and 587, `docker-mailserver` will reject a connection that cannot be secured. Port 25 is [required](https://serverfault.com/questions/623692/is-it-still-wrong-to-require-starttls-on-incoming-smtp-messages) to support insecure connections.
2. Receives email and filters for spam and viruses. For submitting outgoing mail you should prefer the submission ports(465, 587), which require authentication. Unless a relay host is configured, outgoing email will leave the server via port 25(thus outbound traffic must not be blocked by your provider or firewall).
3. A submission port since 2018, [RFC 8314](https://tools.ietf.org/html/rfc8314). Originally a secure variant of port 25.
See the [documentation](https://docker-mailserver.github.io/docker-mailserver/edge/config/security/understanding-the-ports/) for further details and best practice advice, especially regarding security concerns.
## Examples
### With Relevant Environmental Variables
This example provides you only with a basic example of what a minimal setup could look like. We **strongly recommend** that you go through the configuration file yourself and adjust everything to your needs. The default [docker-compose.yml](./docker-compose.yml) can be used for the purpose out-of-the-box, see the [usage section](#usage).
Due to the way DMS is released, the most recent patches and the most current software is published on the `:edge` tag of the container image. Hence, security updates will land on this "rolling release tag". Older tags need manual updating, as we do not usually release an updated image for an existing tag; this will only be done in case of _severe_ vulnerabilities.
| Image Tags | Latest Packages & Patches |
|-------------|:-------------------------:|
| `:edge` | :white_check_mark: |
| not `:edge` | :x: |
## Reporting a Vulnerability
When reporting a vulnerability, you can use GitHub's "Private Vulnerability Reporting". Just navigate to the [Open an Issue](https://github.com/docker-mailserver/docker-mailserver/issues/new/choose) page and choose "Report a security vulnerability". This way, maintainers will privately notified first. Afterwards, in a best-case scenario, if the vulnerability is fixed, the report will be made public.
# This user patches script runs right before starting the daemons. That means, all the other configuration is in place, so the script can make final adjustments.
# If you modify any supervisord configuration, make sure to run "supervisorctl update" or "supervisorctl reload" afterwards.
# To enable the script, you must save it in your config directory as "user-patches.sh".
# `postfix-main.cf`, a single line change to make all outbound SMTP connections over implicit TLS instead of the default explicit TLS (StartTLS).
# NOTE: If you need to only selectively relay mail, you would need to instead adjust this on the relay service in `/etc/postfix/master.cf`,
# However DMS presently modifies this when using the DMS Relay Host feature support, which may override `postfix-master.cf` or `user-patches.sh` due to `check-for-changes.sh`.
dms-main:
content:|
smtp_tls_wrappermode=yes
# DMS expects an account to be configured to run, this example provides accounts already created.
d="m 37.133,105.617 h 68.54 L 133.68,94.983 a 3.822,3.822 0 0 0 2.216,-4.93 L 113.933,32.2 a 3.821,3.821 0 0 0 -4.93,-2.216 l -84.34,32.021 a 3.821,3.821 0 0 0 -2.215,4.93 z"
fill="#f3ac47"
id="path2" />
<path
d="m 105.674,105.617 28.009,-10.634 a 3.8,3.8 0 0 0 2.073,-1.907 L 76.443,71.304 57.995,105.617 Z"
fill="#f19a3d"
id="path4-5" />
<path
d="m 22.386,64.451 59.517,21.233 29.918,-55.65 a 3.799,3.799 0 0 0 -2.817,-0.05 l -84.34,32.022 a 3.796,3.796 0 0 0 -2.278,2.445 z"
d="M 23.77,11.079 C 23.701,11.034 22.272,10.132 20.921,10.347 20.594,8.978 19.338,8.125 19.276,8.084 a 0.499,0.499 0 0 0 -0.638,0.069 c -0.1,0.104 -0.967,1.074 -0.722,3.282 A 0.52,0.52 0 0 1 17.79,11.846 0.447,0.447 0 0 1 17.448,12 H 17 V 9.5 A 0.5,0.5 0 0 0 16.5,9 H 14 V 3.5 A 0.5,0.5 0 0 0 13.5,3 h -3 A 0.5,0.5 0 0 0 10,3.5 V 6 H 4.5 A 0.5,0.5 0 0 0 4,6.5 V 9 H 1.5 A 0.5,0.5 0 0 0 1,9.5 V 12 H 0.5 A 0.5,0.5 0 0 0 0,12.5 c 0,7.672 5.595,8.5 8,8.5 6.193,0 10.498,-3.762 12.377,-7.077 2.476,-0.395 3.509,-2.094 3.554,-2.168 A 0.501,0.501 0 0 0 23.77,11.079 Z"
d="m 37.133,105.617 h 68.54 L 133.68,94.983 a 3.822,3.822 0 0 0 2.216,-4.93 L 113.933,32.2 a 3.821,3.821 0 0 0 -4.93,-2.216 l -84.34,32.021 a 3.821,3.821 0 0 0 -2.215,4.93 z"
fill="#f3ac47"
id="path2-3" />
<path
d="m 105.674,105.617 28.009,-10.634 a 3.8,3.8 0 0 0 2.073,-1.907 L 76.443,71.304 57.995,105.617 Z"
fill="#f19a3d"
id="path4" />
<path
d="m 22.386,64.451 59.517,21.233 29.918,-55.65 a 3.799,3.799 0 0 0 -2.817,-0.05 l -84.34,32.022 a 3.796,3.796 0 0 0 -2.278,2.445 z"
d="M 23.77,11.079 C 23.701,11.034 22.272,10.132 20.921,10.347 20.594,8.978 19.338,8.125 19.276,8.084 a 0.499,0.499 0 0 0 -0.638,0.069 c -0.1,0.104 -0.967,1.074 -0.722,3.282 A 0.52,0.52 0 0 1 17.79,11.846 0.447,0.447 0 0 1 17.448,12 H 0.5 A 0.5,0.5 0 0 0 0,12.5 c 0,7.672 5.595,8.5 8,8.5 6.193,0 10.498,-3.762 12.377,-7.077 2.476,-0.395 3.509,-2.094 3.554,-2.168 A 0.501,0.501 0 0 0 23.77,11.079 Z"
fill="#303c42"
id="path10-5" />
<path
d="M 20.002,12.965 A 0.5,0.5 0 0 0 19.62,13.226 C 17.942,16.351 13.891,20 8,20 5.454,20 1.228,19.076 1.009,13 h 16.439 c 0.422,0 0.809,-0.173 1.089,-0.486 0.287,-0.321 0.424,-0.754 0.375,-1.189 -0.113,-1.02 0.053,-1.687 0.214,-2.069 C 19.501,9.617 20,10.241 20,11 c 0,0.173 0.09,0.334 0.237,0.426 a 0.503,0.503 0 0 0 0.486,0.021 c 0.595,-0.3 1.44,-0.048 2.015,0.212 -0.432,0.455 -1.301,1.138 -2.736,1.306 z"
fill="#42a5f5"
id="path12" />
<path
d="M 20.002,12.465 A 0.5,0.5 0 0 0 19.62,12.726 C 17.942,15.851 13.891,19.5 8,19.5 5.528,19.5 1.482,18.614 1.049,13 h -0.04 c 0.218,6.076 4.445,7 6.991,7 5.892,0 9.942,-3.649 11.62,-6.774 a 0.5,0.5 0 0 1 0.382,-0.261 c 1.436,-0.168 2.305,-0.851 2.736,-1.306 a 4.894,4.894 0 0 0 -0.384,-0.15 c -0.492,0.399 -1.254,0.827 -2.352,0.956 z"
opacity="0.1"
id="path14" />
<circle
cx="5.0409999"
cy="16"
r="1"
fill="#303c42"
id="circle16" />
<circle
cx="5.4689999"
cy="15.729"
r="0.32300001"
fill="#ffffff"
id="circle18" />
<path
d="M 23.77,11.079 C 23.701,11.034 22.272,10.132 20.921,10.347 20.594,8.978 19.338,8.125 19.276,8.084 a 0.499,0.499 0 0 0 -0.638,0.069 c -0.1,0.104 -0.967,1.074 -0.722,3.282 A 0.52,0.52 0 0 1 17.79,11.846 0.447,0.447 0 0 1 17.448,12 H 17 V 9.5 A 0.5,0.5 0 0 0 16.5,9 H 14 V 3.5 A 0.5,0.5 0 0 0 13.5,3 h -3 A 0.5,0.5 0 0 0 10,3.5 V 6 H 4.5 A 0.5,0.5 0 0 0 4,6.5 V 9 H 1.5 A 0.5,0.5 0 0 0 1,9.5 V 12 H 0.5 A 0.5,0.5 0 0 0 0,12.5 c 0,7.672 5.595,8.5 8,8.5 6.193,0 10.498,-3.762 12.377,-7.077 2.476,-0.395 3.509,-2.094 3.554,-2.168 A 0.501,0.501 0 0 0 23.77,11.079 Z"
fill="url(#a)"
id="path20-6" />
</g>
</g>
<path
d="M 23.77,11.079 C 23.701,11.034 22.272,10.132 20.921,10.347 20.594,8.978 19.338,8.125 19.276,8.084 a 0.499,0.499 0 0 0 -0.638,0.069 c -0.1,0.104 -0.967,1.074 -0.722,3.282 A 0.52,0.52 0 0 1 17.79,11.846 0.447,0.447 0 0 1 17.448,12 H 17 V 9.5 A 0.5,0.5 0 0 0 16.5,9 H 14 V 3.5 A 0.5,0.5 0 0 0 13.5,3 h -3 A 0.5,0.5 0 0 0 10,3.5 V 6 H 4.5 A 0.5,0.5 0 0 0 4,6.5 V 9 H 1.5 A 0.5,0.5 0 0 0 1,9.5 V 12 H 0.5 A 0.5,0.5 0 0 0 0,12.5 c 0,7.672 5.595,8.5 8,8.5 6.193,0 10.498,-3.762 12.377,-7.077 2.476,-0.395 3.509,-2.094 3.554,-2.168 A 0.501,0.501 0 0 0 23.77,11.079 Z"
Authentication from the provisioner can be supplemented with additional methods:
- [OAuth2 / OIDC][docs::account-auth::oauth2] (_allow login from an external authentication service_)
- [Master Accounts][docs::account-auth::master-accounts] (_access the mailbox of any DMS account_)
---
For custom authentication requirements, you could [implement this with Lua][docs::examples::auth-lua].
## Accounts
!!! info
To receive or send mail, you'll need to provision user accounts into DMS (_as each provisioner page documents_).
---
A DMS account represents a user with their _login username_ + password, and optional config like aliases and quota.
- Sending mail from different addresses **does not require** aliases or separate accounts.
- Each account is configured with a _primary email address_ that a mailbox is associated to.
??? info "Primary email address"
The email address associated to an account creates a mailbox. This address is relevant:
- When DMS **receives mail** for that address as the recipient (_or an alias that resolves to it_), to identify which mailbox to deliver into.
- With **mail submission**:
- `SPOOF_PROTECTION=1`**restricts the sender address** to the DMS account email address (_unless additional sender addresses have been permitted via supported config_).
- `SPOOF_PROTECTION=0` allows DMS accounts to **use any sender address** (_only a single DMS account is necessary to send mail with different sender addresses_).
---
For more details, see the [Technical Overview](#technical-overview) section.
??? note "Support for multiple mail domains"
No extra configuration in DMS is required after provisioning an account with an email address.
- The DNS records for a domain should direct mail to DMS and allow DMS to send mail on behalf of that domain.
- DMS does not need TLS certificates for your mail domains, only for the DMS FQDN (_the `hostname` setting_).
??? warning "Choosing a compatible email address"
An email address should conform to the standard [permitted charset and format][email-syntax::valid-charset-format] (`local-part@domain-part`).
---
DMS has features that need to reserve special characters to work correctly. Ensure those characters are not present in email addresses you configure for DMS, otherwise disable / opt-out of the feature.
- [Sub-addressing](#sub-addressing) is enabled by default with `+` as the _tag delimiter_. The tag can be changed, feature opt-out when the tag is explicitly unset.
### Aliases
!!! info
Aliases allow receiving mail:
- As an alternative delivery address for a DMS account mailbox.
- To redirect / forward to an external address outside of DMS like `@gmail.com`.
??? abstract "Technical Details (_Local vs Virtual aliases_)"
Aliases are managed through Postfix which supports _local_ and _virtual_ aliases:
- **Local aliases** are for mail routed to the [`local` delivery agent][postfix::delivery-agent::local] (see [associated alias config format][postfix::config-table::local-alias])
- You rarely need to configure this. It is used internally for system unix accounts belonging to the services running in DMS (_including `root`_).
- `postmaster` may be a local alias to `root`, and `root` to a virtual alias or real email address.
- Any mail sent through the `local` delivery agent will not be delivered to an inbox managed by Dovecot (_unless you have configured a local alias to redirect mail to a valid address or alias_).
- The domain-part of an these aliases belongs to your DMS FQDN (_`hostname: mail.example.com`, thus `user@mail.example.com`_). Technically there is no domain-part at this point, that context is used when routing delivery, the local delivery agent only knows of the local-part (_an alias or unix account_).
- [**Virtual aliases**][postfix-docs::virtual-alias] are for mail routed to the [`virtual` delivery agent][postfix::delivery-agent::virtual] (see [associated alias config format][postfix::config-table::virtual-alias])
- When alias support in DMS is discussed without the context of being a local or virtual alias, it's likely the virtual kind (_but could also be agnostic_).
- The domain-part of an these aliases belongs to a mail domain managed by DMS (_like `user@example.com`_).
!!! tip "Verify alias resolves correctly"
You can run `postmap -q <alias> <table>` in the container to verify an alias resolves to the expected target. If the target is also an alias, the command will not expand that alias to resolve the actual recipient(s).
For the `FILE` provisioner, an example would be: `postmap -q alias1@example.com /etc/postfix/virtual`. For the `LDAP` provisioner you'd need to adjust the table path.
!!! info "Side effect - Dovecot Quotas (`ENABLE_QUOTAS=1`)"
As a side effect of the alias workaround for the `FILE` provisioner with this feature, aliases can be used for account login. This is not intentional.
### Quotas
!!! info
Enables mail clients with the capability to query a mailbox for disk-space used and capacity limit.
- This feature is enabled by default, opt-out via [`ENABLE_QUOTAS=0`][docs::env::enable-quotas]
- **Not implemented** for the LDAP provisioner (_PR welcome! View the [feature request for implementation advice][gh-issue::dms-feature-request::dovecot-quotas-ldap]_)
??? tip "How are quotas useful?"
Without quota limits for disk storage, a mailbox could fill up the available storage which would cause delivery failures to all mailboxes.
Quotas help by preventing that abuse, so that only a mailbox exceeding the assigned quota experiences a delivery failure instead of negatively impacting others (_provided disk space is available_).
??? abstract "Technical Details"
The [Dovecot Quotas feature][gh-pr::dms-feature::dovecot-quotas] is configured by enabling the [Dovecot `imap-quota` plugin][dovecot-docs::plugin::imap-quota] and using the [`count` quota backend][dovecot-docs::config::quota-backend-count].
---
**Dovecot workaround for Postfix aliases**
When mail is delivered to DMS, Postfix will query Dovecot with the recipient(s) to verify quota has not been exceeded.
This allows early rejection of mail arriving to DMS, preventing a spammer from taking advantage of a [backscatter][wikipedia::backscatter] source if the mail was accepted by Postfix, only to later be rejected by Dovecot for storage when the quota limit was already reached.
However, Postfix does not resolve aliases until after the incoming mail is accepted.
1. Postfix queries Dovecot (_a [`check_policy_service` restriction tied to the Dovecot `quota-status` service][dms::workaround::dovecot-quotas::notes-1]_) with the recipient (_the alias_).
2. `dovecot: auth: passwd-file(alias@example.com): unknown user` is logged, Postfix is then informed that the recipient mailbox is not full even if it actually was (_since no such user exists in the Dovecot UserDB_).
3. However, when the real mailbox address that the alias would later resolve into does have a quota that exceeded the configured limit, Dovecot will refuse the mail delivery from Postfix which introduces a backscatter source for spammers.
As a [workaround to this problem with the `ENABLE_QUOTAS=1` feature][dms::workaround::dovecot-quotas::summary], DMS will add aliases as fake users into Dovecot UserDB (_that are configured with the same data as the real address the alias would resolve to, thus sharing the same mailbox location and quota limit_). This allows Postfix to properly be aware of an aliased mailbox having exceeded the allowed quota.
**NOTE:** This workaround **only supports** aliases to a single target recipient of a real account address / mailbox.
- Additionally, aliases that resolve to another alias or to an external address would both fail the UserDB lookup, unable to determine if enough storage is available.
- A proper fix would [implement a Postfix policy service][dms::workaround::dovecot-quotas::notes-2] that could correctly resolve aliases to valid entries in the Dovecot UserDB, querying the `quota-status` service and returning that response to Postfix.
## Sub-addressing
!!! info
[Subaddressing][wikipedia::subaddressing] (_aka **Plus Addressing** or **Address Tags**_) is a feature that allows you to receive mail to an address which includes a tag appended to the `local-part` of a valid account address.
- A subaddress has a tag delimiter (_default: `+`_), followed by the tag: `<local-part>+<tag>@<domain-part>`
- The subaddress `user+github@example.com` would deliver mail to the same mailbox as `user@example.com`.
- Tags are dynamic. Anything between the `+` and `@` is understood as the tag, no additional configuration required.
- Only the first occurence of the tag delimiter is recognized. Any additional occurences become part of the tag value itself.
??? tip "When is subaddressing useful?"
A common use-case is to use a unique tag for each service you register your email address with.
- Routing delivery to different folders in your mailbox based on the tag (_via a [Sieve filter][docs::sieve::subaddressing]_).
- Data leaks or bulk sales of email addresses.
- If spam / phishing mail you receive has not removed the tag, you will have better insight into where your address was compromised from.
- When the expected tag is missing, this additionally helps identify bad actors. Especially when mail delivery is routed to subfolders by tag.
- For more use-cases, view the end of [this article][web::subaddress-use-cases].
??? tip "Changing the tag delimiter"
Add `recipient_delimiter = +` to these config override files (_replacing `+` with your preferred delimiter_):
Follow the advice to change the tag delimiter, but instead set an empty value (`recipient_delimiter =`).
??? warning "Only for receiving, not sending"
Do not attempt to send mail from these tagged addresses, they are not equivalent to aliases.
This feature is only intended to be used when a mail client sends to a DMS managed recipient address. While DMS does not restrict the sender address you choose to send mail from (_provided `SPOOF_PROTECTION` has not been enabled_), it is often [forbidden by mail services][ms-exchange-docs::limitations].
??? abstract "Technical Details"
The configured tag delimiter (`+`) allows both Postfix and Dovecot to recognize subaddresses. Without this feature configured, the subaddresses would be considered as separate mail accounts rather than routed to a common account address.
---
Internally DMS has the tag delimiter configured by:
- Applying the Postfix `main.cf` setting: [`recipient_delimiter = +`][postfix-docs::recipient-delimiter]
- Dovecot has the equivalent setting set as `+` by default: [`recipient_delimiter = +`][dovecot-docs::config::recipient-delimiter]
## Technical Overview
!!! info
This section provides insight for understanding how Postfix and Dovecot services are involved. It is intended as a reference for maintainers and contributors.
- **Postfix** - Handles when mail is delivered (inbound) to DMS, or sent (outbound) from DMS.
- **Dovecot** - Manages access and storage for mail delivered to the DMS account mailboxes of your users.
??? abstract "Technical Details - Postfix (Inbound vs Outbound)"
Postfix needs to know how to handle inbound and outbound mail by asking these queries:
=== "Inbound"
- What mail domains is DMS responsible for handling? (_for accepting mail delivered_)
- What are valid mail addresses for those mail domains? (_reject delivery for users that don't exist_)
- Are there any aliases to redirect mail to 1 or more users, or forward to externally?
=== "Outbound"
- When `SPOOF_PROTECTION=1`, how should DMS restrict the sender address? (_eg: Users may only send mail from their associated mailbox address_)
Dovecot additionally handles authenticating user accounts for sending and retrieving mail:
- Over the ports for IMAP and POP3 connections (_110, 143, 993, 995_).
- As the default configured SASL provider, which Postfix delegates user authentication through (_for the submission(s) ports 465 & 587_). Saslauthd can be configured as an alternative SASL provider.
Dovecot splits all authentication lookups into two categories:
- A [PassDB][dovecot::docs::passdb] lookup most importantly authenticates the user. It may also provide any other necessary pre-login information.
- A [UserDB][dovecot::docs::userdb] lookup retrieves post-login information specific to a user.
[Appending a third column will customize "extra fields"][gh-issue::provisioner-file::accounts-extra-fields] when converting account data into a Dovecot UserDB entry.
DMS is not aware of these customizations beyond carrying them over, expect potential for bugs when this feature breaks any assumed conventions used in the scripts (_such as changing the mailbox path or type_).
!!! note
Account creation will normalize the provided email address to lowercase, as DMS does not support multiple case-sensitive address variants.
The email address chosen will also represent the _login username_ credential for mail clients to authenticate with.
The config format is line-based with key value pairs (**alias** --> **target address**), with white-space as a delimiter.
!!! example "`postfix-virtual.cf` config file"
In this example DMS manages mail for the domain `example.com`:
```cf-extra title="postfix-virtual.cf"
# Alias delivers to an existing account:
alias1@example.com hello@example.com
# Alias forwards to an external email address:
alias2@example.com external-account@gmail.com
```
??? warning "Known Issues"
**`setup` CLI prevents an alias and account sharing an address:**
You cannot presently add a new account (`setup email add`) or alias (`setup alias add`) with an address which already exists as an alias or account in DMS.
This [restriction was enforced][gh-issue::bugs::account-alias-overlap] due to [problems it could cause][gh-issue::bugs::account-alias-overlap-problem], although there are [use-cases where you may legitimately require this functionality][gh-issue::feature-request::allow-account-alias-overlap].
For now you must manually edit the `postfix-virtual.cf` file as a workaround. There are no run-time checks outside of the `setup` CLI related to this restriction.
---
**Wildcard catch-all support (`@example.com`):**
While this type of alias without a local-part is supported, you must keep in mind that aliases in Postfix have a higher precedence than a real address associated to a DMS account.
As a result, the wildcard is matched first and will direct mail for that entire domain to the alias target address. To work around this, [you will need an alias for each non-alias address of that domain][gh-issue::bugs::wildcard-catchall].
Additionally, Postfix will read the alias config and choose the alias value that matches the recipient address first. Ensure your more specific aliases for the domain are declared above the wildcard alias in the config file.
---
**Aliasing to another alias or multiple recipients:**
[While aliasing to multiple recipients is possible][gh-discussions::no-support::alias-multiple-targets], DMS does not officially support that.
- You may experience issues when our feature integrations don't expect more than one target per alias.
- These concerns also apply to the usage of nested aliases (_where the recipient target provided is to an alias instead of a real address_). An example is the [incompatibility with `setup alias add`][gh-issue::bugs::alias-nested].
This config file is similar to the above `postfix-virtual.cf`, but the alias value is instead configured with a regex pattern.
There is **no `setup` CLI support** for this feature, it is config only.
!!! example "`postfix-regexp.cf` config file"
Deliver all mail for `test` users to `qa@example.com` instead:
```cf-extra title="postfix-regexp.cf"
# Remember to escape regex tokens like `.` => `\.`, otherwise
# your alias pattern may be more permissive than you intended:
/^test[0-9][0-9]*@example\.com/ qa@example.com
```
??? abstract "Technical Details"
`postfix-virtual.cf` has precedence, `postfix-regexp.cf` will only be checked if no alias match was found in `postfix-virtual.cf`.
These files are both copied internally to `/etc/postfix/` and configured in `main.cf` for the `virtual_alias_maps` setting. As `postfix-virtual.cf` is declared first for that setting, it will be processed before using `postfix-regexp.cf` as a fallback.
Getting started with ldap and DMS we need to take 3 parts in account:
- `postfix` for incoming & outgoing email
- `dovecot` for accessing mailboxes
- `saslauthd` for SMTP authentication (this can also be delegated to dovecot)
## Variables to Control Provisioning by the Container
Have a look at [the ENV page][docs-environment] for information on the default values.
### `LDAP_QUERY_FILTER_*`
Those variables contain the LDAP lookup filters for postfix, using `%s` as the placeholder for the domain or email address in question. This means that...
- ...for incoming email, the domain must return an entry for the `DOMAIN` filter (see [`virtual_alias_domains`](http://www.postfix.org/postconf.5.html#virtual_alias_domains)).
- ...for incoming email, the inboxes which receive the email are chosen by the `USER`, `ALIAS` and `GROUP` filters.
- The `USER` filter specifies personal mailboxes, for which only one should exist per address, for example `(mail=%s)` (also see [`virtual_mailbox_maps`](http://www.postfix.org/postconf.5.html#virtual_mailbox_maps))
- The `ALIAS` filter specifies aliases for mailboxes, using [`virtual_alias_maps`](http://www.postfix.org/postconf.5.html#virtual_alias_maps), for example `(mailAlias=%s)`
- The `GROUP` filter specifies the personal mailboxes in a group (for emails that multiple people shall receive), using [`virtual_alias_maps`](http://www.postfix.org/postconf.5.html#virtual_alias_maps), for example `(mailGroupMember=%s)`.
- Technically, there is no difference between `ALIAS` and `GROUP`, but ideally you should use `ALIAS` for personal aliases for a singular person (like `ceo@example.org`) and `GROUP` for multiple people (like `hr@example.org`).
- ...for outgoing email, the sender address is put through the `SENDERS` filter, and only if the authenticated user is one of the returned entries, the email can be sent.
- This only applies if `SPOOF_PROTECTION=1`.
- If the `SENDERS` filter is missing, the `USER`, `ALIAS` and `GROUP` filters will be used in a disjunction (OR).
- To for example allow users from the `admin` group to spoof any sender email address, and to force everyone else to only use their personal mailbox address for outgoing email, you can use something like this: `(|(memberOf=cn=admin,*)(mail=%s))`
???+ example
A really simple `LDAP_QUERY_FILTER` configuration, using only the _user filter_ and allowing only `admin@*` to spoof any sender addresses.
```yaml
- LDAP_START_TLS=yes
- ACCOUNT_PROVISIONER=LDAP
- LDAP_SERVER_HOST=ldap.example.org
- LDAP_SEARCH_BASE=dc=example,dc=org"
- LDAP_BIND_DN=cn=admin,dc=example,dc=org
- LDAP_BIND_PW=mypassword
- SPOOF_PROTECTION=1
- LDAP_QUERY_FILTER_DOMAIN=(mail=*@%s)
- LDAP_QUERY_FILTER_USER=(mail=%s)
- LDAP_QUERY_FILTER_ALIAS=(|) # doesn't match anything
- LDAP_QUERY_FILTER_GROUP=(|) # doesn't match anything
These variables specify the LDAP filters that dovecot uses to determine if a user can log in to their IMAP account, and which mailbox is responsible to receive email for a specific postfix user.
This is split into the following two lookups, both using `%u` as the placeholder for the full login name ([see dovecot documentation for a full list of placeholders](https://doc.dovecot.org/configuration_manual/config_file/config_variables/)). Usually you only need to set `DOVECOT_USER_FILTER`, in which case it will be used for both filters.
- `DOVECOT_USER_FILTER` is used to get the account details (uid, gid, home directory, quota, ...) of a user.
- `DOVECOT_PASS_FILTER` is used to get the password information of the user, and is in pretty much all cases identical to `DOVECOT_USER_FILTER` (which is the default behavior if left away).
If your directory doesn't have the [postfix-book schema](https://github.com/variablenix/ldap-mail-schema/blob/master/postfix-book.schema) installed, then you must change the internal attribute handling for dovecot. For this you have to change the `pass_attr` and the `user_attr` mapping, as shown in the example below:
For `DOVECOT_*_ATTRS`, you can replace `ldapAttr=dovecotAttr` with `=dovecotAttr=%{ldap:ldapAttr}` for more flexibility, like for example `=home=/var/mail/%{ldap:uid}` or just `=uid=5000`.
A list of dovecot attributes can be found [in the dovecot documentation](https://doc.dovecot.org/configuration_manual/authentication/user_databases_userdb/#authentication-user-database).
The LDAP server configuration for dovecot will be taken mostly from postfix, other options can be found in [the environment section in the docs][docs-environment].
### `DOVECOT_AUTH_BIND`
Set this to `yes` to enable authentication binds ([more details in the dovecot documentation](https://wiki.dovecot.org/AuthDatabase/LDAP/AuthBinds)). Currently, only DN lookup is supported without further changes to the configuration files, so this is only useful when you want to bind as a readonly user without the permission to read passwords.
### `SASLAUTHD_LDAP_FILTER`
This filter is used for `saslauthd`, which is called by postfix when someone is authenticating through SMTP (assuming that `SASLAUTHD_MECHANISMS=ldap` is being used). Note that you'll need to set up the LDAP server for saslauthd separately from postfix.
The filter variables are explained in detail [in the `LDAP_SASLAUTHD` file](https://github.com/winlibs/cyrus-sasl/blob/master/saslauthd/LDAP_SASLAUTHD#L121), but unfortunately, this method doesn't really support domains right now - that means that `%U` is the only token that makes sense in this variable.
!!! note "When to use this and how to avoid it"
Using a separate filter for SMTP authentication allows you to for example allow `noreply@example.org` to send email, but not log in to IMAP or receive email: `(&(mail=%U@example.org)(|(memberOf=cn=email,*)(mail=noreply@example.org)))`
If you don't want to use a separate filter for SMTP authentication, you can set `SASLAUTHD_MECHANISMS=rimap` and `SASLAUTHD_MECH_OPTIONS=127.0.0.1` to authenticate against dovecot instead - this means that the `DOVECOT_USER_FILTER` and `DOVECOT_PASS_FILTER` will be used for SMTP authentication as well.
???+ example "Configure LDAP with `saslauthd`"
```yaml
- ENABLE_SASLAUTHD=1
- SASLAUTHD_MECHANISMS=ldap
- SASLAUTHD_LDAP_FILTER=(mail=%U@example.org)
```
## Secure Connection with LDAPS or StartTLS
To enable LDAPS, all you need to do is to add the protocol to `LDAP_SERVER_HOST`, for example `ldaps://example.org:636`.
To enable LDAP over StartTLS (on port 389), you need to set the following environment variables instead (the **protocol must not be `ldaps://`** in this case!):
```yaml
- LDAP_START_TLS=yes
- DOVECOT_TLS=yes
- SASLAUTHD_LDAP_START_TLS=yes
```
## Active Directory Configurations (Tested with Samba4 AD Implementation)
In addition to LDAP explanation above, when Docker Mailserver is intended to be used with Active Directory (or the equivalent implementations like Samba4 AD DC) the following points should be taken into consideration:
- Samba4 Active Directory requires a **secure connection** to the domain controller (DC), either via SSL/TLS (LDAPS) or via StartTLS.
- The username equivalent in Active Directory is: `sAMAccountName`.
- `proxyAddresses` can be used to store email aliases of single users. The convention is to prefix the email aliases with `smtp:` (e.g: `smtp:some.name@example.com`).
- Active Directory is used typically not only as LDAP Directory storage, but also as a _domain controller_, i.e., it will do many things including authenticating users. Mixing Linux and Windows clients requires the usage of [RFC2307 attributes](https://wiki.samba.org/index.php/Administer_Unix_Attributes_in_AD_using_samba-tool_and_ldb-tools), namely `uidNumber`, `gidNumber` instead of the typical `uid`. Assigning different owner to email folders can also be done in this approach, nevertheless [there is a bug at the moment in Docker Mailserver that overwrites all permissions](https://github.com/docker-mailserver/docker-mailserver/pull/2256) when starting the container. Either a manual fix is necessary now, or a temporary workaround to use a hard-coded `ldap:uidNumber` that equals to `5000` until this issue is fixed.
- To deliver the emails to different members of Active Directory **Security Group** or **Distribution Group** (similar to mailing lists), use a [`user-patches.sh` script][docs-userpatches] to modify `ldap-groups.cf` so that it includes `leaf_result_attribute = mail` and `special_result_attribute = member`. This can be achieved simply by:
The configuration shown to get the Group to work is from [here](https://doc.zarafa.com/trunk/Administrator_Manual/en-US/html/_MTAIntegration.html) and [here](https://kb.kopano.io/display/WIKI/Postfix).
- In `/etc/ldap/ldap.conf`, if the `TLS_REQCERT` is `demand` / `hard` (default), the CA certificate used to verify the LDAP server certificate must be recognized as a trusted CA. This can be done by volume mounting the `ca.crt` file and updating the trust store via a `user-patches.sh` script:
The changes on the configurations necessary to work with Active Directory (**only changes are listed, the rest of the LDAP configuration can be taken from the other examples** shown in this documentation):
```yaml
# If StartTLS is the chosen method to establish a secure connection with Active Directory.
# At the moment to be able to use %{ldap:uidNumber}, a manual bug fix as described above must be used. Otherwise %{ldap:uidNumber} %{ldap:uidNumber} must be replaced by the hard-coded value 5000.
The config format is the same as [`postfix-accounts.cf` for `ACCOUNT_PROVISIONER=FILE`][docs::account-management::file::accounts].
The only difference is the account field has no `@domain-part` suffix, it is only a username.
??? abstract "Technical Details"
[The _Master Accounts_ feature][dms::feature::dovecot-master-accounts] in DMS configures the [Dovecot Master Users][dovecot-docs::auth::master-users] feature with the Dovecot setting [`auth_master_user_separator`][dovecot-docs::config::auth-master-user-separator] (_where the default value is `*`_).
## Login via Master Account
!!! info
To login as another DMS account (`user@example.com`) with POP3 or IMAP, use the following credentials format:
This feature enables support for delegating DMS account authentication through to an external _Identity Provider_ (IdP).
!!! warning "Receiving mail requires a DMS account to exist"
If you expect DMS to receive mail, you must provision an account into DMS in advance. Otherwise DMS has no awareness of your externally manmaged users and will reject delivery.
There are [plans to implement support to provision users through a SCIM 2.0 API][dms-feature-request::scim-api]. An IdP that can operate as a SCIM Client (eg: Authentik) would then integrate with DMS for user provisioning. Until then you must keep your user accounts in sync manually via your configured [`ACCOUNT_PROVISIONER`][docs::env::account-provisioner].
??? info "How the feature works"
1. A **mail client must have support** to acquire an OAuth2 token from your IdP (_however many clients lack generic OAuth2 / OIDC provider support_).
2. The mail client then provides that token as the user password via the login mechanism `XOAUTH2` or `OAUTHBEARER`.
3. DMS (Dovecot) will then check the validity of that token against the Authentication Service it was configured with.
4. If the response returned is valid for the user account, authentication is successful.
[**XOAUTH2**][google::xoauth2-docs] (_Googles widely adopted implementation_) and **OAUTHBEARER** (_the newer variant standardized by [RFC 7628][rfc::7628] in 2015_) are supported as standards for verifying that a OAuth Bearer Token (_[RFC 6750][rfc::6750] from 2012_) is valid at the identity provider that created the token. The token itself in both cases is expected to be can an opaque _Access Token_, but it is possible to use a JWT _ID Token_ (_which encodes additional information into the token itself_).
A mail client like Thunderbird has limited OAuth2 / OIDC support. The software maintains a hard-coded list of providers supported. Roundcube is a webmail client that does have support for generic providers, allowing you to integrate with a broader range of IdP services.
---
**Documentation for this feature is WIP**
See the [initial feature support][dms-feature::oauth2-pr] and [existing issues][dms-feature::oidc-issues] for guidance that has not yet been documented officially.
??? tip "Verify authentication works"
If you have a compatible mail client you can verify login through that.
---
??? example "CLI - Verify with `curl`"
```bash
# Shell into your DMS container:
docker exec -it dms bash
# Adjust these variables for the methods below to use:
- Add `--verbose` to the curl options. This will output the protocol exchange which includes if authentication was successful or failed.
- The above example chains the `curl` commands with `grep` on DMS logs (_for Dovecot and Postfix services_). When not running `curl` from the DMS container, ensure you check the logs correctly, or inspect the `--verbose` output instead.
!!! warning "`curl` bug with `XOAUTH2`"
[Older releases of `curl` have a bug with `XOAUTH2` support][gh-issue::curl::xoauth2-bug] since `7.80.0` (Nov 2021) but fixed from `8.6.0` (Jan 2024). It treats `XOAUTH2` as `OAUTHBEARER`.
If you use `docker exec` to run `curl` from within DMS, the current DMS v14 release (_Debian 12 with curl `7.88.1`_) is affected by this bug.
## Config Examples
### Authentik with Roundcube
This example assumes you have already set up:
- A working DMS server
- An Authentik server ([documentation][authentik::docs::install])
- A Roundcube server ([docker image][roundcube::dockerhub-image] or [bare metal install][roundcube::docs::install])
!!! example "Setup Instructions"
=== "1. Docker Mailserver"
Update your Docker Compose ENV config to include:
```env title="compose.yaml"
services:
mailserver:
env:
# Enable the feature:
- ENABLE_OAUTH2=1
# Specify the user info endpoint URL of the oauth2 server for token inspection:
If your directory has not the postfix-book schema installed, then you must change the internal attribute handling for dovecot. For this you have to change the `pass_attr` and the `user_attr` mapping, as shown in the example below:
Full-text search allows all messages to be indexed, so that mail clients can quickly and efficiently search messages by their full text content.
Full-text search allows all messages to be indexed, so that mail clients can quickly and efficiently search messages by their full text content. Dovecot supports a variety of community supported [FTS indexing backends](https://doc.dovecot.org/configuration_manual/fts/).
The [dovecot-solr Plugin](https://wiki2.dovecot.org/Plugins/FTS/Solr) is used in conjunction with [Apache Solr](https://lucene.apache.org/solr/) running in a separate container. This is quite straightforward to setup using the following instructions.
DMS comes pre-installed with two plugins that can be enabled with a dovecot config file.
## Setup Steps
Please be aware that indexing consumes memory and takes up additional disk space.
1. `docker-compose.yml`:
### Xapian
```yaml
solr:
image: lmmdock/dovecot-solr:latest
volumes:
- solr-dovecot:/opt/solr/server/solr/dovecot
restart: always
The [dovecot-fts-xapian](https://github.com/grosjo/fts-xapian) plugin makes use of [Xapian](https://xapian.org/). Xapian enables embedding an FTS engine without the need for additional backends.
The indexes will be stored as a subfolder named `xapian-indexes` inside your local `mail-data` folder (_`/var/mail` internally_). With the default settings, 10GB of email data may generate around 4GB of indexed data.
While indexing is memory intensive, you can configure the plugin to limit the amount of memory consumed by the index workers. With Xapian being small and fast, this plugin is a good choice for low memory environments (2GB).
#### Setup
1. To configure `fts-xapian` as a dovecot plugin, create a file at `docker-data/dms/config/dovecot/fts-xapian-plugin.conf` and place the following in it:
3. Start the solr container: `docker-compose up -d --remove-orphans solr`
adjust the settings to tune for your desired memory limits, exclude folders and enable searching text inside of attachments
4. Restart the mailserver container: `docker-compose restart mailserver`
2. Update `compose.yaml` to load the previously created dovecot plugin config file:
5. Flag all user mailbox FTS indexes as invalid, so they are rescanned on demand when they are next searched: `docker-compose exec mailserver doveadm fts rescan -A`
docker compose exec mailserver doveadm index -A -q \*
```
5. Run the following command in a daily cron job:
```
docker compose exec mailserver doveadm fts optimize -A
```
Or like the [Spamassassin example][docs-faq-sa-learn-cron] shows, you can instead use `cron` from within DMS to avoid potential errors if the mail server is not running:
??? example
Create a _system_ cron file:
```sh
# in the compose.yaml root directory
mkdir -p ./docker-data/dms/cron # if you didn't have this folder before
touch ./docker-data/dms/cron/fts_xapian
chown root:root ./docker-data/dms/cron/fts_xapian
chmod 0644 ./docker-data/dms/cron/fts_xapian
```
Edit the system cron file `nano ./docker-data/dms/cron/fts_xapian`, and set an appropriate configuration:
```conf
# Adding `MAILTO=""` prevents cron emailing notifications of the task outcome each run
If your container host supports IPv6, then `docker-mailserver` will automatically accept IPv6 connections by way of the docker host's IPv6. However, incoming mail will fail SPF checks because they will appear to come from the IPv4 gateway that docker is using to proxy the IPv6 connection (`172.20.0.1` is the gateway).
Numerous bug reports have been raised in the past about IPv6. Please make sure your setup around DMS is correct when using IPv6!
This can be solved by supporting IPv6 connections all the way to the `docker-mailserver` container.
## IPv6 networking problems with Docker defaults
## Setup steps
### What can go wrong?
```diff
+++ b/serv/docker-compose.yml
@@ -1,4 +1,4 @@
-version: '2'
+version: '2.1'
If your host system supports IPv6 and an `AAAA` DNS record exists to direct IPv6 traffic to DMS, you may experience issues when an IPv6 connection is made:
@@ -32,6 +32,16 @@ services:
- The original client IP is replaced with the gateway IP of a docker network.
- Connections fail or hang.
+ ipv6nat:
+ image: robbertkl/ipv6nat
+ restart: always
+ network_mode: "host"
+ cap_add:
+ - NET_ADMIN
+ - SYS_MODULE
+ volumes:
+ - /var/run/docker.sock:/var/run/docker.sock:ro
+ - /lib/modules:/lib/modules:ro
The impact of losing the real IP of the client connection can negatively affect DMS:
@@ -306,4 +316,13 @@ networks:
- Users unable to login (_Fail2Ban action triggered by repeated login failures all seen as from the same internal Gateway IP_)
- Mail inbound to DMS is rejected (_[SPF verification failure][gh-issue-1438-spf], IP mismatch_)
- Delivery failures from [sender reputation][sender-score] being reduced (_due to [bouncing inbound mail][gh-issue-3057-bounce] from rejected IPv6 clients_)
- Some services may be configured to trust connecting clients within the containers subnet, which includes the Gateway IP. This can risk bypassing or relaxing security measures, such as exposing an [open relay][wikipedia-openrelay].
+ default:
+ driver: bridge
+ enable_ipv6: true
+ ipam:
+ driver: default
+ config:
+ - subnet: fd00:0123:4567::/48
+ gateway: fd00:0123:4567::1
### Why does this happen?
When the host network receives a connection to a containers published port, it is routed to the containers internal network managed by Docker (_typically a bridge network_).
By default, the Docker daemon only assigns IPv4 addresses to containers, thus it will only accept IPv4 connections (_unless a `docker-proxy` process is listening, which the default daemon setting `userland-proxy: true` enables_). With the daemon setting `userland-proxy: true` (default), IPv6 connections from the host can also be accepted and routed to containers (_even when they only have IPv4 addresses assigned_). `userland-proxy: false` will require the container to have atleast an IPv6 address assigned.
This can be problematic for IPv6 host connections when internally the container is no longer aware of the original client IPv6 address, as it has been proxied through the IPv4 or IPv6 gateway address of it's connected network (_eg: `172.17.0.1` - Docker allocates networks from a set of [default subnets][docker-subnets]_).
This can be fixed by enabling a Docker network to assign IPv6 addresses to containers, along with some additional configuration. Alternatively you could configure the opposite to prevent IPv6 connections being made.
## Prevent IPv6 connections
- Avoiding an `AAAA` DNS record for your DMS FQDN would prevent resolving an IPv6 address to connect to.
- You can also use `userland-proxy: false`, which will fail to establish a remote connection to DMS (_provided no IPv6 address was assigned_).
!!! tip "With UFW or Firewalld"
When one of these firewall frontends are active, remote clients should fail to connect instead of being masqueraded as the docker network gateway IP. Keep in mind that this only affects remote clients, it does not affect local IPv6 connections originating within the same host.
## Enable proper IPv6 support
You can enable IPv6 support in Docker for container networks, however [compatibility concerns][docs-compat] may affect your success.
The [official Docker documentation on enabling IPv6][docker-docs-enable-ipv6] has been improving and is a good resource to reference.
Enable `ip6tables` support so that Docker will manage IPv6 networking rules as well. This will allow for IPv6 NAT to work like the existing IPv4 NAT already does for your containers, avoiding the above issue with external connections having their IP address seen as the container network gateway IP (_provided an IPv6 address is also assigned to the container_).
!!! example "Configure the following in `/etc/docker/daemon.json`"
```json
{
"ip6tables": true,
"experimental" : true,
"userland-proxy": true
}
```
- `experimental: true` is currently required for `ip6tables: true` to work.
- `userland-proxy` setting [can potentially affect connection behavior][gh-pull-3244-proxy] for local connections.
Now restart the daemon if it's running: `systemctl restart docker`.
Next, configure a network with an IPv6 subnet for your container with any of these examples:
???+ example "Create an IPv6 ULA subnet"
??? info "About these examples"
These examples are focused on a [IPv6 ULA subnet][wikipedia-ipv6-ula] which is suitable for most users as described in the next section.
- You may prefer a subnet size smaller than `/64` (eg: `/112`, which still provides over 65k IPv6 addresses), especially if instead configuring for an IPv6 GUA subnet.
- The network will also implicitly be assigned an IPv4 subnet (_from the Docker daemon config `default-address-pools`_).
=== "User-defined Network"
The preferred approach is with [user-defined networks][docker-docs-ipv6-create-custom] via `compose.yaml` (recommended) or CLI with `docker network create`:
=== "Compose"
Create the network in `compose.yaml` and attach a service to it:
```yaml title="compose.yaml"
services:
mailserver:
networks:
- dms-ipv6
networks:
dms-ipv6:
enable_ipv6: true
ipam:
config:
- subnet: fd00:cafe:face:feed::/64
```
??? tip "Override the implicit `default` network"
You can optionally avoid the service assignment by [overriding the `default` user-defined network that Docker Compose generates][docker-docs-network-compose-default]. Just replace `dms-ipv6` with `default`.
The Docker Compose `default` bridge is not affected by settings for the default `bridge` (aka `docker0`) in `/etc/docker/daemon.json`.
??? tip "Using the network outside of this `compose.yaml`"
To reference this network externally (_from other compose files or `docker run`_), assign the [networks `name` key in `compose.yaml`][docker-docs-network-external].
=== "CLI"
Create the network via a CLI command (_which can then be used with `docker run --network dms-ipv6`_):
Optionally reference it from one or more `compose.yaml` files:
```yaml title="compose.yaml"
services:
mailserver:
networks:
- dms-ipv6
networks:
dms-ipv6:
external: true
```
=== "Default Bridge (daemon)"
!!! warning "This approach is discouraged"
The [`bridge` network is considered legacy][docker-docs-network-bridge-legacy].
Add these two extra IPv6 settings to your daemon config. They only apply to the [default `bridge` docker network][docker-docs-ipv6-create-default] aka `docker0` (_which containers are attached to by default when using `docker run`_).
```json title="/etc/docker/daemon.json"
{
"ipv6": true,
"fixed-cidr-v6": "fd00:cafe:face:feed::/64",
}
```
Compose projects can also use this network via `network_mode`:
```yaml title="compose.yaml"
services:
mailserver:
network_mode: bridge
```
!!! danger "Do not use `2001:db8:1::/64` for your private subnet"
The `2001:db8` address prefix is [reserved for documentation][wikipedia-ipv6-reserved]. Avoid creating a subnet with this prefix.
Presently this is used in examples for Dockers IPv6 docs as a placeholder, while mixed in with private IPv4 addresses which can be misleading.
### Configuring an IPv6 subnet
If you've [configured IPv6 address pools in `/etc/docker/daemon.json`][docker-docs-ipv6-supernets], you do not need to specify a subnet explicitly. Otherwise if you're unsure what value to provide, here's a quick guide (_Tip: Prefer IPv6 ULA, it's the least hassle_):
- `fd00:cafe:face:feed::/64` is an example of a [IPv6 ULA subnet][wikipedia-ipv6-ula]. ULA addresses are akin to the [private IPv4 subnets][wikipedia-ipv4-private] you may already be familiar with. You can use that example, or choose your own ULA address. This is a good choice for getting Docker containers to their have networks support IPv6 via NAT like they already do by default with IPv4.
- IPv6 without NAT, using public address space like your server is assigned belongs to an [IPv6 GUA subnet][wikipedia-ipv6-gua].
- Typically these will be a `/64` block assigned to your host, but this varies by provider.
- These addresses do not need to publish ports of a container to another IP to be publicly reached (_thus `ip6tables: true` is not required_), you will want a firewall configured to manage which ports are accessible instead as no NAT is involved. Note that this may not be desired if the container should also be reachable via the host IPv4 public address.
- You may want to subdivide the `/64` into smaller subnets for Docker to use only portions of the `/64`. This can reduce some routing features, and [require additional setup / management via a NDP Proxy][gh-pull-3244-gua] for your public interface to know of IPv6 assignments managed by Docker and accept external traffic.
### Verify remote IP is correct
With Docker CLI or Docker Compose, run a `traefik/whoami` container with your IPv6 docker network and port 80 published. You can then send a curl request (or via address in the browser) from another host (as your remote client) with an IPv6 network, the `RemoteAddr` value returned should match your client IPv6 address.
```bash
docker run --rm -d --network dms-ipv6 -p 80:80 traefik/whoami
# On a different host, replace `2001:db8::1` with your DMS host IPv6 address
curl --max-time 5 http://[2001:db8::1]:80
```
## Further Discussion
!!! warning "IPv6 gateway IP"
See [#1438][github-issue-1438]
If instead of the remote IPv6 address, you may notice the gateway IP for the IPv6 subnet your DMS container belongs to.
This will happen when DMS has an IPv6 IP address assigned, for the same reason as with IPv4, `userland-proxy: true`. It indicates that your `daemon.json` has not been configured correctly or had the updated config applied for `ip6tables :true` + `experimental: true`. Make sure you used `systemctl restart docker` after updating `daemon.json`.
!!! info "IPv6 ULA address priority"
DNS lookups that have records for both IPv4 and IPv6 addresses (_eg: `localhost`_) may prefer IPv4 over IPv6 (ULA) for private addresses, whereas for public addresses IPv6 has priority. This shouldn't be anything to worry about, but can come across as a surprise when testing your IPv6 setup on the same host instead of from a remote client.
The preference can be controlled with [`/etc/gai.conf`][networking-gai], and appears was configured this way based on [the assumption that IPv6 ULA would never be used with NAT][networking-gai-blog]. It should only affect the destination resolved for outgoing connections, which for IPv6 ULA should only really affect connections between your containers / host. In future [IPv6 ULA may also be prioritized][networking-gai-rfc].
title: 'Advanced | Email Gathering with Fetchmail'
---
To enable the [fetchmail][fetchmail-website] service to retrieve e-mails set the environment variable `ENABLE_FETCHMAIL` to `1`. Your `docker-compose.yml` file should look like following snippet:
To enable the [fetchmail][fetchmail-website] service to retrieve e-mails, set the environment variable `ENABLE_FETCHMAIL` to `1`. Your `compose.yaml` file should look like following snippet:
```yaml
environment:
@ -10,115 +10,143 @@ environment:
- FETCHMAIL_POLL=300
```
Generate a file called `fetchmail.cf` and place it in the `config` folder. Your `docker-mailserver` folder should look like this example:
Generate a file called `fetchmail.cf` and place it in the `docker-data/dms/config/` folder. Your DMS folder should look like this example:
```txt
├── config
├── docker-data/dms/config
│ ├── dovecot.cf
│ ├── fetchmail.cf
│ ├── postfix-accounts.cf
│ └── postfix-virtual.cf
├── docker-compose.yml
└── README.md
└── compose.yaml
```
## Configuration
A detailed description of the configuration options can be found in the [online version of the manual page][fetchmail-docs].
Configuration options for `fetchmail.cf` are covered at the [official fetchmail docs][fetchmail-docs-config] (_see the section "The run control file" and the table with "keyword" column for all settings_).
### IMAP Configuration
!!! example "Basic `fetchmail.cf` configuration"
!!! example
Retrieve mail from `remote-user@somewhere.com` and deliver it to `dms-user@example.com`:
```fetchmailrc
poll 'imap.example.com' proto imap
user 'username'
pass 'secret'
is 'user1@domain.tld'
ssl
poll 'mail.somewhere.com'
proto imap
user 'remote-user'
pass 'secret'
is 'dms-user@example.com'
```
### POP3 Configuration
- `poll` sets the remote mail server to connect to retrieve mail from.
- `proto` lets you connect via IMAP or POP3.
- `user` and `pass` provide the login credentials for the remote mail service account to access.
- `is` configures where the fetched mail will be sent to (_eg: your local DMS account in `docker-data/dms/config/postfix-accounts.cf`_).
!!! example
---
```fetchmailrc
poll 'pop3.example.com' proto pop3
user 'username'
pass 'secret'
is 'user2@domain.tld'
ssl
??? warning "`proto imap` will still delete remote mail once fetched"
This is due to a separate default setting `no keep`. Adding the setting `keep` to your config on a new line will prevent deleting the remote copy.
??? example "Multiple users or remote servers"
The official docs [config examples][fetchmail-config-examples] show a common convention to indent settings on subsequent lines for visually grouping per server.
=== "Minimal syntax"
```fetchmailrc
poll 'mail.somewhere.com' proto imap
user 'john.doe' pass 'secret' is 'johnny@example.com'
user 'jane.doe' pass 'secret' is 'jane@example.com'
poll 'mail.somewhere-else.com' proto pop3
user 'john.doe@somewhere-else.com' pass 'secret' is 'johnny@example.com'
```
=== "With optional syntax"
- `#` for adding comments.
- The config file may include "noise" keywords to improve readability.
```fetchmailrc
# Retrieve mail for users `john.doe` and `jane.doe` via IMAP at this remote mail server:
poll 'mail.somewhere.com' with proto imap wants:
user 'john.doe' with pass 'secret', is 'johnny@example.com' here
user 'jane.doe' with pass 'secret', is 'jane@example.com' here
# Also retrieve mail from this mail server (but via POP3).
# NOTE: This could also be all on a single line, or with each key + value as a separate line.
# Notice how the remote username includes a full email address,
# Some mail servers like DMS use the full email address as the username:
poll 'mail.somewhere-else.com' with proto pop3 wants:
user 'john.doe@somewhere-else.com' with pass 'secret', is 'johnny@example.com' here
```
!!! tip "`FETCHMAIL_POLL` ENV: Override default polling interval"
By default the fetchmail service will check every 5 minutes for new mail at the configured mail accounts.
```yaml
environment:
# The fetchmail polling interval in seconds:
FETCHMAIL_POLL: 60
```
!!! caution
Don’t forget the last line: eg: `is 'user1@domain.tld'`. After `is` you have to specify one email address from the configuration file `config/postfix-accounts.cf`.
More details how to configure fetchmail can be found in the [fetchmail man page in the chapter “The run control file”][fetchmail-docs-run].
### Polling Interval
By default the fetchmail service searches every 5 minutes for new mails on your external mail accounts. You can override this default value by changing the ENV variable `FETCHMAIL_POLL`:
```yaml
environment:
- FETCHMAIL_POLL=60
```
You must specify a numeric argument which is a polling interval in seconds. The example above polls every minute for new mails.
## Debugging
To debug your `fetchmail.cf` configuration run this command:
To debug your `fetchmail.cf` configuration run this `setup debug` command:
fetchmail: POP3< +OK The Microsoft Exchange POP3 service is ready. [SABFADEAUABSADAAMQBDAEEAMAAwADAANwAuAGUAdQByAHAAcgBkADAAMQAuAHAAcgBvAGQALgBlAHgAYwBoAGEAbgBnAGUAbABhAGIAcwAuAGMAbwBtAA==]
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3<TOP
fetchmail: POP3<UIDL
fetchmail: POP3<SASLPLAIN
fetchmail: POP3<USER
fetchmail: POP3<.
fetchmail: POP3> USER user1@outlook.com
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK User successfully logged on.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for user1@outlook.com at outlook.office365.com
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Microsoft Exchange Server 2016 POP3 server signing off.
fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:11 2016: poll completed
fetchmail: normal termination, status 1
```
```log
fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:09 2016: poll started
Trying to connect to 132.245.48.18/995...connected.
fetchmail: Server certificate:
fetchmail: Issuer Organization: Microsoft Corporation
fetchmail: Issuer CommonName: Microsoft IT SSL SHA2
fetchmail: Subject CommonName: outlook.com
fetchmail: Subject Alternative Name: outlook.com
fetchmail: Subject Alternative Name: *.outlook.com
fetchmail: Subject Alternative Name: office365.com
fetchmail: Subject Alternative Name: *.office365.com
fetchmail: Subject Alternative Name: *.live.com
fetchmail: Subject Alternative Name: *.internal.outlook.com
fetchmail: Subject Alternative Name: *.outlook.office365.com
fetchmail: Subject Alternative Name: outlook.office.com
fetchmail: Subject Alternative Name: attachment.outlook.office.net
fetchmail: Subject Alternative Name: attachment.outlook.officeppe.net
fetchmail: POP3< +OK The Microsoft Exchange POP3 service is ready. [SABFADEAUABSADAAMQBDAEEAMAAwADAANwAuAGUAdQByAHAAcgBkADAAMQAuAHAAcgBvAGQALgBlAHgAYwBoAGEAbgBnAGUAbABhAGIAcwAuAGMAbwBtAA==]
fetchmail: POP3> CAPA
fetchmail: POP3< +OK
fetchmail: POP3<TOP
fetchmail: POP3<UIDL
fetchmail: POP3<SASLPLAIN
fetchmail: POP3<USER
fetchmail: POP3<.
fetchmail: POP3> USER user1@outlook.com
fetchmail: POP3< +OK
fetchmail: POP3> PASS *
fetchmail: POP3< +OK User successfully logged on.
fetchmail: POP3> STAT
fetchmail: POP3< +OK 0 0
fetchmail: No mail for user1@outlook.com at outlook.office365.com
fetchmail: POP3> QUIT
fetchmail: POP3< +OK Microsoft Exchange Server 2016 POP3 server signing off.
fetchmail: 6.3.26 querying outlook.office365.com (protocol POP3) at Mon Aug 29 22:11:11 2016: poll completed
fetchmail: normal termination, status 1
```
!!! tip "Troubleshoot with this reference `compose.yaml`"
[A minimal `compose.yaml` example][fetchmail-compose-example] demonstrates how to run two instances of DMS locally, with one instance configured with `fetchmail.cf` and the other to simulate a remote mail server to fetch from.
New configuration, see [Configure Relay Hosts][docs-relay]
[Amazon SES (Simple Email Service)][aws-ses] provides a simple way for cloud based applications to send and receive email.
Instead of letting postfix deliver mail directly it is possible to configure it to deliver outgoing email via Amazon SES (Simple Email Service). (Receiving inbound email via SES is not implemented.) The configuration follows the guidelines provided by AWS in https://docs.aws.amazon.com/ses/latest/DeveloperGuide/postfix.html, specifically, the `STARTTLS` method.
!!! example "Configuration via ENV"
As described in the AWS Developer Guide you will have to generate SMTP credentials and define the following two environment variables in the docker-compose.yml with the appropriate values for your AWS SES subscription (the values for `AWS_SES_USERPASS` are the "SMTP username" and "SMTP password" provided when you create SMTP credentials for SES):
[Configure a relay host in DMS][docs::relay] to forward all your mail through AWS SES:
When you start the container you will see a log line as follows confirming the configuration:
RELAY_USER=aws-user
RELAY_PASSWORD=secret
```
```log
Setting up outgoing email via AWS SES host email-smtp.us-east-1.amazonaws.com
```
!!! tip
To verify proper operation, send an email to some external account of yours and inspect the mail headers. You will also see the connection to SES in the mail logs. For example:
If you have set up [AWS Easy DKIM][aws-ses::easy-dkim], you can safely skip setting up DKIM as AWS SES will take care of signing your outbound mail.
```log
May 23 07:09:36 mail postfix/smtp[692]: Trusted TLS connection established to email-smtp.us-east-1.amazonaws.com[107.20.142.169]:25:
TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
May 23 07:09:36 mail postfix/smtp[692]: 8C82A7E7: to=<someone@example.com>, relay=email-smtp.us-east-1.amazonaws.com[107.20.142.169]:25,
delay=0.35, delays=0/0.02/0.13/0.2, dsn=2.0.0, status=sent (250 Ok 01000154dc729264-93fdd7ea-f039-43d6-91ed-653e8547867c-000000)
```
!!! note "Verify the relay host is configured correctly"
[docs-relay]: ./relay-hosts.md
To verify proper operation, send an email to some external account of yours and inspect the mail headers.
You will also see the connection to SES in the mail logs:
```log
postfix/smtp[692]: Trusted TLS connection established to email-smtp.us-west-1.amazonaws.com[107.20.142.169]:25:
TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
title: 'Mail Forwarding | Configure Gmail as a relay host'
---
This page provides a guide for configuring DMS to use [GMAIL as an SMTP relay host][gmail-smtp].
!!! example "Configuration via ENV"
[Configure a relay host in DMS][docs::relay]. This example shows how the related ENV settings map to the Gmail service config:
- `RELAY_HOST` should be configured as [advised by Gmail][gmail-smtp::relay-host], there are two SMTP endpoints to choose:
- `smtp.gmail.com` (_for a personal Gmail account_)
- `smtp-relay.gmail.com` (_when using Google Workspace_)
- `RELAY_PORT` should be set to [one of the supported Gmail SMTP ports][gmail-smtp::relay-port] (_eg: 587 for STARTTLS_).
- `RELAY_USER` should be your gmail address (`user@gmail.com`).
- `RELAY_PASSWORD` should be your [App Password][gmail-smtp::app-password], **not** your personal gmail account password.
```env
RELAY_HOST=smtp.gmail.com
RELAY_PORT=587
# Alternative to RELAY_HOST + RELAY_PORT which is compatible with LDAP:
DEFAULT_RELAY_HOST=[smtp.gmail.com]:587
RELAY_USER=username@gmail.com
RELAY_PASSWORD=secret
```
!!! tip
- As per our main [relay host docs page][docs::relay], you may prefer to configure your credentials via `setup relay add-auth` instead of the `RELAY_USER` + `RELAY_PASSWORD` ENV.
- If you configure for `smtp-relay.gmail.com`, the `DEFAULT_RELAY_HOST` ENV should be all you need as shown in the above example. Credentials can be optional when using Google Workspace (`smtp-relay.gmail.com`), which supports restricting connections to trusted IP addresses.
!!! note "Verify the relay host is configured correctly"
To verify proper operation, send an email to an external account of yours and inspect the mail headers.
You will also see the connection to the Gmail relay host (`smtp.gmail.com`) in the mail logs:
```log
postfix/smtp[910]: Trusted TLS connection established to smtp.gmail.com[64.233.188.109]:587:
TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
Rather than having Postfix deliver mail directly, you can configure Postfix to send mail via another mail relay (smarthost). Examples include [Mailgun](https://www.mailgun.com/), [Sendgrid](https://sendgrid.com/) and [AWS SES](https://aws.amazon.com/ses/).
An SMTP relay service (_aka relay host / [smarthost][wikipedia::smarthost]_) is an MTA that relays (_forwards_) mail on behalf of third-parties (_it does not manage the mail domains_).
Depending on the domain of the sender, you may want to send via a different relay, or authenticate in a different way.
- Instead of DMS handling SMTP delivery directly itself (_via Postfix_), it can be configured to delegate delivery by sending all outbound mail through a relay service.
- Examples of popular mail relay services: [AWS SES][smarthost::aws-ses], [Mailgun][smarthost::mailgun], [Mailjet][smarthost::mailjet], [SendGrid][smarthost::sendgrid]
## Basic Configuration
!!! info "When can a relay service can be helpful?"
Basic configuration is done via environment variables:
- Your network provider has blocked outbound connections on port 25 (_required for direct delivery_).
- To improve delivery success via better established reputation (trust) of a relay service.
* `RELAY_HOST`: _default host to relay mail through, empty will disable this feature_
* `RELAY_PORT`: _port on default relay, defaults to port 25_
* `RELAY_USER`: _username for the default relay_
* `RELAY_PASSWORD`: _password for the default user_
## Configuration
Setting these environment variables will cause mail for all sender domains to be routed via the specified host, authenticating with the user/password combination.
All mail sent outbound from DMS (_where the sender address is a DMS account or a virtual alias_) will be relayed through the configured relay host.
!!! warning
For users of the previous `AWS_SES_*` variables: please update your configuration to use these new variables, no other configuration is required.
!!! info "Configuration via ENV"
## Advanced Configuration
Configure the default relayhost with either of these ENV:
- `RELAY_USER` + `RELAY_PASSWORD` provides credentials for authenticating with the default relayhost.
An example configuration file looks like this:
!!! warning "Providing secrets via ENV"
```txt
@domain1.com relay_user_1:password_1
@domain2.com relay_user_2:password_2
```
While ENV is convenient, the risk of exposing secrets is higher.
If there is no other configuration, this will cause Postfix to deliver email throught the relay specified in `RELAY_HOST` env variable, authenticating as `relay_user_1` when sent from `domain1.com` and authenticating as `relay_user_2` when sending from domain2.com.
`setup relay add-auth` is a better alternative, which manages the credentials via a config file.
!!! note
To activate the configuration you must either restart the container, or you can also trigger an update by modifying a mail account.
??? tip "Excluding specific sender domains from relay"
### Sender-dependent Relay Host
You can opt-out with: `setup relay exclude-domain <domain>`
Sender dependent relay hosts are configured in `config/postfix-relaymap.cf`. You can create this file manually, or use:
Outbound mail from senders of that domain will be sent normally (_instead of through the configured `RELAY_HOST`_).
```sh
setup.sh relay add-domain <domain><host> [<port>]
```
!!! warning "When any relay host credentials are configured"
An example configuration file looks like this:
It will still be expected that mail is sent over a secure connection with credentials provided.
```txt
@domain1.com [relay1.org]:587
@domain2.com [relay2.org]:2525
```
Thus this opt-out feature is rarely practical.
Combined with the previous configuration in `config/postfix-sasl-password.cf`, this will cause Postfix to deliver mail sent from domain1.com via `relay1.org:587`, authenticating as `relay_user_1`, and mail sent from domain2.com via `relay2.org:2525` authenticating as `relay_user_2`.
### Advanced Configuration
!!! note
You still have to define `RELAY_HOST` to activate the feature
When mail is sent, there is support to change the relay service or the credentials configured based on the sender address domain used.
### Excluding Sender Domains
We provide this support via two config files:
If you want mail sent from some domains to be delivered directly, you can exclude them from being delivered via the default relay by adding them to `config/postfix-relaymap.cf` with no destination. You can also do this via:
# Optionally avoid relaying from senders of this domain:
# NOTE: Only supported when configured with the `RELAY_HOST` ENV!
setup relay exclude-domain <domain>
```
Thanks to the author of [this article][1] for the inspiration. This is also worth reading to understand a bit more about how to set up Mailgun to work with this.
!!! example "Config file: `postfix-sasl-password.cf`"
When Postfix sends mail outbound from these sender domains, the above config will:
- Relay mail through `[smtp.mailgun.org]:587` when mail is sent from a sender of `@domain1.com`
- Relay mail through `[smtp.sendgrid.net]:2525` when mail is sent from a sender of `@domain1.com`
- Mail with a sender from `@domain3.com` is not sent through a relay (_**Only applicable** when using `RELAY_HOST`_)
### Technical Details
- Both the supported ENV and config files for this feature have additional details covered in our ENV docs [Relay Host section][docs::env-relay].
- For troubleshooting, a [minimal `compose.yaml` config with several DMS instances][dms-gh::relay-example] demonstrates this feature for local testing.
- [Subscribe to this tracking issue][dms-gh::pr-3607] for future improvements intended for this feature.
!!! abstract "Postfix Settings"
Internally this feature is implemented in DMS by [`relay.sh`][dms-repo::helpers-relay].
The `relay.sh` script manages configuring these Postfix settings:
```cf-extra
# Send all outbound mail through this relay service:
To enable the [getmail][getmail-website] service to retrieve e-mails set the environment variable `ENABLE_GETMAIL` to `1`. Your `compose.yaml` file should include the following:
```yaml
environment:
- ENABLE_GETMAIL=1
- GETMAIL_POLL=5
```
In your DMS config volume (eg: `docker-data/dms/config/`), add a subdirectory `getmail/` for including your getmail config files (eg: `imap-example.cf`) for each remote account that you want to retrieve mail from and deliver to the mailbox of a DMS account.
The content of these config files is documented in the next section with an IMAP and POP3 example to reference.
The directory structure should look similar to this:
```txt
├── docker-data/dms/config
│ ├── dovecot.cf
│ ├── getmail
│ │ ├── getmailrc_general.cf
│ │ ├── remote-account1.cf
│ │ ├── remote-account2.cf
│ ├── postfix-accounts.cf
│ └── postfix-virtual.cf
├── docker-compose.yml
└── README.md
```
## Configuration
A detailed description of the configuration options can be found in the [online version of the manual page][getmail-docs].
### Common Options
The default options added to each `getmail` config are:
```getmailrc
[options]
verbose = 0
read_all = false
delete = false
max_messages_per_session = 500
received = false
delivered_to = false
```
The DMS integration for Getmail generates a `getmailrc` config that prepends the common options of the base config to each remote account config file (`*.cf`) found in the DMS Config Volume `getmail/` directory.
!!! tip "Change the base options"
Add your own base config as `getmail/getmailrc_general.cf` into the DMS Config Volume. It will replace the DMS defaults shown above.
??? example "IMAP Configuration"
This example will:
1. Connect to the remote IMAP server from Gmail.
2. Retrieve mail from the gmail account `alice` with password `notsecure`.
3. Store any mail retrieved from the remote mail-server into DMS for the `user1@example.com` account that DMS manages.
```getmailrc
[retriever]
type = SimpleIMAPSSLRetriever
server = imap.gmail.com
username = alice
password = notsecure
[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
allow_root_commands = true
arguments =("-d","user1@example.com")
```
??? example "POP3 Configuration"
Just like the IMAP example above, but instead via POP3 protocol if you prefer that over IMAP.
```getmailrc
[retriever]
type = SimplePOP3SSLRetriever
server = pop3.gmail.com
username = alice
password = notsecure
[destination]
type = MDA_external
path = /usr/lib/dovecot/deliver
allow_root_commands = true
arguments =("-d","user1@example.com")
```
### Polling Interval
By default the `getmail` service checks external mail accounts for new mail every 5 minutes. That polling interval is configurable via the `GETMAIL_POLL` ENV variable, with a value in minutes (_default: 5, min: 1_):
```yaml
environment:
- GETMAIL_POLL=1
```
### XOAUTH2 Authentication
It is possible to utilize the `getmail-gmail-xoauth-tokens` helper to provide authentication using `xoauth2` for [gmail (example 12)][getmail-docs-xoauth-12] or [Microsoft Office 365 (example 13)][getmail-docs-xoauth-13]
[Sieve](http://sieve.info/) allows to specify filtering rules for incoming emails that allow for example sorting mails into different folders depending on the title of an email.
There are global and user specific filters which are filtering the incoming emails in the following order:
!!! warning "Advice may be outdated"
- Global-before -> User specific -> Global-after
This section was contributed by the community some time ago and some configuration examples may be outdated.
Global filters are applied to EVERY incoming mail for EVERY email address.
To specify a global Sieve filter provide a `config/before.dovecot.sieve` or a `config/after.dovecot.sieve` file with your filter rules.
If any filter in this filtering chain discards an incoming mail, the delivery process will stop as well and the mail will not reach any following filters(e.g. global-before stops an incoming spam mail: The mail will get discarded and a user-specific filter won't get applied.)
[Sieve][sieve-info] allows to specify filtering rules for incoming emails that allow for example sorting mails into different folders depending on the title of an email.
To specify a user-defined Sieve filter place a `.dovecot.sieve` file into a virtual user's mail folder e.g. `/var/mail/domain.com/user1/.dovecot.sieve`. If this file exists dovecot will apply the filtering rules.
!!! info "Global vs User order"
It's even possible to install a user provided Sieve filter at startup during users setup: simply include a Sieve file in the `config` path for each user login that need a filter. The file name provided should be in the form `<user_login>.dovecot.sieve`, so for example for `user1@domain.tld` you should provide a Sieve file named `config/user1@domain.tld.dovecot.sieve`.
There are global and user specific filters which are filtering the incoming emails in the following order:
Global-before -> User specific -> Global-after
Global filters are applied to EVERY incoming mail for EVERY email address.
- To specify a global Sieve filter provide a `docker-data/dms/config/before.dovecot.sieve` or a `docker-data/dms/config/after.dovecot.sieve` file with your filter rules.
- If any filter in this filtering chain discards an incoming mail, the delivery process will stop as well and the mail will not reach any following filters (e.g. global-before stops an incoming spam mail: The mail will get discarded and a user-specific filter won't get applied.)
To specify a user-defined Sieve filter place a `.dovecot.sieve` file into a virtual user's mail folder (e.g. `/var/mail/example.com/user1/home/.dovecot.sieve`). If this file exists dovecot will apply the filtering rules.
It's even possible to install a user provided Sieve filter at startup during users setup: simply include a Sieve file in the `docker-data/dms/config/` path for each user login that needs a filter. The file name provided should be in the form `<user_login>.dovecot.sieve`, so for example for `user1@example.com` you should provide a Sieve file named `docker-data/dms/config/user1@example.com.dovecot.sieve`.
An example of a sieve filter that moves mails to a folder `INBOX/spam` depending on the sender address:
@ -32,16 +40,17 @@ An example of a sieve filter that moves mails to a folder `INBOX/spam` depending
```
!!! warning
That folders have to exist beforehand if sieve should move them.
Another example of a sieve filter that forward mails to a different address:
!!! example
!!! example
```sieve
require ["copy"];
redirect :copy "user2@otherdomain.tld";
redirect :copy "user2@not-example.com";
```
Just forward all incoming emails and do not save them locally:
@ -49,17 +58,99 @@ Just forward all incoming emails and do not save them locally:
!!! example
```sieve
redirect "user2@otherdomain.tld";
redirect "user2@not-example.com";
```
You can also use external programs to filter or pipe (process) messages by adding executable scripts in `config/sieve-pipe` or `config/sieve-filter`. This can be used in lieu of a local alias file, for instance to forward an email to a webservice. These programs can then be referenced by filename, by all users. Note that the process running the scripts run as a privileged user. For further information see [Dovecot's wiki](https://wiki.dovecot.org/Pigeonhole/Sieve/Plugins/Pipe).
You can also use external programs to filter or pipe (process) messages by adding executable scripts in `docker-data/dms/config/sieve-pipe` or `docker-data/dms/config/sieve-filter`.
This can be used in lieu of a local alias file, for instance to forward an email to a webservice.
- These programs can then be referenced by filename, by all users.
- Note that the process running the scripts run as a privileged user.
- For further information see [Dovecot's docs][dovecot-docs::sieve-pipe].
```sieve
require ["vnd.dovecot.pipe"];
pipe "external-program";
```
For more examples or a detailed description of the Sieve language have a look at [the official site](http://sieve.info/examplescripts). Other resources are available on the internet where you can find several [examples](https://support.tigertech.net/sieve#sieve-example-rules-jmp).
For more examples or a detailed description of the Sieve language have a look at [the official site][sieve-info::examples]. Other resources are available on the internet where you can find several [examples][third-party::sieve-examples].
## Automatic Sorting Based on Sub-addresses { #subaddress-mailbox-routing }
When mail is delivered to your account, it is possible to organize storing mail into folders by the [subaddress (tag)][docs::accounts-subaddressing] used.
!!! example "Example: `user+<tag>@example.com` to `INBOX/<Tag>`"
This example sorts mail into inbox folders by their tag:
# Similar to `:matches`, except `:regex` provides enhanced pattern matching.
# NOTE: This example needs you to `require` the "regex" extension
if envelope :detail :regex "to" "^cloud-(azure|aws)$" {
# Normalize the captured azure/aws tag as the resolved value is no longer fixed:
set :lower :upperfirst "vendor" "${1}";
# If a `.` exists in the tag, it will create nested folders:
set "tag" "Cloud.${vendor}";
```
**NOTE:** There is no need to lowercase the tag in the conditional as the [`to` value is a case-insensitive check][sieve-docs::envelope].
??? abstract "Technical Details"
- Dovecot supports this feature via the _Sieve subaddress extension_ ([RFC 5233][rfc::5233::sieve-subaddress]).
- Only a single tag per subaddress is supported. Any additional tag delimiters are part of the tag value itself.
- The Dovecot setting [`recipient_delimiter`][dovecot-docs::config::recipient_delimiter] (default: `+`) configures the tag delimiter. This is where the `local-part` of the recipient address will split at, providing the `:detail` (tag) value for Sieve.
---
`INBOX` is the [default namespace configured by Dovecot][dovecot-docs::namespace].
- If you omit the `INBOX.` prefix from the sieve script above, the mailbox (folder) for that tag is created at the top-level alongside your Trash and Junk folders.
- The `.` between `INBOX` and `${tag}` is important as a [separator to distinguish mailbox names][dovecot-docs::mailbox-names]. This can vary by mailbox format or configuration. DMS uses [`Maildir`][dovecot-docs::mailbox-formats::maildir] by default, which uses `.` as the separator.
- [`lmtp_save_to_detail_mailbox = yes`][dovecot-docs::config::lmtp_save_to_detail_mailbox] can be set in `/etc/dovecot/conf.d/20-lmtp.conf`:
- This implements the feature globally, except for the tag normalization and `INBOX.` prefix parts of the example script.
- However, if the sieve script is also present, the script has precedence and will handle this task instead when the condition is successful, otherwise falling back to the global feature.
## Manage Sieve
@ -67,19 +158,31 @@ The [Manage Sieve](https://doc.dovecot.org/admin_manual/pigeonhole_managesieve_s
!!! example
```yaml
# docker-compose.yml
```yaml title="compose.yaml"
ports:
- "4190:4190"
environment:
- ENABLE_MANAGESIEVE=1
```
All user defined sieve scripts that are managed by ManageSieve are stored in the user's home folder in `/var/mail/domain.com/user1/sieve`. Just one sieve script might be active for a user and is sym-linked to `/var/mail/domain.com/user1/.dovecot.sieve` automatically.
All user defined sieve scripts that are managed by ManageSieve are stored in the user's home folder in `/var/mail/example.com/user1/home/sieve`. Just one Sieve script might be active for a user and is sym-linked to `/var/mail/example.com/user1/home/.dovecot.sieve` automatically.
!!! note
ManageSieve makes sure to not overwrite an existing `.dovecot.sieve` file. If a user activates a new sieve script the old one is backuped and moved to the `sieve` folder.
ManageSieve makes sure to not overwrite an existing `.dovecot.sieve` file. If a user activates a new sieve script the old one is backed up and moved to the `sieve` folder.
The extension is known to work with the following ManageSieve clients:
- **Sieve Editor** a portable standalone application based on the former Thunderbird plugin (https://github.com/thsmi/sieve).
- **[Sieve Editor](https://github.com/thsmi/sieve)** a portable standalone application based on the former Thunderbird plugin.
- **[Kmail](https://kontact.kde.org/components/kmail/)** the mail client of [KDE](https://kde.org/)'s Kontact Suite.
[`containrrr/watchtower`][watchtower-dockerhub] is a service that monitors Docker images for updates, automatically applying them to running containers.
Docker images are handy but it can get a a hassle to keep them updated. Also when a repository is automated you want to get these images when they get out.
!!! example "Automatic image updates + cleanup"
One could setup a complex action/hook-based workflow using probes, but there is a nice, easy to use docker image that solves this issue and could prove useful: [`watchtower`](https://hub.docker.com/r/containrrr/watchtower).
Run a `watchtower` container with access to `docker.sock`, enabling the service to manage Docker:
A docker-compose example:
```yaml title="compose.yaml"
services:
watchtower:
image: containrrr/watchtower:latest
# Automatic cleanup (removes older image pulls from wasting disk space):
environment:
- WATCHTOWER_CLEANUP=true
volumes:
- /var/run/docker.sock:/var/run/docker.sock
```
```yaml
services:
watchtower:
restart: always
image: containrrr/watchtower:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
```
!!! tip "The image tag used for a container is monitored for updates (eg: `:latest`, `:edge`, `:13`)"
For more details, see the [manual](https://containrrr.github.io/watchtower/)
The automatic update support is **only for updates to that specific image tag**.
## Automatic Cleanup
- Your container will not update to a new major version tag (_unless using `:latest`_).
- Omit the minor or patch portion of the semver tag to receive updates for the omitted portion (_eg: `13` will represent the latest minor + patch release of `v13`_).
When you are pulling new images in automatically, it would be nice to have them cleaned up as well. There is also a docker image for this: [`spotify/docker-gc`](https://hub.docker.com/r/spotify/docker-gc/).
!!! tip "Updating only specific containers"
A docker-compose example:
By default the `watchtower` service will check every 24 hours for new image updates to pull, based on currently running containers (_**not restricted** to only those running within your `compose.yaml`_).
```yaml
services:
docker-gc:
restart: always
image: spotify/docker-gc:latest
volumes:
- /var/run/docker.sock:/var/run/docker.sock
```
Images eligible for updates can configured with a [custom `command`][docker-docs-compose-command] that provides a list of container names, or via other supported options (eg: labels). This configuration is detailed in the [`watchtower` docs][watchtower-docs].
For more details, see the [manual](https://github.com/spotify/docker-gc/blob/master/README.md)
!!! info "Manual cleanup"
Or you can just use the [`--cleanup`](https://containrrr.github.io/watchtower/arguments/#cleanup) option provided by `containrrr/watchtower`.
`watchtower` also supports running on-demand with `docker run` or `compose.yaml` via the `--run-once` option.
You can alternatively invoke cleanup of Docker storage directly with:
This is a list of all configuration files and directories which are optional or automatically generated in your `config` directory.
## Volumes
## Directories
DMS has several locations in the container which may be worth persisting externally via [Docker Volumes][docker-docs::volumes].
- Often you will want to prefer [bind mount volumes][docker-docs::volumes::bind-mount] for easy access to files at a local location on your filesystem.
- As a convention for our docs and example configs, the local location has the common prefix `docker-data/dms/` for grouping these related volumes.
!!! info "Reference - Volmes for DMS"
Our docs may refer to these DMS specific volumes only by name, or the host/container path for brevity.
This is the location where mail is delivered to your mailboxes.
### State Volume { #volumes-state }
Run-time specific state lives here, but so does some data you may want to keep if a failure event occurs (_crash, power loss_).
!!! example "Examples of relevant data"
- The Postfix queue (eg: mail pending delivery attempt)
- Fail2Ban blocks.
- ClamAV signature updates.
- Redis storage for Rspamd.
!!! info "When a volume is mounted to `/var/mail-state/`"
- Service run-time data is [consolidated into the `/var/mail-state/` directory][mail-state-folders]. Otherwise the original locations vary and would need to be mounted individually.
- The original locations are updated with symlinks to redirect to their new path in `/var/mail-state/` (_eg: `/var/lib/redis` => `/var/mail-state/lib-redis/`_).
Sometimes it is helpful to disable this volume when troubleshooting to verify if the data stored here is in a bad state (_eg: caused by a failure event_).
This can be a useful volume to persist for troubleshooting needs for the full set of log files.
### Config Volume { #volumes-config }
Most configuration files for Postfix, Dovecot, etc. are persisted here.
This is a list of all configuration files and directories which are optional, automatically generated / updated by our `setup` CLI, or other internal scripts.
#### Directories
- **sieve-filter:** directory for sieve filter scripts. (Docs: [Sieve][docs-sieve])
- **sieve-pipe:** directory for sieve pipe scripts. (Docs: [Sieve][docs-sieve])
- **ssl:** SSL Certificate directory if `SSL_TYPE` is set to `self-signed` or `custom`. (Docs: [SSL][docs-ssl])
- **rspamd:** Override directory for custom settings when using Rspamd (Docs: [Rspamd][docs-rspamd-override-d])
## Files
#### Files
- **{user_email_address}.dovecot.sieve:** User specific Sieve filter file. (Docs: [Sieve][docs-sieve])
- **before.dovecot.sieve:** Global Sieve filter file, applied prior to the `${login}.dovecot.sieve` filter. (Docs: [Sieve][docs-sieve])
@ -24,33 +77,38 @@ This is a list of all configuration files and directories which are optional or
- **postfix-send-access.cf:** List of users denied sending. Modify via [`setup.sh email restrict`][docs-setupsh].
- **postfix-receive-access.cf:** List of users denied receiving. Modify via [`setup.sh email restrict`][docs-setupsh].
- **postfix-virtual.cf:** Alias configuration file. Modify via [`setup.sh alias`][docs-setupsh].
- **postfix-sasl-password.cf:** listing of relayed domains with their respective `<username>:<password>`. Modify via `setup.sh relay add-auth <domain> <username> [<password>]`. (Docs: [Relay-Hosts Auth][docs-relayhosts-senderauth])
- **postfix-relaymap.cf:** domain-specific relays and exclusions. Modify via `setup.sh relay add-domain` and `setup.sh relay exclude-domain`. (Docs: [Relay-Hosts Senders][docs-relayhosts-senderhost])
- **postfix-sasl-password.cf:** listing of relayed domains with their respective `<username>:<password>`. Modify via `setup.sh relay add-auth <domain> <username> [<password>]`. (Docs: [Relay-Hosts Auth][docs::relay-hosts::advanced])
- **postfix-relaymap.cf:** domain-specific relays and exclusions. Modify via `setup.sh relay add-domain` and `setup.sh relay exclude-domain`. (Docs: [Relay-Hosts Senders][docs::relay-hosts::advanced])
- **postfix-regexp.cf:** Regular expression alias file. (Docs: [Aliases][docs-aliases-regex])
- **ldap-users.cf:** Configuration for the virtual user mapping `virtual_mailbox_maps`. See the [`setup-stack.sh`][github-commit-setup-stack.sh-L411] script.
- **ldap-groups.cf:** Configuration for the virtual alias mapping `virtual_alias_maps`. See the [`setup-stack.sh`][github-commit-setup-stack.sh-L411] script.
- **ldap-aliases.cf:** Configuration for the virtual alias mapping `virtual_alias_maps`. See the [`setup-stack.sh`][github-commit-setup-stack.sh-L411] script.
- **ldap-domains.cf:** Configuration for the virtual domain mapping `virtual_mailbox_domains`. See the [`setup-stack.sh`][github-commit-setup-stack.sh-L411] script.
- **whitelist_clients.local:** Whitelisted domains, not considered by postgrey. Enter one host or domain per line.
- **dovecot-quotas.cf:** list of custom quotas per mailbox. (Docs: [Accounts][docs-accounts-quota])
- **user-patches.sh:** this file will be run after all configuration files are set up, but before the postfix, amavis and other daemons are started. (Docs: [FAQ - How to adjust settings with the `user-patches.sh` script][docs-faq-userpatches])
- **rspamd/custom-commands.conf:** list of simple commands to adjust Rspamd modules in an easy way (Docs: [Rspamd][docs-rspamd-commands])
Another important option is the `default_process_limit` (defaults to `100`). If high-security mode is enabled you'll need to make sure this count is higher than the maximum number of users that can be logged in simultaneously.
This limit is quickly reached if users connect to the mail server with multiple end devices.
This limit is quickly reached if users connect to DMS with multiple end devices.
## Override Configuration
@ -34,10 +34,17 @@ For major configuration changes it’s best to override the dovecot configuratio
The Postfix default configuration can easily be extended by providing a `config/postfix-main.cf` in postfix format.
This can also be used to add configuration that is not in our default configuration.
[Our default Postfix configuration](https://github.com/docker-mailserver/docker-mailserver/blob/master/target/postfix/main.cf) can easily be extended to add parameters or modify existing ones by providing a `docker-data/dms/config/postfix-main.cf`. This file uses the same format as Postfix `main.cf` does ([See official docs](http://www.postfix.org/postconf.5.html) for all parameters and syntax rules).
For example, one common use of this file is for increasing the default maximum message size:
```cf
# increase maximum message size
message_size_limit = 52428800
```
That specific example is now supported and can be handled by setting `POSTFIX_MESSAGE_SIZE_LIMIT`.
!!! seealso
[Postfix documentation](http://www.postfix.org/documentation.html) remains the best place to find configuration options.
Each line in the provided file will be loaded into postfix.
In the same way it is possible to add a custom `config/postfix-master.cf` file that will override the standard `master.cf`. Each line in the file will be passed to `postconf -P`. The expected format is `<service_name>/<type>/<parameter>`, for example:
!!! example "Example"
One can easily increase the [backwards-compatibility level](http://www.postfix.org/postconf.5.html#compatibility_level) and set new Postscreen options:
```cf
# increase the compatibility level from 2 (default) to 3
compatibility_level = 3
# set a threshold value for Spam detection
postscreen_dnsbl_threshold = 4
```
!!! help "How are your changes applied?"
The custom configuration you supply is appended to the default configuration located at `/etc/postfix/main.cf`, and then `postconf -nf` is run to remove earlier duplicate entries that have since been replaced. This happens early during container startup before Postfix is started.
---
Similarly, it is possible to add a custom `docker-data/dms/config/postfix-master.cf` file that will override the standard `master.cf`. **Note**: Each line in this file will be passed to `postconf -P`, i.e. **the file is not appended as a whole** to `/etc/postfix/master.cf` like `docker-data/dms/config/postfix-main.cf`! The expected format is `<service_name>/<type>/<parameter>`, for example:
```cf
# adjust the submission "reject_unlisted_recipient" option
title: 'Custom User Changes & Patches | Scripting'
---
If you'd like to change, patch or alter files or behavior of DMS, you can use a script.
In case you cloned this repository, you can copy the file [`user-patches.sh.dist` (_under `config/`_)][github-file-userpatches] with `#!sh cp config/user-patches.sh.dist docker-data/dms/config/user-patches.sh` in order to create the `user-patches.sh` script.
If you are managing your directory structure yourself, create a `docker-data/dms/config/` directory and add the `user-patches.sh` file yourself.
``` sh
# 1. Either create the docker-data/dms/config/ directory yourself
# or let docker-mailserver create it on initial startup
/tmp $ mkdir -p docker-data/dms/config/ && cd docker-data/dms/config/
#------------ Do not modify anything below this line -------------
1; # ensure a defined return
END
```
And you're done. The user patches script runs right before starting daemons. That means, all the other configuration is in place, so the script can make final adjustments.
!!! note
Many "patches" can already be done with the Docker Compose-/Stack-file. Adding hostnames to `/etc/hosts` is done with the `#!yaml extra_hosts:` section, `sysctl` commands can be managed with the `#!yaml sysctls:` section, etc.
Podman is a daemonless container engine for developing, managing, and running OCI Containers on your Linux System.
!!! warning "About Support for Podman"
Please note that Podman **is not** officially supported as DMS is built and verified on top of the _Docker Engine_. This content is entirely community supported. If you find errors, please open an issue and provide a PR.
!!! warning "About this Guide"
This guide was tested with Fedora 34 using `systemd` and `firewalld`. Moreover, it requires Podman version >= 3.2. You may be able to substitute `dnf` - Fedora's package manager - with others such as `apt`.
!!! warning "About Security"
Running podman in rootless mode requires additional modifications in order to keep your mailserver secure.
Make sure to read the related documentation.
## Installation in Rootfull Mode
While using Podman, you can just manage docker-mailserver as what you did with Docker. Your best friend `setup.sh` includes the minimum code in order to support Podman since it's 100% compatible with the Docker CLI.
The installation is basically the same. Podman v3.2 introduced a RESTful API that is 100% compatible with the Docker API, so you can use Docker Compose with Podman easily. Install Podman and Docker Compose with your package manager first.
```bash
sudo dnf install podman docker-compose
```
Then enable `podman.socket` using `systemctl`.
```bash
systemctl enable --now podman.socket
```
This will create a unix socket locate under `/run/podman/podman.sock`, which is the entrypoint of Podman's API. Now, configure docker-mailserver and start it.
You should see that docker-mailserver is running now.
### Self-start in Rootfull Mode
Podman is daemonless, that means if you want docker-mailserver self-start while boot up the system, you have to generate a systemd file with Podman CLI.
Running rootless containers is one of Podman's major features. But due to some restrictions, deploying docker-mailserver in rootless mode is not as easy compared to rootfull mode.
- a rootless container is running in a user namespace so you cannot bind ports lower than 1024
- a rootless container's systemd file can only be placed in folder under `~/.config`
- a rootless container can result in an open relay, make sure to read the [security section](#security-in-rootless-mode).
Also notice that Podman's rootless mode is not about running as a non-root user inside the container, but about the mapping of (normal, non-root) host users to root inside the container.
!!! warning
In order to make rootless DMS work we must modify some settings in the Linux system, it requires some basic linux server knowledge so don't follow this guide if you not sure what this guide is talking about. Podman rootfull mode and Docker are still good and security enough for normal daily usage.
First, enable `podman.socket` in systemd's userspace with a non-root user.
```bash
systemctl enable --now --user podman.socket
```
The socket file should be located at `/var/run/user/$(id -u)/podman/podman.sock`. Then, modify `compose.yaml` to make sure all ports are bindings are on non-privileged ports.
```yaml
services:
mailserver:
ports:
- "10025:25" # SMTP (explicit TLS => STARTTLS)
- "10143:143" # IMAP4 (explicit TLS => STARTTLS)
- "10465:465" # ESMTP (implicit TLS)
- "10587:587" # ESMTP (explicit TLS => STARTTLS)
- "10993:993" # IMAP4 (implicit TLS)
```
Then, setup your `mailserver.env` file follow the documentation and use Docker Compose to start the container.
In rootless mode, podman resolves all incoming IPs as localhost, which results in an open gateway in the default configuration. There are two workarounds to fix this problem, both of which have their own drawbacks.
#### Enforce authentication from localhost
The `PERMIT_DOCKER` variable in the `mailserver.env` file allows to specify trusted networks that do not need to authenticate. If the variable is left empty, only requests from localhost and the container IP are allowed, but in the case of rootless podman any IP will be resolved as localhost. Setting `PERMIT_DOCKER=none` enforces authentication also from localhost, which prevents sending unauthenticated emails.
#### Use the slip4netns network driver
The second workaround is slightly more complicated because the `compose.yaml` has to be modified.
As shown in the [fail2ban section][docs::fail2ban::rootless] the `slirp4netns` network driver has to be enabled.
This network driver enables podman to correctly resolve IP addresses but it is not compatible with
user defined networks which might be a problem depending on your setup.
[Rootless Podman][rootless::podman] requires adding the value `slirp4netns:port_handler=slirp4netns` to the `--network` CLI option, or `network_mode` setting in your `compose.yaml`.
You must also add the ENV `NETWORK_INTERFACE=tap0`, because Podman uses a [hard-coded interface name][rootless::podman::interface] for `slirp4netns`.
Systemd's user space service is only started when a specific user logs in and stops when you log out. In order to make it to start with the system, we need to enable linger with `loginctl`
```bash
loginctl enable-linger <username>
```
Remember to run this command as root user.
### Port Forwarding
When it comes to forwarding ports using `firewalld`, see [these port forwarding docs][firewalld-port-forwarding] for more information.
Email auto-discovery means a client email is able to automagically find out about what ports and security options to use, based on the mail server URL. It can help simplify the tedious / confusing task of adding own's email account for non-tech savvy users.
# Auto-Discovery of Services
Email auto-discovery means a client email is able to automagically find out about what ports and security options to use, based on the mail server URI. It can help simplify the tedious / confusing task of adding own's email account for non-tech savvy users.
Email clients will search for auto-discoverable settings and prefill almost everything when a user enters its email address :heart:
DKIM is a security measure targeting email spoofing. It is greatly recommended one activates it.
!!! seealso
See [the Wikipedia page](https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail) for more details on DKIM.
## Enabling DKIM Signature
To enable DKIM signature, **you must have created at least one email account**. Once its done, just run the following command to generate the signature:
```sh
./setup.sh config dkim
```
After generating DKIM keys, you should restart the mail server. DNS edits may take a few minutes to hours to propagate. The script assumes you're being in the directory where the `config/` directory is located. The default keysize when generating the signature is 4096 bits for now. If you need to change it (e.g. your DNS provider limits the size), then provide the size as the first parameter of the command:
```sh
./setup.sh config dkim keysize <keysize>
```
For LDAP systems that do not have any directly created user account you can run the following command (since `8.0.0`) to generate the signature by additionally providing the desired domain name (if you have multiple domains use the command multiple times or provide a comma-separated list of domains):
Now the keys are generated, you can configure your DNS server with DKIM signature, simply by adding a TXT record. If you have direct access to your DNS zone file, then it's only a matter of pasting the content of `config/opendkim/keys/domain.tld/mail.txt` in your `domain.tld.hosts` zone.
```console
$ dig mail._domainkey.domain.tld TXT
---
;; ANSWER SECTION
mail._domainkey.<DOMAIN> 300 IN TXT "v=DKIM1; k=rsa; p=AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN/AZERTYUIOPQSDFGHJKLMWXCVBN"
```
## Configuration using a Web Interface
1. Generate a new record of the type `TXT`.
2. Paste `mail._domainkey` the `Name` txt field.
3. In the `Target` or `Value` field fill in `v=DKIM1; k=rsa; p=AZERTYUGHJKLMWX...`.
4. In `TTL` (time to live): Time span in seconds. How long the DNS server should cache the `TXT` record.
5. Save.
!!! note
Sometimes the key in `config/opendkim/keys/domain.tld/mail.txt` can be on multiple lines. If so then you need to concatenate the values in the TXT record:
```console
$ dig mail._domainkey.domain.tld TXT
---
;; ANSWER SECTION
mail._domainkey.<DOMAIN> 300 IN TXT "v=DKIM1; k=rsa; "
"p=AZERTYUIOPQSDF..."
"asdfQWERTYUIOPQSDF..."
```
The target (or value) field must then have all the parts together: `v=DKIM1; k=rsa; p=AZERTYUIOPQSDF...asdfQWERTYUIOPQSDF...`
## Verify-Only
If you want DKIM to only _verify_ incoming emails, the following version of `/etc/opendkim.conf` may be useful (right now there is no easy mechanism for installing it other than forking the repo):
```conf
# This is a simple config file verifying messages only
#LogWhy yes
Syslog yes
SyslogSuccess yes
Socket inet:12301@localhost
PidFile /var/run/opendkim/opendkim.pid
ReportAddress postmaster@my-domain.com
SendReports yes
Mode v
```
## Switch Off DKIM
Simply remove the DKIM key by recreating (not just relaunching) the mailserver container.
## Debugging
- [DKIM-verifer](https://addons.mozilla.org/en-US/thunderbird/addon/dkim-verifier): A add-on for the mail client Thunderbird.
- You can debug your TXT records with the `dig` tool.
mail._domainkey.domain.tld. 3600 IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCxBSjG6RnWAdU3oOlqsdf2WC0FOUmU8uHVrzxPLW2R3yRBPGLrGO1++yy3tv6kMieWZwEBHVOdefM6uQOQsZ4brahu9lhG8sFLPX4MaKYN/NR6RK4gdjrZu+MYSdfk3THgSbNwIDAQAB"
;; Query time: 50 msec
;; SERVER: 127.0.1.1#53(127.0.1.1)
;; WHEN: Wed Sep 07 18:22:57 CEST 2016
;; MSG SIZE rcvd: 310
```
---
!!! warning "Key sizes >=4096-bit"
Keys of 4096 bits could de denied by some mailservers. According to https://tools.ietf.org/html/rfc6376 keys are preferably between 512 and 2048 bits. See issue [#1854][github-issue-1854].
Cloudflare has written an [article about DKIM, DMARC and SPF][cloudflare-dkim-dmarc-spf] that we highly recommend you to read to get acquainted with the topic.
!!! note "Rspamd vs Individual validators"
With v12.0.0, Rspamd was integrated into DMS. It can perform validations for DKIM, DMARC and SPF as part of the `spam-score-calculation` for an email. DMS provides individual alternatives for each validation that can be used instead of deferring to Rspamd:
- DKIM: `opendkim` is used as a milter (like Rspamd)
- DMARC: `opendmarc` is used as a milter (like Rspamd)
- SPF: `policyd-spf` is used in Postfix's `smtpd_recipient_restrictions`
In a future release Rspamd will become the default for these validations, with a deprecation notice issued prior to the removal of the above alternatives.
We encourage everyone to prefer Rspamd via `ENABLE_RSPAMD=1`.
!!! warning "DNS Caches & Propagation"
While modern DNS providers are quick, it may take minutes or even hours for new DNS records to become available / propagate.
## DKIM
!!! quote "What is DKIM"
DomainKeys Identified Mail (DKIM) is an email authentication method designed to detect forged sender addresses in email (email spoofing), a technique often used in phishing and email spam.
[Source][wikipedia-dkim]
When DKIM is enabled:
1. Inbound mail will verify any included DKIM signatures
2. Outbound mail is signed (_when your sending domain has a configured DKIM key_)
DKIM requires a public/private key pair to enable **signing (_via private key_)** your outgoing mail, while the receiving end must query DNS to **verify (_via public key_)** that the signature is trustworthy.
??? info "Verification expiry"
Unlike your TLS certificate, your DKIM keypair does not have a fixed expiry associated to it.
Instead, an expiry may be included in your DKIM signature for each mail sent, where a receiver will [refuse to validate the signature for an email after that expiry date][dkim-verification-expiry-refusal]. This is an added precaution to mitigate malicious activity like "DKIM replay attacks", where an already delivered email from a third-party with a trustworthy DKIM signature is leveraged by a spammer when sending mail to an MTA which verifies the DKIM signature successfully, enabling the spammer to bypass spam protections.
Unlike a TLS handshake where you are authenticating trust with future communications, with DKIM once the mail has been received and trust of the signature has been verified, the value of verifying the signature again at a later date is less meaningful since the signature was to ensure no tampering had occurred during delivery through the network.
??? tip "DKIM key rotation"
You can rotate your DKIM keypair by switching to a new DKIM selector (_and DNS updates_), while the previous key and selector remains valid for verification until the last mail signed with that key reaches it's expiry.
DMS does not provide any automation or support for key rotation, [nor is it likely to provide a notable security benefit][gh-discussion::dkim-key-rotation-expiry] to the typical small scale DMS deployment.
### Generating Keys
You'll need to repeat this process if you add any new domains.
You should have:
- At least one [email account setup][docs-accounts]
- Attached a [volume for config][docs-volumes-config] to persist the generated files to local storage
!!! example "Creating DKIM Keys"
DKIM keys can be generated with good defaults by running:
```bash
docker exec -it <CONTAINERNAME> setup config dkim
```
If you need to generate your keys with different settings, check the `help` output for supported config options and examples:
```bash
docker exec -it <CONTAINERNAME> setup config dkim help
```
As described by the help output, you may need to use the `domain` option explicitly when you're using LDAP or Rspamd.
??? info "Changing the key size"
The keypair generated for using with DKIM presently defaults to RSA-2048. This is a good size but you can lower the security to `1024-bit`, or increase it to `4096-bit` (_discouraged as that is excessive_).
To generate a key with different size (_for RSA 1024-bit_) run:
```sh
setup config dkim keysize 1024
```
!!! warning "RSA Key Sizes >= 4096 Bit"
According to [RFC 8301][rfc-8301], keys are preferably between 1024 and 2048 bits. Keys of size 4096-bit or larger may not be compatible to all systems your mail is intended for.
You [should not need a key length beyond 2048-bit][gh-issue::dkim-length]. If 2048-bit does not meet your security needs, you may want to instead consider adopting key rotation or switching from RSA to ECC keys for DKIM.
??? note "You may need to specify mail domains explicitly"
Required when using LDAP and Rspamd.
`setup config dkim` will generate DKIM keys for what is assumed as the primary mail domain (_derived from the FQDN assigned to DMS, minus any subdomain_).
When the DMS FQDN is `mail.example.com` or `example.com`, by default this command will generate DKIM keys for `example.com` as the primary domain for your users mail accounts (eg: `hello@example.com`).
The DKIM generation does not have support to query LDAP for additional mail domains it should know about. If the primary mail domain is not sufficient, then you must explicitly specify any extra domains via the `domain` option:
!!! info "OpenDKIM with `ACCOUNT_PROVISIONER=FILE`"
When DMS uses this configuration, it will by default also detect mail domains (_from accounts added via `setup email add`_), generating additional DKIM keys.
DKIM is currently supported by either OpenDKIM or Rspamd:
=== "OpenDKIM"
OpenDKIM is currently [enabled by default][docs-env-opendkim].
After running `setup config dkim`, your new DKIM key files (_and OpenDKIM config_) have been added to `/tmp/docker-mailserver/opendkim/`.
!!! info "Restart required"
After restarting DMS, outgoing mail will now be signed with your new DKIM key(s) :tada:
=== "Rspamd"
Requires opt-in via [`ENABLE_RSPAMD=1`][docs-env-rspamd] (_and disable the default OpenDKIM: `ENABLE_OPENDKIM=0`_).
Rspamd provides DKIM support through two separate modules:
1. [Verifying DKIM signatures from inbound mail][rspamd-docs-dkim-checks] is enabled by default.
2. [Signing outbound mail with your DKIM key][rspamd-docs-dkim-signing] needs additional setup (key + dns + config).
??? warning "Using Multiple Domains"
If you have multiple domains, you need to:
- Create a key wth `docker exec -it <CONTAINER NAME> setup config dkim domain <DOMAIN>` for each domain DMS should sign outgoing mail for.
- Provide a custom `dkim_signing.conf` (for which an example is shown below), as the default config only supports one domain.
!!! info "About the Helper Script"
The script will persist the keys in `/tmp/docker-mailserver/rspamd/dkim/`. Hence, if you are already using the default volume mounts, the keys are persisted in a volume. The script also restarts Rspamd directly, so changes take effect without restarting DMS.
The script provides you with log messages along the way of creating keys. In case you want to read the complete log, use `-v` (verbose) or `-vv` (very verbose).
---
In case you have not already provided a default DKIM signing configuration, the script will create one and write it to `/etc/rspamd/override.d/dkim_signing.conf`. If this file already exists, it will not be overwritten.
When you're already using [the `rspamd/override.d/` directory][docs-rspamd-config-dropin], the file is created inside your volume and therefore persisted correctly. If you are not using `rspamd/override.d/`, you will need to persist the file yourself (otherwise it is lost on container restart).
An example of what a default configuration file for DKIM signing looks like can be found by expanding the example below.
??? example "DKIM Signing Module Configuration Examples"
- You can add more domains into the `domain { ... }` section (in the following example: `example.com` and `example.org`).
- A domain can also be configured with multiple selectors and keys within a `selectors [ ... ]` array (in the following example, this is done for `example.org`).
This modern elliptic curve is supported by Rspamd, but support by third-parties for [verifying Ed25519 DKIM signatures is unreliable][dkim-ed25519-support].
If you sign your mail with this key type, you should include RSA as a fallback, like shown in the above example.
??? tip "Let Rspamd Check Your Keys"
When `check_pubkey = true;` is set, Rspamd will query the DNS record for each DKIM selector, verifying each public key matches the private key configured.
If there is a mismatch, a warning will be emitted to the Rspamd log `/var/log/mail/rspamd.log`.
### DNS Record { #dkim-dns }
When mail signed with your DKIM key is sent from your mail server, the receiver needs to check a DNS `TXT` record to verify the DKIM signature is trustworthy.
!!! example "Configuring DNS - DKIM record"
When you generated your key in the previous step, the DNS data was saved into a file `<selector>.txt` (default: `mail.txt`). Use this content to update your [DNS via Web Interface][dns::example-webui] or directly edit your [DNS Zone file][dns::wikipedia-zonefile]:
| Name | `<selector>._domainkey` (_default: `mail._domainkey`_) |
| TTL | Use the default (_otherwise [3600 seconds is appropriate][dns::digicert-ttl]_) |
| Data | File content within `( ... )` (_formatted as advised below_) |
When using Rspamd, the helper script has already provided you with the contents (the "Data" field) of the DNS record you need to create - you can just copy-paste this text.
=== "DNS Zone file"
`<selector>.txt` is already formatted as a snippet for adding to your [DNS Zone file][dns::wikipedia-zonefile].
Just copy/paste the file contents into your existing DNS zone. The `TXT` value has been split into separate strings every 255 characters for compatibility.
??? info "`<selector>.txt` - Formatting the `TXT` record value correctly"
This file was generated for use within a [DNS zone file][dns::wikipedia-zonefile]. The file name uses the DKIM selector it was generated with (default DKIM selector is `mail`, which creates `mail.txt`_).
For your DNS setup, DKIM support needs to create a `TXT` record to store the public key for mail clients to use. `TXT` records with values that are longer than 255 characters need to be split into multiple parts. This is why the generated `<selector>.txt` file (_containing your public key for use with DKIM_) has multiple value parts wrapped within double-quotes between `(` and `)`.
A DNS web-interface may handle this separation internally instead, and [could expect the value provided all as a single line][dns::webui-dkim] instead of split. When that is required, you'll need to manually format the value as described below.
Your generated DNS record file (`<selector>.txt`) should look similar to this:
Take the content between `( ... )`, and combine all the quote wrapped content and remove the double-quotes including the white-space between them. That is your `TXT` record value, the above example would become this:
To test that your new DKIM record is correct, query it with the `dig` command. The `TXT` value response should be a single line split into multiple parts wrapped in double-quotes:
[MxToolbox has a DKIM Verifier][mxtoolbox-dkim-verifier] that you can use to check your DKIM DNS record(s).
When using Rspamd, we recommend you turn on `check_pubkey = true;` in `dkim_signing.conf`. Rspamd will then check whether your private key matches your public key, and you can check possible mismatches by looking at `/var/log/mail/rspamd.log`.
## DMARC
With DMS, DMARC is pre-configured out of the box. You may disable extra and excessive DMARC checks when using Rspamd via `ENABLE_OPENDMARC=0`.
The only thing you need to do in order to enable DMARC on a "DNS-level" is to add new `TXT`. In contrast to [DKIM](#dkim), DMARC DNS entries do not require any keys, but merely setting the [configuration values][dmarc-howto-configtags]. You can either handcraft the entry by yourself or use one of available generators (like [this one][dmarc-tool-gca]).
Typically something like this should be good to start with:
The DMARC status may not be displayed instantly due to delays in DNS (caches). Dmarcian has [a few tools][dmarcian-tools] you can use to verify your DNS records.
## SPF
!!! quote "What is SPF"
Sender Policy Framework (SPF) is a simple email-validation system designed to detect email spoofing by providing a mechanism to allow receiving mail exchangers to check that incoming mail from a domain comes from a host authorized by that domain's administrators.
[Source][wikipedia-spf]
!!! tip "Disabling the default SPF service `policy-spf`"
Set [`ENABLE_POLICYD_SPF=0`][docs-env-spf-policyd] to opt-out of the default SPF service. Advised when Rspamd is configured to handle SPF instead.
### Adding an SPF Record
To add a SPF record in your DNS, insert the following line in your DNS zone:
```txt
example.com. IN TXT "v=spf1 mx ~all"
```
This enables the _Softfail_ mode for SPF. You could first add this SPF record with a very low TTL. _SoftFail_ is a good setting for getting started and testing, as it lets all email through, with spams tagged as such in the mailbox.
After verification, you _might_ want to change your SPF record to `v=spf1 mx -all` so as to enforce the _HardFail_ policy. See <http://www.open-spf.org/SPF_Record_Syntax> for more details about SPF policies.
In any case, increment the SPF record's TTL to its final value.
### Backup MX & Secondary MX for `policyd-spf`
For whitelisting an IP Address from the SPF test, you can create a config file (see [`policyd-spf.conf`](https://www.linuxcertif.com/man/5/policyd-spf.conf)) and mount that file into `/etc/postfix-policyd-spf-python/policyd-spf.conf`.
**Example:** Create and edit a `policyd-spf.conf` file at `docker-data/dms/config/postfix-policyd-spf.conf`:
In `docker-mailserver`, DMARC is pre-configured out-of the box. The only thing you need to do in order to enable it, is to add new TXT entry to your DNS.
In contrast with [DKIM][docs-dkim], DMARC DNS entry does not require any keys, but merely setting the [configuration values](https://github.com/internetstandards/toolbox-wiki/blob/master/DMARC-how-to.md#overview-of-dmarc-configuration-tags). You can either handcraft the entry by yourself or use one of available generators (like https://dmarcguide.globalcyberalliance.org/).
Typically something like this should be good to start with (don't forget to replace `@domain.com` to your actual domain)
```
_dmarc.domain.com. IN TXT "v=DMARC1; p=none; rua=mailto:dmarc.report@domain.com; ruf=mailto:dmarc.report@domain.com; sp=none; ri=86400"
```
Or a bit more strict policies (mind `p=quarantine` and `sp=quarantine`):
DMARC status is not being displayed instantly in Gmail for instance. If you want to check it directly after DNS entries, you can use some services around the Internet such as https://dmarcguide.globalcyberalliance.org/ or https://ondmarc.redsift.com/. In other case, email clients will show "DMARC: PASS" in ~1 day or so.
MTA-STS is an optional mechanism for a domain to signal support for STARTTLS.
- It can be used to prevent man-in-the-middle-attacks from hiding STARTTLS support that would force DMS to send outbound mail through an insecure connection.
- MTA-STS is an alternative to DANE without the need of DNSSEC.
- MTA-STS is supported by some of the biggest mail providers like Google Mail and Outlook.
## Supporting MTA-STS for outbound mail
Enable this feature via the ENV setting [`ENABLE_MTA_STS=1`](../environment.md#enable_mta_sts).
!!! warning "If you have configured DANE"
Enabling MTA-STS will by default override DANE if both are configured for a domain.
This can be partially addressed by configuring a dane-only policy resolver before the MTA-STS entry in `smtp_tls_policy_maps`. See the [`postfix-mta-sts-resolver` documentation][postfix-mta-sts-resolver::dane] for further details.
While this feature in DMS supports ensuring STARTTLS is used when mail is sent to another mail server, you may setup similar for mail servers sending mail to DMS.
This requires configuring your DNS and hosting the MTA-STS policy file via a webserver. A good introduction can be found on [dmarcian.com](https://dmarcian.com/mta-sts/).
Some files were not shown because too many files have changed in this diff
Show More
Brennan Kinney
beertje44
Christian Schmidt
Georg Lauterbach
Moritz Poldrack
tranquillity-codes
Lasslos
Dmitry R.
dependabot[bot]
Casper
Alessio Artoni
Zlatibor Veljkovic
Wojciech Woźniak
Pooyan Khanjankhani
Alvaro Muñoz
pitilux
GallowsDove
Vetu
github-actions[bot]
Moritz Poldrack
mmehnert
Guillaume VARA
Tobias Knecht
Jiří Štefka
pyy
F. Eber
Wael
Fürst
Tobia Bocchi
Iztok Fister Jr
fanqiaojun
Inseo Song
Rahil Bhimjiani
Ikko Eltociear Ashimine
Kirill Kirilenko
Dominic Germain
Jesse Portnoy
Robbert Klarenbeek
Frugan
Jackson Zheng
Hans-Cees Speel
Aaron Spettl
Andreas Perhab
Jam Balaya
Roy Sindre Norangshol
Den
Joerg Sonnenberger
Keval Kapdee
René Plötz
Peter Adam
Jean-Kevin KPADEY
Zepmann
allddd
Vincent Ducamps
jpduyx
Lucas Bartholemy
Milas Bowman
HeySora
H4R0
Nils Höll
rmlhuk
rriski
Felix N
wligtenberg
Claude Brisson
Thomas Butter
Arun
LucidityCrash
ghnp5
James
Jack Pearson
Lin Han
Leon Busch-George
i-C-o-d-e-r
Jeidnx
worldworm
Y.C.Huang
Gabriel Euzet
Vasiliy Gokoyev
pravynandas
Leon George
Felix
ErikEngerd
JiLleON
smargold476
Thomas VIAL
Frederic Werner
Maximilian Hippler
Naveen
Andrew Cornford
ontheair81
Nathan Pierce
NorseGaud
sportshead
John McClean
Philipp Fruck
wolkenschieber
Moritz Poldrack
Christian Raue
schnippl0r
Twist235
Khue Doan
Semir Patel
Felix
Robert Schumann
James Calfee
Mohammed Noureldin
Simon J Mudd
Andreas Egli
2b
Frugan
Amin Vakil
dborowy
Michael Jensen
William Desportes
Orville Q. Song
Daniël van den Berg
Olivier Picquenot
allcontributors[bot]
matrixes
Andrew Low
eleith
20th
hnws
Georg Lauterbach
okami
Moritz Marquardt
Patrizio Bekerle
Moritz Marquardt
Shyim
