Stage dedicated Agent Mail infrastructure - #467
Conversation
|
@codex review |
|
Failed to generate code suggestions for PR |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98104567bb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| agentmail: | ||
| ansible_host: 2a0c:b641:b50:2::110 |
There was a problem hiding this comment.
Keep the planned host out of fleet-wide play targets
Until Stage 1 provisions this VM, adding agentmail to linux makes it part of the canonical nightly/post-merge drift sweep: scripts/ci/check-drift.sh limits to all:!ci-pr, while the firewall, monitoring, and logs playbooks target all or linux. Those applies will attempt SSH to the unprovisioned ::110 address and fail, contrary to the runbook's instruction to apply monitoring only after the host is reachable. Exclude the staged host from fleet-wide limits until provisioning or model it outside the active groups.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 89601cc. The planned host is a member of the staged group, and the canonical drift limit is now all:!ci-pr:!staged. Contract tests verify both the membership and exclusion.
| {% endif %} | ||
| {% if agent_mail_public_enabled | bool %} | ||
| - "[{{ agent_mail_overlay_ipv6 }}]:25:25/tcp" | ||
| - "{{ agent_mail_public_ipv4 }}:25:25/tcp" |
There was a problem hiding this comment.
Allow Docker-forwarded SMTP through the forward chain
When the public IPv4 gate is enabled on the normal Docker bridge setup, this published port is DNATed to the container and traverses the host's forward hook rather than input. The generated Agent Mail nftables rules put the TCP/25 allowance only in input, while ansible/roles/firewall/templates/nftables.conf.j2 leaves forward at policy drop with only established/related traffic allowed and agentmail.yml supplies no forward exception. Consequently new external IPv4 SMTP connections are dropped even with every launch gate enabled; add a scoped Docker-forward rule or use host networking.
Useful? React with 👍 / 👎.
| changed_when: >- | ||
| 'Started' in _agent_mail_up.stderr or 'Created' in _agent_mail_up.stderr or | ||
| 'Recreated' in _agent_mail_up.stderr | ||
| when: agent_mail_start | bool |
There was a problem hiding this comment.
Stop Stalwart when the start gate is disabled
On a host where Stalwart has previously been started, changing agent_mail_start back to false merely skips this compose up task; there is no corresponding compose stop or down, and the service has restart: unless-stopped. The container and its currently published listeners therefore remain active despite the supposedly disabled service-start gate, making rollback of that safety control ineffective.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
| 1. Set Hyrule Cloud `MAIL_ENABLED=false` to stop new activation/send traffic. | ||
| 2. Set `agent_mail_public_enabled`, its SMTP firewall twin, and both TCP/25 | ||
| rule fields false; apply Compose and nftables. This preserves stored mail | ||
| while stopping inbound/outbound public delivery. |
There was a problem hiding this comment.
Block outbound SMTP during emergency shutdown
In the documented abuse/emergency scenario, disabling MAIL_ENABLED stops new Cloud requests and disabling agent_mail_public_enabled removes inbound port mappings, but neither action blocks the running Stalwart container's unrestricted egress; the host firewall output policy remains accept. Already queued mail can therefore continue outbound delivery while operators believe public delivery is stopped. The shutdown procedure needs an outbound TCP/25 gate or an explicit service stop before making this claim.
Useful? React with 👍 / 👎.
| - agent_mail_backup_restore_verified | bool | ||
| - agent_mail_canaries_verified | bool | ||
| - agent_mail_recovery_admin_secret | length == 0 | ||
| - agent_mail_public_ipv4 is match('^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$') |
There was a problem hiding this comment.
Validate that the launch address is a dedicated IPv4
At public launch this regex accepts invalid addresses such as 999.999.999.999 and also accepts the explicitly forbidden corporate mail_failover_ipv4 (51.91.236.215). The former passes every readiness assertion only for Compose to fail while binding the port, and the latter defeats the role's stated isolation boundary. Parse the value as an IPv4 address and explicitly reject mail_failover_ipv4 before allowing the public gate.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 89601cc. Public launch now requires an IPv4 whose octets are each in the 0-255 range and explicitly rejects mail_failover_ipv4.
| -cf "$partial" \ | ||
| etc/stalwart var/lib/stalwart | ||
| mv "$partial" "$archive" | ||
| sha256sum "$archive" > "$archive.sha256" |
There was a problem hiding this comment.
Write portable checksums for off-host restores
Because $archive is an absolute path, the checksum file records /var/backups/agent-mail/stalwart-…tar.zst rather than the archive basename. After the archive and checksum are copied to the required off-host repository or isolated restore host, a normal sha256sum -c therefore looks back under /var/backups/agent-mail and fails or checks an unrelated local file. Generate the checksum from inside $backup_dir or write the basename so the paired artifacts remain verifiable after transfer.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 89601cc. The backup script changes into the backup directory and writes the checksum against the archive basename, so the archive and checksum remain portable together.
| `agent_mail_bootstrap_firewall_enabled`, and the TCP/8080 rule's `enabled` | ||
| field. Keep public SMTP false. Supply a new temporary recovery password of | ||
| at least 32 characters. | ||
| 2. Apply this role and connect only from the ops prefix or VPN. Submit the |
There was a problem hiding this comment.
Reapply nftables when enabling the bootstrap listener
Stage 1 applies the firewall while the TCP/8080 rule is disabled, but after Stage 2 changes that rule this instruction runs only the agent_mail role, which does not include or reload the firewall role. The live nftables input chain therefore still lacks the 8080 allowance and the documented bootstrap curl remains blocked even though all three inventory controls are true. Explicitly apply playbooks/firewall.yml with firewall_apply=true after enabling the rule and before attempting bootstrap.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 89601cc. The bootstrap runbook now applies playbooks/firewall.yml with firewall_apply=true before the Agent Mail role and before curl, then reapplies the firewall when bootstrap is disabled.
|
Failed to generate code suggestions for PR |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89601cc375
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| networks: | ||
| default: | ||
| name: agent-mail | ||
| driver: bridge |
There was a problem hiding this comment.
Enable IPv6 connectivity for the Stalwart container
When Stage 2 starts this project, the custom Docker bridge is created without enable_ipv6, so the Stalwart container receives no IPv6 address or route under Docker's default configuration (Compose network documentation). The bootstrap plan nevertheless configures Knot using the literal IPv6 address 2a0c:b641:b50:2::10; consequently TSIG updates, ACME DNS reconciliation, and certificate bootstrap cannot reach Knot from the container. Configure routed IPv6 with a source accepted by the Knot ACL, or use an IPv4 endpoint reachable from this bridge.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 65ec286. The Agent Mail bridge now has enable_ipv6=true with an explicit ULA IPAM subnet, and the role enables IPv6 forwarding. Docker default NAT mode makes outbound IPv6 use the host overlay address already accepted by the Knot ACL. The rendered Compose model validates successfully.
| - "[2a0c:b641:b50:2::d0]:9100" # ci (privileged runner) | ||
| - "[2a0c:b641:b50:2::e0]:9100" # netproxy | ||
| - "[2a0c:b641:b50:2::f0]:9100" # loop (Engineering Loop / Knowledge MCP) | ||
| - "[2a0c:b641:b50:2::110]:9100" # agentmail (dedicated Stalwart) |
There was a problem hiding this comment.
Keep the staged host out of live Prometheus targets
Although check-drift.sh now excludes the staged inventory group, this target is part of the Prometheus configuration applied directly to mon, so that exclusion does not protect it. After the normal post-merge Prometheus apply, the known-unprovisioned address is scraped and the new AgentMailNodeUnavailable and AgentMailStalwartMetricsUnavailable rules become critical after five minutes, contradicting the runbook's instruction to install monitoring only after reachability. Add these targets and alerts only when the host is provisioned, or suppress them behind an explicit staging gate.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 65ec286. The unprovisioned node target and agent-mail-stalwart scrape job were removed from live Prometheus configuration. The runbook requires adding both in the reviewed activation change after reachability and staged-group removal, so the committed alert rules remain dormant meanwhile.
| that: | ||
| - not (agent_mail_start | bool) or (agent_mail_apply | bool) | ||
| - not (agent_mail_bootstrap_enabled | bool) or (agent_mail_start | bool) | ||
| - not (agent_mail_backup_enabled | bool) or (agent_mail_start | bool) |
There was a problem hiding this comment.
Allow the emergency stop while backups are enabled
In the launched state, agent_mail_backup_enabled must be true, but the documented emergency step changes only agent_mail_start to false and applies the role. This assertion therefore fails before apply.yml can reach the new docker compose down task, leaving the container and queued outbound delivery running during the shutdown attempt. The new stop task does not resolve the earlier rollback issue because validation executes first; either disable the backup/public gates atomically in the shutdown procedure or permit the role to stop both the timer and container in one apply.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 65ec286. Emergency shutdown now requires one atomic inventory change setting start, backup, public SMTP, its firewall twin, and both TCP/25 rules false before applying. Validation therefore permits the stop path, which stops both the backup timer and container.
| ports: | ||
| - "[{{ agent_mail_overlay_ipv6 }}]:443:443/tcp" |
There was a problem hiding this comment.
Preserve Docker networking across nftables reloads
Once Stalwart is running, any later firewall apply loads ansible/roles/firewall/templates/nftables.conf.j2, whose line 6 executes flush ruleset; this removes the NAT, forwarding, and masquerade rules Docker creates for bridge networks (Docker firewall documentation). The firewall handler reloads nftables without restarting Docker or reconverging this Compose project, so these published listeners and container egress can remain broken until Docker recreates its rules. Preserve Docker-managed tables during reload or explicitly restore the Compose network after every firewall apply.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Addressed in 65ec286. Agent Mail opts into preserving external nftables tables, so reload destroys and recreates only the role-owned inet filter table instead of flushing the ruleset. An isolated behavioral check applied the file twice and confirmed a simulated Docker table survived.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 65ec286bc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| - name: Stop and remove Agent Mail when the start gate is disabled | ||
| ansible.builtin.command: | ||
| argv: [docker, compose, --file, "{{ agent_mail_compose_file }}", down, --remove-orphans, --timeout, "120"] |
There was a problem hiding this comment.
Stop the backup service before taking Stalwart down
When an emergency shutdown overlaps the daily backup, this command can remove the container while agent-mail-backup.service is still running; the role only stops the timer afterward, and stopping a timer does not stop the service it already activated. The backup script's EXIT trap then runs docker compose up --detach stalwart, resurrecting the service after the shutdown path completed. Stop agent-mail-backup.service as well as the timer before invoking docker compose down so an in-flight backup cannot undo the shutdown.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a807ddb. The role now inspects the installed backup units at the start of apply, stops and disables the timer first, then waits for any in-flight backup service to stop before the later Compose shutdown. This lets the backup EXIT restart finish before docker compose down, and the same quiesce path runs when backups are disabled while Stalwart remains up. An ordering regression test covers timer, service, then teardown.
| fi | ||
|
|
||
| limit="${CHECK_DRIFT_LIMIT:-all:!ci-pr}" | ||
| limit="${CHECK_DRIFT_LIMIT:-all:!ci-pr:!staged}" |
There was a problem hiding this comment.
Add Agent Mail to the canonical drift playbook list
After provisioning, removing agentmail from staged does not actually bring its role under the canonical drift/apply sweep: this limit would then include the host, but default_playbooks still omits agent_mail. Consequently, drift in the Compose project, secret environment file, backup units, or service state is never detected or auto-applied even though the runbook says removing the staged membership makes the host part of the managed fleet. Add agent_mail to the canonical list, with whatever secret-safe check-mode handling the role requires.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a807ddb. agent_mail is now part of the canonical drift playbook list. The existing all:!ci-pr:!staged limit still excludes the unprovisioned host; before it leaves staged, the runbook requires its Vault values in the privileged runner environment. Secret validation and the runtime env template remain no_log. I also ran the canonical agent_mail drift entry and confirmed the current staged exclusion succeeds without requesting secrets.
| STALWART_DNS_TSIG_SECRET={{ agent_mail_dns_tsig_secret }} | ||
| STALWART_WEBHOOK_SECRET={{ agent_mail_webhook_secret }} | ||
| {% if agent_mail_bootstrap_enabled | bool %} | ||
| STALWART_RECOVERY_ADMIN=admin:{{ agent_mail_recovery_admin_secret }} |
There was a problem hiding this comment.
Preserve special characters in generated secrets
When any Vault-generated secret contains $ or another Compose env-file metacharacter, writing it unquoted here changes the value before it reaches Stalwart; Docker documents that unquoted env-file values undergo interpolation in its env-file syntax. This can make bootstrap authentication fail or cause webhook signatures and TSIG authentication to use a different secret from their peers. Render these values in a literal/raw-safe form or validate and encode them into a restricted character set.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a807ddb. Runtime Compose env values are now single-quoted, so dollar signs, hashes, spaces, and equals signs remain literal instead of being interpolated. Apply validation fails closed for quote, backslash, or line-break values and directs operators to unwrapped base64 or base64url; secret-bearing tasks stay no_log. Regression coverage renders dollar-sign and punctuation-bearing secrets.
| tar --acls --xattrs --numeric-owner --zstd -C / \ | ||
| -cf "$partial" \ | ||
| etc/stalwart var/lib/stalwart |
There was a problem hiding this comment.
Move full mailbox snapshots off the data disk
Once mailbox usage grows beyond roughly 5 GB, retaining 14 daily full-tree archives under /var/backups on the same planned 80 GB disk can exhaust the filesystem even while all live mailbox quotas are respected; the configured maximum is 25 mailboxes at 1 GB each, so incompressible mail and attachments can consume about 25 GB per archive and fill the host after only a few runs. A full filesystem can take RocksDB and mail delivery down, so these snapshots need a separate-capacity volume or deduplicating/incremental off-host storage with an explicit local space bound.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in a807ddb. Full snapshots now target a dedicated mount at /mnt/agent-mail-backup instead of the 80 GiB mail-data disk. The role and backup script require an actual mount, the systemd unit declares RequiresMountsFor, and the script enforces at least 100 GiB capacity, 32 GiB free space, two-day retention, and preflight cleanup before Stalwart is stopped. The rollout runbook now provisions and verifies that separate volume plus off-host transfer and restore evidence.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a807ddb2ce
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| archive="$backup_dir/stalwart-$timestamp.tar.zst" | ||
| was_running=0 | ||
|
|
||
| if docker compose --file "$compose_file" ps --services --status running | grep -Fxq stalwart; then |
There was a problem hiding this comment.
Fail closed when container-state detection errors
If docker compose ps fails—for example during a transient Docker daemon error—the pipeline is merely treated as a false if condition, so the script proceeds to archive /var/lib/stalwart without knowing whether Stalwart is still running. That can produce an inconsistent RocksDB backup while the timer still reports success; explicitly distinguish an empty running-service result from a nonzero Compose command before starting the archive.
Useful? React with 👍 / 👎.
| # ansible-playbook playbooks/agent_mail.yml --tags apply \ | ||
| # -e '{"agent_mail_apply":true}' --limit agentmail |
There was a problem hiding this comment.
Register Agent Mail in the protected apply workflow
When an operator reaches a rollout stage requiring this apply, the protected manual workflow cannot dispatch it: the workflow_dispatch.inputs.playbook.options list in .github/workflows/apply.yml contains mail_openbsd but not agent_mail. The direct command documented here therefore bypasses the repository's production environment gate and runner secret-loading path; add this playbook to that workflow before relying on it for the controlled rollout.
Useful? React with 👍 / 👎.
| exec 9>"$lock_file" | ||
| flock -n 9 || exit 0 | ||
|
|
||
| if ! mountpoint --quiet "$backup_dir"; then |
There was a problem hiding this comment.
Verify the backup mount uses a separate filesystem
If /mnt/agent-mail-backup is accidentally a bind mount or another mount of the root/data filesystem, this check succeeds and the subsequent capacity check can also pass, leaving the supposedly dedicated backup on the same failure domain as the mailbox data. As mountpoint --help states, it only checks whether the path is a mountpoint; compare its filesystem device/source with / and /var/lib/stalwart to enforce the runbook's distinct-volume requirement.
Useful? React with 👍 / 👎.
| http_ipv6: true | ||
| http_timeout: 10 | ||
|
|
||
| logs_register: true |
There was a problem hiding this comment.
Allow Agent Mail traffic through the log firewall
When the logs role is applied to the new host, its generated Vector configuration sends journald events to log:6000, but the definitive ingress list in ansible/inventory/host_vars/log.yml has no TCP/6000 rule for peers.agentmail.ipv6. Since the log host enforces a drop policy, all Stalwart and host logs from Agent Mail will be rejected even though the inventory and generated network-flow documentation claim that logging is enabled; add the corresponding source-scoped firewall rule on log.
Useful? React with 👍 / 👎.
| monitoring_register: true | ||
| monitoring_role: agent-mail | ||
| monitoring_display_name: "agentmail (Stalwart, API-only)" | ||
| monitoring_node_listen: "[{{ peers.agentmail.ipv6 }}]:9100" | ||
| monitoring_extra_services: |
There was a problem hiding this comment.
Monitor the dedicated backup filesystem
Once backups are enabled, the generated Icinga host object contains only the inherited disk / check; /mnt/agent-mail-backup is absent from monitoring_disks, and there is also no service check for the backup unit. If this dedicated volume fills, the backup script will begin exiting at its free-space guard without any configured disk alert, so backups can stop silently while node and Stalwart readiness remain green; add the backup mount to this host's monitored disks.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3bcbbf6964
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - rtr_routing | ||
| - networkd_resolved | ||
| - mail_openbsd | ||
| - agent_mail |
There was a problem hiding this comment.
Wire Agent Mail secrets into the apply runner
When an operator follows the documented live agent_mail workflow, the role receives agent_mail_apply=true and immediately requires the three AGENT_MAIL_* environment variables, but the workflow only sources /etc/github-runner/secrets.env, and ansible/roles/vault_agent/templates/github-runner.env.ctmpl.j2 exports none of those variables. A repo-wide search found no other workflow injection path, so every real Agent Mail apply reaches validate.yml with empty secret lookups and fails before staging the host; add the corresponding Vault-backed runner mappings before exposing this workflow option.
Useful? React with 👍 / 👎.
| if ! backup_device="$(stat --file-system --format=%d "$backup_dir")" || | ||
| ! root_device="$(stat --file-system --format=%d /)" || | ||
| ! data_device="$(stat --file-system --format=%d /var/lib/stalwart)"; then |
There was a problem hiding this comment.
Compare filesystem IDs instead of free-inode counts
On GNU stat, the documented --file-system format %d is “free file nodes in file system,” while %i is the filesystem ID. These three values therefore fluctuate with inode activity rather than identifying devices, so the runtime safety check can reject a valid dedicated volume or, if inode counts change between calls, accept a bind mount backed by the data/root filesystem; use an actual filesystem/device identifier for this comparison.
Useful? React with 👍 / 👎.
| Type=oneshot | ||
| ExecStart=/usr/local/sbin/agent-mail-backup |
There was a problem hiding this comment.
Remove the default timeout from the backup oneshot
Once stopping and compressing the Stalwart data takes longer than the systemd manager's default start timeout—normally 90 seconds, as confirmed by systemd-analyze cat-config systemd/system.conf—this Type=oneshot service is terminated before producing an archive. The script explicitly allows 120 seconds just for Docker shutdown and may then archive an 80 GiB data disk, so scheduled backups can repeatedly fail on a realistically populated host; set an unlimited or appropriately sized TimeoutStartSec for this unit.
Useful? React with 👍 / 👎.
| - agent_mail_backup_restore_verified | bool | ||
| - agent_mail_canaries_verified | bool | ||
| - agent_mail_recovery_admin_secret | length == 0 | ||
| - agent_mail_public_ipv4 is match('^(?:(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])$') |
There was a problem hiding this comment.
Reject wildcard and non-public SMTP bind addresses
At public launch this regex accepts syntactically valid but unsuitable values such as 0.0.0.0, loopback, private, multicast, and broadcast addresses. In particular, agent_mail_public_ipv4=0.0.0.0 passes every assertion and renders Docker's 0.0.0.0:25:25 wildcard binding, exposing SMTP on every IPv4 assigned to the host despite the dedicated-address requirement and the separate failover-address comparison; validate that the value is a usable, explicitly assigned public unicast address.
Useful? React with 👍 / 👎.
| - name: Assert apply/start/bootstrap/backup sequencing | ||
| ansible.builtin.assert: | ||
| that: | ||
| - not (agent_mail_start | bool) or (agent_mail_apply | bool) |
There was a problem hiding this comment.
Decouple desired start state from the transient apply gate
When the reviewed rollout enables agent_mail_start in inventory while leaving the safe execution gate false, every render-only invocation fails this assertion before the protected workflow can supply its -e agent_mail_apply=true. Setting agent_mail_apply: true in inventory instead makes an ordinary untagged playbook invocation mutate the host, contradicting the documented safe default; the public-launch assertion repeats the same coupling, so bootstrap and launch cannot be represented safely. Validate desired-state relationships independently of the transient apply gate.
Useful? React with 👍 / 👎.
| was_running=0 | ||
|
|
||
| running_services="" | ||
| if ! running_services="$(docker compose --file "$compose_file" ps --services --status running)"; then |
There was a problem hiding this comment.
Quiesce containers that are restarting before backup
If Stalwart is in Docker's restarting state when the timer fires, docker compose ps --services --status running succeeds but omits the service, so this condition skips docker compose stop and immediately archives RocksDB while the restart policy can launch it again. That can produce an inconsistent snapshot precisely during a service failure; detect every existing non-stopped container state and stop it before archiving rather than treating only the instantaneously running state as active.
Useful? React with 👍 / 👎.
| notes: "Agent Mail quiesced-backup timer is active" | ||
| vars: | ||
| prom_instance: "[{{ peers.agentmail.ipv6 }}]:9100" | ||
| systemd_unit: agent-mail-backup.timer |
There was a problem hiding this comment.
Monitor successful backups instead of timer liveness
After agent-mail-backup.service fails because of a full disk, tar error, or restart failure, agent-mail-backup.timer remains active and this Prometheus query continues returning healthy indefinitely. The newly added check therefore cannot detect that no usable snapshots are being produced; monitor the service's failed state and the age or success marker of the latest completed archive rather than only the timer unit.
Useful? React with 👍 / 👎.
| ct status dnat oifname "{{ agent_mail_bridge_name }}" tcp dport 25 counter accept comment "Agent Mail SMTP reception DNAT" | ||
| iifname "{{ agent_mail_bridge_name }}" tcp dport 25 counter accept comment "Agent Mail SMTP delivery" | ||
| {% else %} | ||
| iifname "{{ agent_mail_bridge_name }}" tcp dport 25 counter drop comment "Agent Mail outbound SMTP kill switch" |
There was a problem hiding this comment.
Place the SMTP kill switch before established accepts
When the SMTP firewall gate is disabled while Stalwart still has an established outbound connection—for example before Compose shutdown completes or if that apply fails—the forward chain's earlier ct state established,related accept terminates evaluation before this drop rule. The advertised emergency kill switch therefore permits that connection to keep delivering messages; evaluate the disabled-gate drop before the established-state accept or explicitly terminate matching conntrack state.
Useful? React with 👍 / 👎.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c2eef072ee
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - agent_mail_dns_verified | bool | ||
| - agent_mail_ptr_verified | bool | ||
| - agent_mail_backup_restore_verified | bool | ||
| - agent_mail_canaries_verified | bool |
There was a problem hiding this comment.
Separate canary access from the public launch gate
Requiring agent_mail_canaries_verified before agent_mail_public_enabled can become true makes the documented pre-launch inbound/outbound canaries impossible: while the public gate is false, firewall_forward_pre_established_raw_nft drops outbound TCP/25 and the synchronized firewall rules omit inbound TCP/25. An operator therefore has to assert that the Gmail/Outlook and inbound journeys passed before the network permits them, or bypass the reviewed firewall. Add a restricted temporary canary gate/path that can be enabled before this assertion is satisfied.
Useful? React with 👍 / 👎.
| agent_mail_image: "stalwartlabs/stalwart:v0.16.4@sha256:c8aee803933a643558a9afaa3c208d4175a4ac09884f555b821aa5df1e89230c" | ||
| agent_mail_packages: | ||
| - docker.io | ||
| - docker-compose-v2 |
There was a problem hiding this comment.
Install the Debian Compose package name
The documented target is Debian 13/trixie, whose repository packages Compose v2 as docker-compose, not Ubuntu's docker-compose-v2 package name. On a fresh Agent Mail VM this apt package list therefore has no installation candidate, so the first apply stops before any directories, validation, or container setup occurs; use the Debian package that provides the docker compose plugin.
Useful? React with 👍 / 👎.
|
|
||
| - name: Start the staged Agent Mail container | ||
| ansible.builtin.command: | ||
| argv: [docker, compose, --file, "{{ agent_mail_compose_file }}", up, --detach, --remove-orphans] |
There was a problem hiding this comment.
Serialize normal applies with the backup service
When agent_mail_start and agent_mail_backup_enabled remain true, the earlier tasks do not stop an in-flight backup, yet this command can start or recreate Stalwart while the timer job has deliberately stopped it and is archiving RocksDB. An apply overlapping the daily backup therefore makes the supposedly quiesced archive inconsistent and can invalidate restore evidence; every Compose lifecycle operation should wait for or stop the backup service, or share its lock.
Useful? React with 👍 / 👎.
| # Dedicated API-only Agent Mail service. This is separate from the | ||
| # corporate OpenBSD mail host at ::90. While it is also a member of | ||
| # `staged`, fleet-wide drift/apply automation excludes it. | ||
| agentmail: |
There was a problem hiding this comment.
Exclude staged hosts from unbounded apply runs
Adding the not-yet-reachable host to linux and all does not actually exclude it from the protected apply workflow: apply.yml defaults limit to an empty string and omits --limit in that case. A normal fleet-wide firewall, monitoring, or logs apply will therefore contact agentmail despite its staged membership, fail or stall on the planned VM, and can leave a serialized deployment partially applied; make the workflow's empty-limit default exclude staged (and ci-pr) just like the drift sweep.
Useful? React with 👍 / 👎.
|
@codex review |
|
Codex Review: Didn't find any major issues. You're on a roll. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
agentmail.hyrule.hostand keep custom-domain records in the Cloud DNS control planeWhy
Agent Email Accounts need a dedicated, API-only mail plane with explicit abuse, deliverability, retention, restore, and public-network gates. It must remain separate from the corporate mail system and impossible to launch through an incidental playbook run.
Rollout safety
falseDependencies
Validation
uv run pytest -q— 144 passed, 1 skipped, 233 subtestsscripts/ci/iac-static.sh— 142 tests passed and all DNS zones validated