Skip to content

Stage dedicated Agent Mail infrastructure - #467

Open
Svaag wants to merge 8 commits into
mainfrom
feat/agent-mail-campaign
Open

Stage dedicated Agent Mail infrastructure#467
Svaag wants to merge 8 commits into
mainfrom
feat/agent-mail-campaign

Conversation

@Svaag

@Svaag Svaag commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Summary

  • stage a dedicated Debian 13 Agent Mail host running digest-pinned Stalwart v0.16.4
  • add fail-closed Ansible role, desired state, backup timer, monitoring, logs, firewall policy, network flows, DNS, and Vault-rendered Cloud settings
  • restrict JMAP/admin access to API, monitoring, operations, and VPN sources
  • reserve public port 25 for a separately approved launch while excluding submission, IMAP, POP, and webmail exposure
  • scope Knot dynamic updates to agentmail.hyrule.host and keep custom-domain records in the Cloud DNS control plane

Why

Agent Email Accounts need a dedicated, API-only mail plane with explicit abuse, deliverability, retention, restore, and public-network gates. It must remain separate from the corporate mail system and impossible to launch through an incidental playbook run.

Rollout safety

  • every apply, service start, public network, SMTP, bootstrap, backup, legal, abuse, DNS/PTR, restore, and canary gate remains false
  • no production application SHA pin is changed
  • generated retention state leaves mailbox deletion authoritative to the Cloud JMAP sweep
  • this PR stages infrastructure only; it does not deploy anything

Dependencies

Validation

  • uv run pytest -q — 144 passed, 1 skipped, 233 subtests
  • scripts/ci/iac-static.sh — 142 tests passed and all DNS zones validated
  • Knot playbook syntax validation passed during implementation
  • advisory target-binary/module warnings and the unprivileged nft skip remain expected on this runner

@Svaag

Svaag commented Jul 19, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@github-actions

Copy link
Copy Markdown
Contributor

Failed to generate code suggestions for PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 98104567bb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +40 to +41
agentmail:
ansible_host: 2a0c:b641:b50:2::110

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the planned host out of fleet-wide play targets

Until Stage 1 provisions this VM, adding agentmail to linux makes it part of the canonical nightly/post-merge drift sweep: scripts/ci/check-drift.sh limits to all:!ci-pr, while the firewall, monitoring, and logs playbooks target all or linux. Those applies will attempt SSH to the unprovisioned ::110 address and fail, contrary to the runbook's instruction to apply monitoring only after the host is reachable. Exclude the staged host from fleet-wide limits until provisioning or model it outside the active groups.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc. The planned host is a member of the staged group, and the canonical drift limit is now all:!ci-pr:!staged. Contract tests verify both the membership and exclusion.

{% endif %}
{% if agent_mail_public_enabled | bool %}
- "[{{ agent_mail_overlay_ipv6 }}]:25:25/tcp"
- "{{ agent_mail_public_ipv4 }}:25:25/tcp"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow Docker-forwarded SMTP through the forward chain

When the public IPv4 gate is enabled on the normal Docker bridge setup, this published port is DNATed to the container and traverses the host's forward hook rather than input. The generated Agent Mail nftables rules put the TCP/25 allowance only in input, while ansible/roles/firewall/templates/nftables.conf.j2 leaves forward at policy drop with only established/related traffic allowed and agentmail.yml supplies no forward exception. Consequently new external IPv4 SMTP connections are dropped even with every launch gate enabled; add a scoped Docker-forward rule or use host networking.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc and reinforced in 65ec286. The Compose bridge name is deterministic and the nftables forward chain has launch-gated SMTP DNAT and delivery allowances. The IPv6-capable bridge and Docker tables now also survive later firewall reloads.

changed_when: >-
'Started' in _agent_mail_up.stderr or 'Created' in _agent_mail_up.stderr or
'Recreated' in _agent_mail_up.stderr
when: agent_mail_start | bool

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Stop Stalwart when the start gate is disabled

On a host where Stalwart has previously been started, changing agent_mail_start back to false merely skips this compose up task; there is no corresponding compose stop or down, and the service has restart: unless-stopped. The container and its currently published listeners therefore remain active despite the supposedly disabled service-start gate, making rollback of that safety control ineffective.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc. When agent_mail_start is false, the apply path runs docker compose down with orphan removal and a 120-second timeout. Commit 65ec286 also makes the emergency runbook disable backup and public gates atomically so validation reaches this stop task.

Comment thread ansible/roles/agent_mail/README.md Outdated
Comment on lines +130 to +133
1. Set Hyrule Cloud `MAIL_ENABLED=false` to stop new activation/send traffic.
2. Set `agent_mail_public_enabled`, its SMTP firewall twin, and both TCP/25
rule fields false; apply Compose and nftables. This preserves stored mail
while stopping inbound/outbound public delivery.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Block outbound SMTP during emergency shutdown

In the documented abuse/emergency scenario, disabling MAIL_ENABLED stops new Cloud requests and disabling agent_mail_public_enabled removes inbound port mappings, but neither action blocks the running Stalwart container's unrestricted egress; the host firewall output policy remains accept. Already queued mail can therefore continue outbound delivery while operators believe public delivery is stopped. The shutdown procedure needs an outbound TCP/25 gate or an explicit service stop before making this claim.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc and clarified in 65ec286. The stopped-public state installs an outbound TCP/25 forward-chain kill switch, and emergency shutdown now atomically disables start, backup, public SMTP, and firewall gates before taking the container down.

- agent_mail_backup_restore_verified | bool
- agent_mail_canaries_verified | bool
- agent_mail_recovery_admin_secret | length == 0
- agent_mail_public_ipv4 is match('^(?:[0-9]{1,3}\\.){3}[0-9]{1,3}$')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate that the launch address is a dedicated IPv4

At public launch this regex accepts invalid addresses such as 999.999.999.999 and also accepts the explicitly forbidden corporate mail_failover_ipv4 (51.91.236.215). The former passes every readiness assertion only for Compose to fail while binding the port, and the latter defeats the role's stated isolation boundary. Parse the value as an IPv4 address and explicitly reject mail_failover_ipv4 before allowing the public gate.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc. Public launch now requires an IPv4 whose octets are each in the 0-255 range and explicitly rejects mail_failover_ipv4.

-cf "$partial" \
etc/stalwart var/lib/stalwart
mv "$partial" "$archive"
sha256sum "$archive" > "$archive.sha256"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Write portable checksums for off-host restores

Because $archive is an absolute path, the checksum file records /var/backups/agent-mail/stalwart-…tar.zst rather than the archive basename. After the archive and checksum are copied to the required off-host repository or isolated restore host, a normal sha256sum -c therefore looks back under /var/backups/agent-mail and fails or checks an unrelated local file. Generate the checksum from inside $backup_dir or write the basename so the paired artifacts remain verifiable after transfer.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc. The backup script changes into the backup directory and writes the checksum against the archive basename, so the archive and checksum remain portable together.

Comment thread ansible/roles/agent_mail/README.md Outdated
`agent_mail_bootstrap_firewall_enabled`, and the TCP/8080 rule's `enabled`
field. Keep public SMTP false. Supply a new temporary recovery password of
at least 32 characters.
2. Apply this role and connect only from the ops prefix or VPN. Submit the

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Reapply nftables when enabling the bootstrap listener

Stage 1 applies the firewall while the TCP/8080 rule is disabled, but after Stage 2 changes that rule this instruction runs only the agent_mail role, which does not include or reload the firewall role. The live nftables input chain therefore still lacks the 8080 allowance and the documented bootstrap curl remains blocked even though all three inventory controls are true. Explicitly apply playbooks/firewall.yml with firewall_apply=true after enabling the rule and before attempting bootstrap.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 89601cc. The bootstrap runbook now applies playbooks/firewall.yml with firewall_apply=true before the Agent Mail role and before curl, then reapplies the firewall when bootstrap is disabled.

@Svaag
Svaag marked this pull request as draft July 19, 2026 00:33
@Svaag
Svaag marked this pull request as ready for review July 19, 2026 12:21
@github-actions

Copy link
Copy Markdown
Contributor

Failed to generate code suggestions for PR

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 89601cc375

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +27 to +30
networks:
default:
name: agent-mail
driver: bridge

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Enable IPv6 connectivity for the Stalwart container

When Stage 2 starts this project, the custom Docker bridge is created without enable_ipv6, so the Stalwart container receives no IPv6 address or route under Docker's default configuration (Compose network documentation). The bootstrap plan nevertheless configures Knot using the literal IPv6 address 2a0c:b641:b50:2::10; consequently TSIG updates, ACME DNS reconciliation, and certificate bootstrap cannot reach Knot from the container. Configure routed IPv6 with a source accepted by the Knot ACL, or use an IPv4 endpoint reachable from this bridge.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 65ec286. The Agent Mail bridge now has enable_ipv6=true with an explicit ULA IPAM subnet, and the role enables IPv6 forwarding. Docker default NAT mode makes outbound IPv6 use the host overlay address already accepted by the Knot ACL. The rendered Compose model validates successfully.

Comment thread configs/mon/prometheus.yml Outdated
- "[2a0c:b641:b50:2::d0]:9100" # ci (privileged runner)
- "[2a0c:b641:b50:2::e0]:9100" # netproxy
- "[2a0c:b641:b50:2::f0]:9100" # loop (Engineering Loop / Knowledge MCP)
- "[2a0c:b641:b50:2::110]:9100" # agentmail (dedicated Stalwart)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the staged host out of live Prometheus targets

Although check-drift.sh now excludes the staged inventory group, this target is part of the Prometheus configuration applied directly to mon, so that exclusion does not protect it. After the normal post-merge Prometheus apply, the known-unprovisioned address is scraped and the new AgentMailNodeUnavailable and AgentMailStalwartMetricsUnavailable rules become critical after five minutes, contradicting the runbook's instruction to install monitoring only after reachability. Add these targets and alerts only when the host is provisioned, or suppress them behind an explicit staging gate.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 65ec286. The unprovisioned node target and agent-mail-stalwart scrape job were removed from live Prometheus configuration. The runbook requires adding both in the reviewed activation change after reachability and staged-group removal, so the committed alert rules remain dormant meanwhile.

that:
- not (agent_mail_start | bool) or (agent_mail_apply | bool)
- not (agent_mail_bootstrap_enabled | bool) or (agent_mail_start | bool)
- not (agent_mail_backup_enabled | bool) or (agent_mail_start | bool)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Allow the emergency stop while backups are enabled

In the launched state, agent_mail_backup_enabled must be true, but the documented emergency step changes only agent_mail_start to false and applies the role. This assertion therefore fails before apply.yml can reach the new docker compose down task, leaving the container and queued outbound delivery running during the shutdown attempt. The new stop task does not resolve the earlier rollback issue because validation executes first; either disable the backup/public gates atomically in the shutdown procedure or permit the role to stop both the timer and container in one apply.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 65ec286. Emergency shutdown now requires one atomic inventory change setting start, backup, public SMTP, its firewall twin, and both TCP/25 rules false before applying. Validation therefore permits the stop path, which stops both the backup timer and container.

Comment on lines +13 to +14
ports:
- "[{{ agent_mail_overlay_ipv6 }}]:443:443/tcp"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve Docker networking across nftables reloads

Once Stalwart is running, any later firewall apply loads ansible/roles/firewall/templates/nftables.conf.j2, whose line 6 executes flush ruleset; this removes the NAT, forwarding, and masquerade rules Docker creates for bridge networks (Docker firewall documentation). The firewall handler reloads nftables without restarting Docker or reconverging this Compose project, so these published listeners and container egress can remain broken until Docker recreates its rules. Preserve Docker-managed tables during reload or explicitly restore the Compose network after every firewall apply.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 65ec286. Agent Mail opts into preserving external nftables tables, so reload destroys and recreates only the role-owned inet filter table instead of flushing the ruleset. An isolated behavioral check applied the file twice and confirmed a simulated Docker table survived.

@Svaag

Svaag commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65ec286bc0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".


- name: Stop and remove Agent Mail when the start gate is disabled
ansible.builtin.command:
argv: [docker, compose, --file, "{{ agent_mail_compose_file }}", down, --remove-orphans, --timeout, "120"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Stop the backup service before taking Stalwart down

When an emergency shutdown overlaps the daily backup, this command can remove the container while agent-mail-backup.service is still running; the role only stops the timer afterward, and stopping a timer does not stop the service it already activated. The backup script's EXIT trap then runs docker compose up --detach stalwart, resurrecting the service after the shutdown path completed. Stop agent-mail-backup.service as well as the timer before invoking docker compose down so an in-flight backup cannot undo the shutdown.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a807ddb. The role now inspects the installed backup units at the start of apply, stops and disables the timer first, then waits for any in-flight backup service to stop before the later Compose shutdown. This lets the backup EXIT restart finish before docker compose down, and the same quiesce path runs when backups are disabled while Stalwart remains up. An ordering regression test covers timer, service, then teardown.

Comment thread scripts/ci/check-drift.sh
fi

limit="${CHECK_DRIFT_LIMIT:-all:!ci-pr}"
limit="${CHECK_DRIFT_LIMIT:-all:!ci-pr:!staged}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Add Agent Mail to the canonical drift playbook list

After provisioning, removing agentmail from staged does not actually bring its role under the canonical drift/apply sweep: this limit would then include the host, but default_playbooks still omits agent_mail. Consequently, drift in the Compose project, secret environment file, backup units, or service state is never detected or auto-applied even though the runbook says removing the staged membership makes the host part of the managed fleet. Add agent_mail to the canonical list, with whatever secret-safe check-mode handling the role requires.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a807ddb. agent_mail is now part of the canonical drift playbook list. The existing all:!ci-pr:!staged limit still excludes the unprovisioned host; before it leaves staged, the runbook requires its Vault values in the privileged runner environment. Secret validation and the runtime env template remain no_log. I also ran the canonical agent_mail drift entry and confirmed the current staged exclusion succeeds without requesting secrets.

Comment on lines +3 to +6
STALWART_DNS_TSIG_SECRET={{ agent_mail_dns_tsig_secret }}
STALWART_WEBHOOK_SECRET={{ agent_mail_webhook_secret }}
{% if agent_mail_bootstrap_enabled | bool %}
STALWART_RECOVERY_ADMIN=admin:{{ agent_mail_recovery_admin_secret }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve special characters in generated secrets

When any Vault-generated secret contains $ or another Compose env-file metacharacter, writing it unquoted here changes the value before it reaches Stalwart; Docker documents that unquoted env-file values undergo interpolation in its env-file syntax. This can make bootstrap authentication fail or cause webhook signatures and TSIG authentication to use a different secret from their peers. Render these values in a literal/raw-safe form or validate and encode them into a restricted character set.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a807ddb. Runtime Compose env values are now single-quoted, so dollar signs, hashes, spaces, and equals signs remain literal instead of being interpolated. Apply validation fails closed for quote, backslash, or line-break values and directs operators to unwrapped base64 or base64url; secret-bearing tasks stay no_log. Regression coverage renders dollar-sign and punctuation-bearing secrets.

Comment on lines +32 to +34
tar --acls --xattrs --numeric-owner --zstd -C / \
-cf "$partial" \
etc/stalwart var/lib/stalwart

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move full mailbox snapshots off the data disk

Once mailbox usage grows beyond roughly 5 GB, retaining 14 daily full-tree archives under /var/backups on the same planned 80 GB disk can exhaust the filesystem even while all live mailbox quotas are respected; the configured maximum is 25 mailboxes at 1 GB each, so incompressible mail and attachments can consume about 25 GB per archive and fill the host after only a few runs. A full filesystem can take RocksDB and mail delivery down, so these snapshots need a separate-capacity volume or deduplicating/incremental off-host storage with an explicit local space bound.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a807ddb. Full snapshots now target a dedicated mount at /mnt/agent-mail-backup instead of the 80 GiB mail-data disk. The role and backup script require an actual mount, the systemd unit declares RequiresMountsFor, and the script enforces at least 100 GiB capacity, 32 GiB free space, two-day retention, and preflight cleanup before Stalwart is stopped. The rollout runbook now provisions and verifies that separate volume plus off-host transfer and restore evidence.

@Svaag

Svaag commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a807ddb2ce

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

archive="$backup_dir/stalwart-$timestamp.tar.zst"
was_running=0

if docker compose --file "$compose_file" ps --services --status running | grep -Fxq stalwart; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Fail closed when container-state detection errors

If docker compose ps fails—for example during a transient Docker daemon error—the pipeline is merely treated as a false if condition, so the script proceeds to archive /var/lib/stalwart without knowing whether Stalwart is still running. That can produce an inconsistent RocksDB backup while the timer still reports success; explicitly distinguish an empty running-service result from a nonzero Compose command before starting the archive.

Useful? React with 👍 / 👎.

Comment thread ansible/playbooks/agent_mail.yml Outdated
Comment on lines +5 to +6
# ansible-playbook playbooks/agent_mail.yml --tags apply \
# -e '{"agent_mail_apply":true}' --limit agentmail

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Register Agent Mail in the protected apply workflow

When an operator reaches a rollout stage requiring this apply, the protected manual workflow cannot dispatch it: the workflow_dispatch.inputs.playbook.options list in .github/workflows/apply.yml contains mail_openbsd but not agent_mail. The direct command documented here therefore bypasses the repository's production environment gate and runner secret-loading path; add this playbook to that workflow before relying on it for the controlled rollout.

Useful? React with 👍 / 👎.

exec 9>"$lock_file"
flock -n 9 || exit 0

if ! mountpoint --quiet "$backup_dir"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Verify the backup mount uses a separate filesystem

If /mnt/agent-mail-backup is accidentally a bind mount or another mount of the root/data filesystem, this check succeeds and the subsequent capacity check can also pass, leaving the supposedly dedicated backup on the same failure domain as the mailbox data. As mountpoint --help states, it only checks whether the path is a mountpoint; compare its filesystem device/source with / and /var/lib/stalwart to enforce the runbook's distinct-volume requirement.

Useful? React with 👍 / 👎.

http_ipv6: true
http_timeout: 10

logs_register: true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow Agent Mail traffic through the log firewall

When the logs role is applied to the new host, its generated Vector configuration sends journald events to log:6000, but the definitive ingress list in ansible/inventory/host_vars/log.yml has no TCP/6000 rule for peers.agentmail.ipv6. Since the log host enforces a drop policy, all Stalwart and host logs from Agent Mail will be rejected even though the inventory and generated network-flow documentation claim that logging is enabled; add the corresponding source-scoped firewall rule on log.

Useful? React with 👍 / 👎.

Comment on lines +77 to +81
monitoring_register: true
monitoring_role: agent-mail
monitoring_display_name: "agentmail (Stalwart, API-only)"
monitoring_node_listen: "[{{ peers.agentmail.ipv6 }}]:9100"
monitoring_extra_services:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Monitor the dedicated backup filesystem

Once backups are enabled, the generated Icinga host object contains only the inherited disk / check; /mnt/agent-mail-backup is absent from monitoring_disks, and there is also no service check for the backup unit. If this dedicated volume fills, the backup script will begin exiting at its free-space guard without any configured disk alert, so backups can stop silently while node and Stalwart readiness remain green; add the backup mount to this host's monitored disks.

Useful? React with 👍 / 👎.

@Svaag

Svaag commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

All five findings from the latest review are addressed in 3bcbbf6. Verification: 148 IaC tests plus 233 subtests passed; IaC static checks, yamllint, ansible-lint, backup shell syntax, full rendering, and render idempotence passed.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3bcbbf6964

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- rtr_routing
- networkd_resolved
- mail_openbsd
- agent_mail

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wire Agent Mail secrets into the apply runner

When an operator follows the documented live agent_mail workflow, the role receives agent_mail_apply=true and immediately requires the three AGENT_MAIL_* environment variables, but the workflow only sources /etc/github-runner/secrets.env, and ansible/roles/vault_agent/templates/github-runner.env.ctmpl.j2 exports none of those variables. A repo-wide search found no other workflow injection path, so every real Agent Mail apply reaches validate.yml with empty secret lookups and fails before staging the host; add the corresponding Vault-backed runner mappings before exposing this workflow option.

Useful? React with 👍 / 👎.

Comment on lines +19 to +21
if ! backup_device="$(stat --file-system --format=%d "$backup_dir")" ||
! root_device="$(stat --file-system --format=%d /)" ||
! data_device="$(stat --file-system --format=%d /var/lib/stalwart)"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compare filesystem IDs instead of free-inode counts

On GNU stat, the documented --file-system format %d is “free file nodes in file system,” while %i is the filesystem ID. These three values therefore fluctuate with inode activity rather than identifying devices, so the runtime safety check can reject a valid dedicated volume or, if inode counts change between calls, accept a bind mount backed by the data/root filesystem; use an actual filesystem/device identifier for this comparison.

Useful? React with 👍 / 👎.

Comment on lines +8 to +9
Type=oneshot
ExecStart=/usr/local/sbin/agent-mail-backup

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Remove the default timeout from the backup oneshot

Once stopping and compressing the Stalwart data takes longer than the systemd manager's default start timeout—normally 90 seconds, as confirmed by systemd-analyze cat-config systemd/system.conf—this Type=oneshot service is terminated before producing an archive. The script explicitly allows 120 seconds just for Docker shutdown and may then archive an 80 GiB data disk, so scheduled backups can repeatedly fail on a realistically populated host; set an unlimited or appropriately sized TimeoutStartSec for this unit.

Useful? React with 👍 / 👎.

- agent_mail_backup_restore_verified | bool
- agent_mail_canaries_verified | bool
- agent_mail_recovery_admin_secret | length == 0
- agent_mail_public_ipv4 is match('^(?:(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])$')

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject wildcard and non-public SMTP bind addresses

At public launch this regex accepts syntactically valid but unsuitable values such as 0.0.0.0, loopback, private, multicast, and broadcast addresses. In particular, agent_mail_public_ipv4=0.0.0.0 passes every assertion and renders Docker's 0.0.0.0:25:25 wildcard binding, exposing SMTP on every IPv4 assigned to the host despite the dedicated-address requirement and the separate failover-address comparison; validate that the value is a usable, explicitly assigned public unicast address.

Useful? React with 👍 / 👎.

- name: Assert apply/start/bootstrap/backup sequencing
ansible.builtin.assert:
that:
- not (agent_mail_start | bool) or (agent_mail_apply | bool)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Decouple desired start state from the transient apply gate

When the reviewed rollout enables agent_mail_start in inventory while leaving the safe execution gate false, every render-only invocation fails this assertion before the protected workflow can supply its -e agent_mail_apply=true. Setting agent_mail_apply: true in inventory instead makes an ordinary untagged playbook invocation mutate the host, contradicting the documented safe default; the public-launch assertion repeats the same coupling, so bootstrap and launch cannot be represented safely. Validate desired-state relationships independently of the transient apply gate.

Useful? React with 👍 / 👎.

was_running=0

running_services=""
if ! running_services="$(docker compose --file "$compose_file" ps --services --status running)"; then

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Quiesce containers that are restarting before backup

If Stalwart is in Docker's restarting state when the timer fires, docker compose ps --services --status running succeeds but omits the service, so this condition skips docker compose stop and immediately archives RocksDB while the restart policy can launch it again. That can produce an inconsistent snapshot precisely during a service failure; detect every existing non-stopped container state and stop it before archiving rather than treating only the instantaneously running state as active.

Useful? React with 👍 / 👎.

notes: "Agent Mail quiesced-backup timer is active"
vars:
prom_instance: "[{{ peers.agentmail.ipv6 }}]:9100"
systemd_unit: agent-mail-backup.timer

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Monitor successful backups instead of timer liveness

After agent-mail-backup.service fails because of a full disk, tar error, or restart failure, agent-mail-backup.timer remains active and this Prometheus query continues returning healthy indefinitely. The newly added check therefore cannot detect that no usable snapshots are being produced; monitor the service's failed state and the age or success marker of the latest completed archive rather than only the timer unit.

Useful? React with 👍 / 👎.

ct status dnat oifname "{{ agent_mail_bridge_name }}" tcp dport 25 counter accept comment "Agent Mail SMTP reception DNAT"
iifname "{{ agent_mail_bridge_name }}" tcp dport 25 counter accept comment "Agent Mail SMTP delivery"
{% else %}
iifname "{{ agent_mail_bridge_name }}" tcp dport 25 counter drop comment "Agent Mail outbound SMTP kill switch"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Place the SMTP kill switch before established accepts

When the SMTP firewall gate is disabled while Stalwart still has an established outbound connection—for example before Compose shutdown completes or if that apply fails—the forward chain's earlier ct state established,related accept terminates evaluation before this drop rule. The advertised emergency kill switch therefore permits that connection to keep delivering messages; evaluate the disabled-gate drop before the established-state accept or explicitly terminate matching conntrack state.

Useful? React with 👍 / 👎.

@Svaag

Svaag commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c2eef072ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

- agent_mail_dns_verified | bool
- agent_mail_ptr_verified | bool
- agent_mail_backup_restore_verified | bool
- agent_mail_canaries_verified | bool

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Separate canary access from the public launch gate

Requiring agent_mail_canaries_verified before agent_mail_public_enabled can become true makes the documented pre-launch inbound/outbound canaries impossible: while the public gate is false, firewall_forward_pre_established_raw_nft drops outbound TCP/25 and the synchronized firewall rules omit inbound TCP/25. An operator therefore has to assert that the Gmail/Outlook and inbound journeys passed before the network permits them, or bypass the reviewed firewall. Add a restricted temporary canary gate/path that can be enabled before this assertion is satisfied.

Useful? React with 👍 / 👎.

agent_mail_image: "stalwartlabs/stalwart:v0.16.4@sha256:c8aee803933a643558a9afaa3c208d4175a4ac09884f555b821aa5df1e89230c"
agent_mail_packages:
- docker.io
- docker-compose-v2

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Install the Debian Compose package name

The documented target is Debian 13/trixie, whose repository packages Compose v2 as docker-compose, not Ubuntu's docker-compose-v2 package name. On a fresh Agent Mail VM this apt package list therefore has no installation candidate, so the first apply stops before any directories, validation, or container setup occurs; use the Debian package that provides the docker compose plugin.

Useful? React with 👍 / 👎.


- name: Start the staged Agent Mail container
ansible.builtin.command:
argv: [docker, compose, --file, "{{ agent_mail_compose_file }}", up, --detach, --remove-orphans]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Serialize normal applies with the backup service

When agent_mail_start and agent_mail_backup_enabled remain true, the earlier tasks do not stop an in-flight backup, yet this command can start or recreate Stalwart while the timer job has deliberately stopped it and is archiving RocksDB. An apply overlapping the daily backup therefore makes the supposedly quiesced archive inconsistent and can invalidate restore evidence; every Compose lifecycle operation should wait for or stop the backup service, or share its lock.

Useful? React with 👍 / 👎.

# Dedicated API-only Agent Mail service. This is separate from the
# corporate OpenBSD mail host at ::90. While it is also a member of
# `staged`, fleet-wide drift/apply automation excludes it.
agentmail:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exclude staged hosts from unbounded apply runs

Adding the not-yet-reachable host to linux and all does not actually exclude it from the protected apply workflow: apply.yml defaults limit to an empty string and omits --limit in that case. A normal fleet-wide firewall, monitoring, or logs apply will therefore contact agentmail despite its staged membership, fail or stall on the planned VM, and can leave a serialized deployment partially applied; make the workflow's empty-limit default exclude staged (and ci-pr) just like the drift sweep.

Useful? React with 👍 / 👎.

@Svaag

Svaag commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: fb6179239b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant