jobs(#864): retention holds untriaged Missing rows past 90d + dlq_untriaged_missing alert

[radandevist]

Closes #864

Implementer: Ox Alpha (stealth/ox-alpha via Nous Portal, max effort, jcode). Reviewer: pending adversarial review.

Problem (K-2, #852 design §11)

job-dead-letter-retention hard-deletes any job_dead_letter row older than JOB_DEAD_LETTER_RETENTION_DAYS (90 default). A future 4 Missing row records an integrity anomaly (prepared state that should exist does not); deleting it at the age floor silently clears its alert with nobody having looked.

Owner decision on #864 (Radan, 2026-08-01): acknowledgement is required before a Missing row becomes retention-eligible; ship the flag + exemption now; the operator surface arrives with #636; and an alert must watch the unacknowledged count between the two so an accumulating class cannot become a silent second starvation.

Note: the external-state machinery (external_state_status, classifier, job_dead_letter_events) that would produce Missing rows does not exist yet — this lands the rule that governs them, implementable today, plus the reserved jobs.missing. job_type prefix convention for those producers (declared on JobDeadLetter).

What ships (3 commits, TDD order)

1. feat(api) — expand-only migration AddDeadLetterTriage: three nullable columns on job_dead_letter (triaged_at, triaged_by, triage_note) + partial index ix_job_dead_letter_untriaged_missing on (failed_at) WHERE triaged_at IS NULL AND job_type LIKE 'jobs.missing.%'. No NOT NULL, no drops: rolling-deploy safe (ci-migration-expand-contract green). "Triaged" means triaged_at IS NOT NULL. No writer exists before A5 — Staff job-visibility dashboard #636 — deliberate.
2. test(api) — the four failing-first specs + the inert seams they compile against (public CountUntriagedMissingRowsAsync seam, sample field). Proven RED on this exact tree (transcript below).
3. fix(api) — the behavioral fix: retention delete predicate gains (triaged_at IS NOT NULL OR job_type NOT LIKE 'jobs.missing.%'); every pass logs how many rows it HELD back; the monitor counts the same predicate inside its existing single-statement sample (dlq_metrics CTE), adds it to the structured sample log, emits gauge jobs.dlq.untriaged_missing, and raises a separate dlq_untriaged_missing breach — re-alerting every sample while > 0, recovering at 0, same anomaly semantics as dlq_growth.

Non-goals (per issue thread + design): no API/front surface (that is #636 Phase 4; no contract change, no client regen), no classifier/status machinery, no webhook alerting (WARNING log remains §7.2 v1's hook).

Paired proof (RED without the fix, GREEN with it)

Method: the test commit (fd171b647) contains the specs and compile-only inert seams but none of the behavioral changes; the fix commit (640f23405) differs from it ONLY by DeadLetterRetentionHandler.cs + JobQueueMonitorService.cs. Both runs below executed against the exact committed trees, on the Testcontainers PostgreSQL integration suite (just test-api harness), env APP_ROLE=api ASPNETCORE_ENVIRONMENT=Testing.

RED — commit fd171b6 (tests, no fix)

[xUnit.net 00:00:21.16] PublyApp.Api.Infrastructure.Jobs.JobQueueMonitorServiceSpec.ItShouldSampleAndEmitTheUntriagedMissingCountUntilTriaged [FAIL]
[xUnit.net 00:00:21.16]   Expected held.MissingTriagedCount to be 2L because two durable untriaged missing-anomaly rows are counted exactly, but found 0L (difference of -2).
[xUnit.net 00:00:21.16] PublyApp.Api.Infrastructure.Jobs.JobQueueMonitorServiceSpec.ItShouldAlertWhileUntriagedMissingRowsAreHeldAndStaySilentAtZero [FAIL]
[xUnit.net 00:00:21.16]   Expected monitor.EvaluateAndAlert(JobQueueSample.Empty with { MissingTriagedCount = 1 }) {empty} to contain "dlq_untriaged_missing".
[xUnit.net 00:00:21.18] PublyApp.Api.Modules.Jobs.Jobs.DeadLetterRetentionHandlerSpec.ItShouldHoldAnUntriagedMissingRowBeyondTheHorizonAndReportIt [FAIL]
[xUnit.net 00:00:21.18]   Expected (verify.JobDeadLetter.AnyAsync(d => d.JobType == untriagedMissing)) to be True because an untriaged missing-anomaly row is held however old it is, but found False.

Failed!  - Failed:     3, Passed:     1, Skipped:     0, Total:     4, Duration: 298 ms - PublyApp.Api.Tests.dll (net10.0)

The one passing test (ItShouldDeleteATriagedMissingRowBeyondTheHorizonLikeAnyOtherRow) passes vacuously without the fix: the unfixed sweep deletes everything past the horizon indiscriminately. It becomes meaningful paired with the hold test above (which fails RED precisely because an untriaged row was deleted).

GREEN — commit 640f234 (fix applied)

Passed!  - Failed:     0, Passed:     4, Skipped:     0, Total:     4, Duration: 163 ms - PublyApp.Api.Tests.dll (net10.0)

Verification

• ~/ai-orchestration-playbook/tools/heavy.sh just test-api on the final tree: 1922/1922 passed, exit 0 (~4 min).
• OpenApiContractSpec filter (the only dotnet test CI itself runs): 4/4 passed.
• just ci-drift ✅ · just ci-migration-expand-contract ✅ (18/18) · pnpm lint ✅ 0 errors · pnpm format (oxfmt --check) ✅.
• Migration snapshot regenerated via just db-add AddDeadLetterTriage (EF 10.0.7 pin); expand-only shape asserted by AddDeadLetterTriage.Spec.cs against real Postgres (columns nullable, index filter present).
• Front untouched: no typecheck/test/e2e obligations from this diff; local e2e intentionally not run per lane policy (CI front-e2e remains the evidence for anything it still covers).

Unverified / honest limits

• No operator path exercises this in production yet. Until A5 — Staff job-visibility dashboard #636 ships the staff surface (and the external-state producers land), every row has triaged_at IS NULL, no producer writes jobs.missing.* types, so the held count is structurally 0 in production and dlq_untriaged_missing cannot fire outside tests. The rule and its alarm are proven by specs, not by a live anomaly.
• Log-based alert delivery is not exercised end-to-end: the WARNING lines are asserted through the returned breach codes and capturing-logger specs, but whatever routes worker logs to a pager downstream of the process is out of scope here.
• just knip fails, identically on current develop (22 unused files / 9 unused deps, all front/scripts-ts). Pre-existing, unrelated to this diff, not fixed here to keep the lane scoped.
• The retention exemption uses a LIKE 'jobs.missing.%' marker convention documented on the entity rather than a status column; if a future producer ever stamps an anomaly row WITHOUT the reserved prefix, the exemption will not protect it. The prefix discipline lives in JobDeadLetter.MissingJobTypePrefix docs and the docs: jobs & worker infrastructure design #852 design; there is no DB constraint forcing producers to honor it.
• Batch-bound sweep loop re-runs CountUntriagedMissingRowsAsync once per pass (one extra indexed COUNT per daily run) — measured negligible, not benchmarked.