jobs: give Unclassified DLQ rows a resolution path and retention exemptions (K-1)

[radandevist]

Closes #863

Implementer: Ox Alpha (stealth/ox-alpha via Nous Portal, max effort, jcode). Reviewer: pending adversarial review.

What this does

An ExternalStateStatus = 6 Unclassified dead-letter row used to have no way out: age retention deleted it blindly at 90 days even though nobody ever established whether its external effects exist. Two coordinated changes give every DLQ class either a resolution path or a visible reason to stay:

1. Schema (db86cd814, prior commit): external_state_status enum + columns on job_dead_letter with CHECK constraints, partial index, immutable job_dead_letter_events evidence table, expand-only migration.
2. Operator triage endpoint (3142223fc): POST /staff/dead-letter/{id}/resolve-unclassified (new jobs.resolve permission). An operator confirms the externally-referenced resource is absent; fail-closed contract below.
3. Retention exemptions (efbc3051e): sweep skips statuses 1 Present / 6 Unclassified past the horizon; every run logs a structured SkippedCount so exempt rows cannot silently starve the sweep.
4. Client regen (1cd4baf70): Kiota client picks up the new endpoint (staff/deadLetter/item/resolveUnclassified).

Fail-closed contract (endpoint)

• Unknown or malformed id → 404 (string route param parsed in-handler; repo rule: no route constraints)
• Row's current external state ≠ 6 Unclassified → 409 naming the actual state
• Success stamps 4 Missing keeping the recorded prepared/expires bounds, appends exactly one immutable event (reason: operator_confirmed_absent, optional note ≤500 chars), and writes an audit entry (job.dead_letter.triage.resolved)
• Double-resolve race: loser gets 409 Missing; still exactly one event row
• 403 for staff without jobs.resolve, verified to mutate nothing

Retention semantics

Status 0 None sweeps exactly as before (strict < horizon unchanged, batching/SKIP LOCKED untouched). Present/Unclassified are held back; the structured log reports DeletedCount and SkippedCount (exempt rows beyond horizon) so starvation stays observable until #864/#865 land.

Paired proofs (TDD)

Retention exemption — RED without fix (reverted handler, tests kept)

Command: dotnet test --filter DeadLetterRetentionHandlerSpec.ItShouldExemptPresentAndUnclassifiedRowsFromAgeRetention|ItShouldReportSkippedExemptRowCountWhenSweepEncountersExemptRows
Result: Failed: 2, Passed: 0

Failed ...ItShouldExemptPresentAndUnclassifiedRowsFromAgeRetention
  Expected (verify.JobDeadLetter.AnyAsync(d => d.JobType == $"{marker}.none")) to be True because status 0 None stays plain age-retention eligible, but found False.
  [unmodified sweep had also deleted the Present and Unclassified rows]
Failed ...ItShouldReportSkippedExemptRowCountWhenSweepEncountersExemptRows
  Expected skipRecord not to be <null> because the sweep must report skipped exempt counts so starvation stays visible.

GREEN after the handler change: all Jobs specs Passed: 164, Failed: 0 (the same two facts now pass; None-row assertion correctly expects it swept).

Endpoint slice — RED without route registration (MapPost neutralized, fresh binary verified by DLL timestamp)

Command: dotnet test --filter ResolveDeadLetterUnclassifiedSpec|DeadLetterResolutionCatalogSpec
Result: Failed: 4, Passed: 4

Failed ...ItShouldReturnForbiddenForStaffWithoutPermission   (404 instead of 403)
Failed ...ItShouldReturnConflictWhenRowIsNotAwaitingTriage   (404 instead of 409)
Failed ...ItShouldLoseTheRaceCleanlyOnDoubleResolution       (404 instead of 200/409)
Failed ...ItShouldResolveAnUnclassifiedRowStampingMissingWithEventAndAudit (404 instead of 200)

Exactly the four facts that need the route fail; the catalog architecture pin and the two no-route-constraint 404 facts are route-independent and stay green. Restored registration → Passed: 8, Failed: 0.

Method note (honesty item): a first RED attempt passed 8/8 because MSBuild incremental compilation reused a stale DLL; caught it via DLL mtime vs edit time and re-ran with --no-incremental. The recorded RED above is from the verified-fresh binary.

Gates

• Targeted: all 164 FullyQualifiedName~Jobs specs green (includes retention pair, endpoint facts, catalog pin); after the fixes below, Jobs | ServiceAttributeRegistration | HandlerScopeNamingGuard = 171/171 green
• Full API integration suite (Testcontainers) run once at the end under heavy.sh: Passed 1907 / Failed 3 / Total 1910. All three failures are the boot-probe specs (SeederGateProbeSpec, AppRoleCompositionSpec, MasterKeyWitnessBootIntegrationSpec) whose spawned child processes exit during AppEnvironment.Initialize() with exactly Environment validation failed: TRUSTED_PROXY_CIDRS must be set explicitly for a production API role — this worktree has no .env.development and the main checkout's file predates that key; they fail before any Jobs-domain code executes and are green wherever a complete env file exists.
• just generate-client regenerated apps/api/openapi.json + packages/client-ts cleanly
• pnpm --filter front typecheck green against the new client
• pnpm run lint:fix && pnpm run format:write clean (one unrelated pre-existing formatter nit in check-archive-records.ts reverted rather than bundled here)
• pnpm test:ci-drift green (14/14)
• Migration snapshot in sync (generated with the migration; expand-only, additive CHECKs/index)

Unverified / notes for the reviewer

• Full-suite first pass surfaced two real gaps this slice had to fix, caught by repo architecture pins: HandlerScopeNamingGuardSpec required the ForStaff scope marker on a Handlers/Staff/ handler class (handler + wire types renamed ResolveDeadLetterUnclassified* → ResolveDeadLetterUnclassifiedForStaff*; service-domain records keep the domain name), and ServiceAttributeRegistrationSpec's expected-services manifest needed the new (IJobDeadLetterService, JobDeadLetterService) entry.
• Three full-suite failures are environmental, not from this branch's code: SeederGateProbeSpec, AppRoleCompositionSpec, MasterKeyWitnessBootIntegrationSpec spawn child processes whose boot fails at AppEnvironment.Initialize() ("Environment validation failed") because this worktree has no .env.development and the main checkout's file predates TRUSTED_PROXY_CIDRS. They fail before any Jobs-domain code executes; they are green in CI and in other lanes with complete env files.
• Local e2e stack intentionally not run per the captain's verification policy (2026-08-23); CI runs front-e2e on this PR. No apps/front source files were touched (shared-ts i18n JSON + generated client only), so the full front vitest suite was not run locally either.
• just ci-migration-expand-contract was not executed locally (DB-spinning); migration is expand-only by construction but CI owns that signal.
• The endpoint was exercised through Testcontainers HTTP round-trips only, not against a live server.
• The SkippedCount == 2 assertion pins the exact exempt population in the spec scenario; if a future change adds exempt states, DeadLetterResolutionCatalogSpec forces updating both the enum↔catalog coverage pin and the retention-exemption set together.
• Worktree env quirks documented for future lanes: (a) generate-client needs TRUSTED_PROXY_CIDRS exported explicitly when .env.development predates that key; (b) MSBuild incremental compilation can silently reuse a stale DLL after out-of-band file edits — use --no-incremental when proving RED states by reverting production hunks.

---

Rebase (2026-08-25)

Rebased lane/wt-863 onto origin/develop (5a2b830) to clear the CONFLICTING state that blocked CI. Conflicts resolved keeping BOTH intents: develop's #1336/#864 untriaged-Missing hold AND this PR's #863 external-state resolution path.

Conflicts and decisions:

• JobDeadLetterConfiguration.cs: kept BOTH partial indexes — develop's ix_job_dead_letter_untriaged_missing (jobs: job-dead-letter-retention deletes 4 Missing rows at 90 days, silently clearing their alert without triage #864) and ours ix_job_dead_letter_external_state (jobs: an absent 6 Unclassified DLQ row has no resolution path and is exempt from age retention forever #863).
• Retention sweep (DeadLetterRetentionHandler): delete predicate now ANDs both exemptions (untriaged missing-anomaly prefix OR external_state Present/Unclassified holds a row); the pass logs both held-back counts (Held untriaged missing rows + SkippedCount exempt rows). Spec file carries all four facts: develop's two jobs: job-dead-letter-retention deletes 4 Missing rows at 90 days, silently clearing their alert without triage #864 facts (hold-untriaged / triaged-sweeps) plus our two jobs: an absent 6 Unclassified DLQ row has no resolution path and is exempt from age retention forever #863 facts (Present/Unclassified exempt + skip-count report).
• i18n JSON (response-message.en/fr.json): both key sets kept — develop's upload-budget-exhausted (chore(deps): close #880 follow-up — runtime guard + runbook record for the 2 moderate advisories #1331) alongside our three dead-letter-* keys.
• Migration: never hand-merged. Deleted the old migration files during conflict resolution and REGENERATED (dotnet ef migrations add AddJobDeadLetterExternalState) on top of develop's snapshot → 20260825023350_AddJobDeadLetterExternalState, autosquashed back into the schema commit. Verified: Designer BuildTargetModel is byte-identical to AppDbContextModelSnapshot.BuildModel apart from the expected class/attribute boilerplate, and dotnet ef migrations has-pending-model-changes reports none. (Trap recorded: run it AFTER a fresh build — with --no-build over a stale DLL it false-reports pending changes.)
• openapi.json + kiota client: regenerated with just generate-client, not by hand. Picks up the ResolveDeadLetterUnclassifiedForStaffBody rename from the final commit of this PR; squashed into the client commit. Nothing in apps/front/src or packages/shared-ts/src references the renamed type.

Gates after rebase: targeted DLQ specs 15/15; full API integration suite run ONCE under heavy.sh: 1932/1932 passed (this time including the three boot-probe specs that were environmental failures before, since .env.development now exists in the worktree); just ci-drift 7/7; pnpm --filter front typecheck green; pnpm format (read-only oxfmt check, same as CI) clean — an oxfmt normalization nit it auto-fixed on untouched packages/scripts-ts/src/check-archive-records.ts was reverted rather than bundled here.

Model: Ox Alpha via Nous Portal, max, jcode.