feat(health): observe expired scoring-job leases

[cursor]

Why

PR #113 classifies scoring-job queue depth and created-at age, but a dead worker whose lease is already past active_lease_expires_at_unix_ms can still look WithinBounds when those bounds are enqueue-to-complete. Operators then see a healthy backlog while results stop arriving.

This branch is #113 at 49dbb2b plus expired-lease observation. Prefer this head over merging #113 alone if dead-worker visibility is in the same integration window.

TDD

RED: expired_scoring_lease_cannot_be_masked_by_healthy_count_or_created_at_age failed to compile (E0560) because expired_lease_count and max_expired_lease_count did not exist.

GREEN: the probe counts leased rows whose persisted expiry is already past the database clock, the classifier fails closed against caller policy, apply_scoring_job_migration applies 0022 after 0002/0021, and a second apply is idempotent.

What changed

• observe expired leased-job count separately from active queue depth;
• compare lease expiry to clock_timestamp() so a later created-at age bound cannot hide a dead worker;
• fail closed on non-positive stored lease expiry;
• classify only against caller-supplied max_expired_lease_count, with no universal SLO defaults;
• expose aggregate counts only — never worker, lease, request, or payload identities;
• apply the leased-expiry partial index through the existing product apply_scoring_job_migration path;
• reject reused migration numeric prefixes so draft feat(identity): persist append-only account-link history #114 cannot land a second 0021_* file.

Verification

• composition: expired lease cannot be masked by a healthy outbox or created-at age;
• empty, mixed-state, live-versus-expired lease, count/age/quarantine, future-evidence, invalid created-at, transaction success, and missing-relation cases;
• index contract for active, quarantined, and leased-expiry partial indexes;
• cargo fmt, cargo clippy --all-targets -- -D warnings, PostgreSQL 18.6 scoring-job backlog tests.

Coordination

• Prefer this head over feat(health): observe PostgreSQL operational backlogs #82 and feat(health): observe PostgreSQL scoring-job backlogs #113 if scoring-job readiness is in the same window.
• Draft feat(identity): persist append-only account-link history #114 must renumber 0021_participant_identity_link.sql to 0023 (or later) before it leaves draft.
• HTTP probes remain feat(health): serve operator probes until accept fails #111. No universal SLO defaults. Live fast-mlsirm execution remains outside this repository.

  

[cursor]

Stale comment

Review — 8c51e3d expired scoring-job leases

This is the current scoring-job readiness successor of #113. Prefer #131 over merging #82 or #113 alone. This comment is not an approval.

What is sound

• Expired leased rows are counted against clock_timestamp(), so a healthy created-at age cannot hide a dead worker.
• Classification stays caller-policy-only. Evidence remains counts, not worker/lease/request identities.
• 0022 applies through apply_scoring_job_migration. Composition still fails closed across integration, data-rights, and scoring-job families.

Required before this is treated as shipped truth

1. Independent last-push review (seonghobae already requested).
2. Exact-head required checks on 8c51e3d.
3. Name 0020/0021/0022 in the TRACEABILITY migrations/ tree and add the Active-PR note to docs/architecture/AS_BUILT_SCHEMA.md.

Coordination

• Do not merge #113 or #82 as siblings while this head is the scoring-job readiness vehicle.
• Draft #114 must keep its identity-link migration number after 0022.
• HTTP probes stay on #122 / #111.

  

_{Sent by Cursor Automation: Fix Issues}

[cursor]

Review: expired-lease observation (#131 @ 8c51e3d)

Prefer this head over merging #113 / #82 alone when dead-worker visibility is in the same window. Do not merge until exact-head required checks and an independent last-push approval land. This review does not approve.

What holds

• apply_scoring_job_migration composes immutable 0002 + 0021 + 0022 (CREATE INDEX IF NOT EXISTS). Second apply is idempotent.
• Probe counts leased AND active_lease_expires_at_unix_ms < clock_timestamp() separately from active queue depth. Classification uses only caller-supplied max_expired_lease_count. No universal SLO.
• Schema already requires leased rows to have a positive expiry (scoring_active_lease_shape_check, scoring_lease_expiry_positive_check). Aggregate-only output: no worker, lease, request, or payload identities.
• Composition test expired_scoring_lease_cannot_be_masked_by_healthy_count_or_created_at_age plus the mixed-state / refuse-expired PostgreSQL cases match the claimed fail-closed policy.
• Migration prefix 0022 is unique on this head. Draft #114 must keep 0021_participant_identity_link.sql off this number and use 0023 or later before it leaves draft.

Residual that a buyer/operator will feel

The probe queries MIN(active_lease_expires_at_unix_ms) for all leased rows and discards it. QA-OBS-03 promises counts and timestamps. A purchaser operating a dead worker can see expired_lease_count = 1 and still not know how stale that lease is. Restrict the MIN to already-expired leased rows against one clock_timestamp() snapshot and expose oldest_expired_lease_at_unix_ms. Keep classification on the count bound.

HTTP probe wiring is still a separate slice. Current landing vehicle is #132, not #111.

Coordination

• Prefer this over #113 if only one scoring-job health head can land.
• Do not fold HTTP listen/bind, session persist, or identity-link 0021 into this slice.
• Live fast-mlsirm execution remains outside this repository.

  

_{Sent by Cursor Automation: Fix Issues}

[cursor]

This MIN(active_lease_expires_at_unix_ms) is taken over every leased row, including live future expiries, and is then discarded after the positivity check. Schema scoring_lease_expiry_positive_check already forbids non-positive leased expiry, so the extra column does not change classification.

QA-OBS-03 says the probe exposes aggregate counts and timestamps. Operators can see expired_lease_count but cannot tell whether the oldest dead lease expired 5 seconds or 5 hours ago. Restrict the MIN to leased AND expiry < clock_timestamp() (one shared clock snapshot with the count) and return it as oldest_expired_lease_at_unix_ms. Do not invent an age SLO; keep classification on the caller-supplied count bound.

[cursor]

HTTP probe transport is no longer #111. The current landing vehicle is #132 (feat(health): bind probe process from listen env). Point evidence and risk text at that head, or say "HTTP probe transport remains a separate active PR," so this slice does not freeze a superseded number.