[Kohandezh]

Lane D, first PR of six (order HOM-113 → 114 → 115 → 126 → 28 → 29). No migration, no RLS change.

What it does

• Correlation ids: one id per unit of work, in a context variable: the validated X-Request-ID (or a UUID) per request, job:<id>:<delivery> per processing attempt, sweep:<task>:<run> per periodic sweep. JSON logs carry correlation_id; record_audit_event stores it when the caller passes no request id (41 of 83 call sites passed none).
• GET /metrics on the API port (the proxy never routes it; a test parses infra/nginx/conf.d/homayar.conf to prove it):
  • in-process: homayar_http_requests_total{method,route,status_class} (route = path template, unmatched otherwise), latency histogram, collection errors (the auth-failure and audit-write-failure counters move to the HOM-114 PR with their emitters);
  • scrape-time state, cached METRICS_CACHE_SECONDS (15 s), every source bounded by a deadline (see review fixes): homayar_dependency_up{database|redis|storage}, homayar_schema_up_to_date, homayar_build_info, durable queue depth and latency (homayar_jobs_queued_due, homayar_jobs_oldest_queued_due_age_seconds), jobs by state, outcomes / failure categories / failure classes / retries of the last 15 min, turnaround / processing / queue-wait quantiles, the dispatch outbox, provider circuits by stage, homayar_broker_queue_length, homayar_workspaces_with_failed_jobs_15m;
  • sweep heartbeats: after each periodic task the worker writes a Redis hash (last run/success/failure, duration, exception class, the sweep's integer counters, 7-day TTL). Exposed as homayar_periodic_task_*{task}; this is the "worker/scheduler stopped" signal and carries purge/reconciliation failure counters.
• Log scrubbing on every message and exception text: credentials (headers, bearer/basic, JWT, Homayar/provider keys, password=-style pairs and dict reprs, every secret setting name, URL passwords) and text PII (phone, e-mail, national id, card, IBAN, postal code), bounded to 2000 characters; database errors logged by class and SQLSTATE only.
• Settings: METRICS_BEARER_TOKEN (SecretSetting, _FILE supported, ≥ 32 chars), METRICS_ALLOW_UNAUTHENTICATED, METRICS_CACHE_SECONDS. ADR 0036 (first section), docs/operations/observability.md, .env.example.

Production rule for metrics without a token (Integrator request)

Refuse, not fail startup: in staging/production without METRICS_BEARER_TOKEN, /metrics answers 404 and startup logs METRICS_ENDPOINT_DISABLED reason=no_bearer_token, unless METRICS_ALLOW_UNAUTHENTICATED=true opts out explicitly (private scrape network). Chosen so enabling the feature cannot take down an existing production configuration; Prometheus sees the target as down (HomayarApiScrapeFailing in the alerting PR).

Owned files touched

New: Homayar/backend/observability/{__init__,correlation,scrub,heartbeats,metrics,release}.py, Homayar/backend/tests/test_observability_metrics.py, docs/operations/observability.md, docs/architecture/decisions/0036-operational-evidence-observability-audit-release.md.
Edited: Homayar/backend/main.py, Homayar/backend/config.py (additive, go-ahead 1), Homayar/backend/jobs/tasks.py, Homayar/backend/services/audit_service.py (correlation default only), Homayar/backend/services/logging_service.py, Homayar/backend/tests/test_config_secrets.py + test_settings_dump_redaction.py (secret-field contract: 15 → 16), Homayar/.env.example, docs/README.md.
Not touched: Lane C files (backup/restore/tombstone scripts and modules, compose, infra/scripts), the report, migrations, rls.py, saas_models.py, frozen HOM-120 files.

Tests (local PostgreSQL 16 on 54334 + Redis 7.4 on 63794)

• test_observability_metrics.py: 40 passed (scrape content after a queued and a completed job, failure categories and affected-workspace count, provider circuits, route templates, no tenant/job/user/provider identity in the exposition, bearer-token rules, 404 when disabled, staging refusal + log + opt-out, policy table, nginx never routes /metrics, heartbeats success/failure/bounded counters, unreachable store never fails a sweep, real Redis round trip with TTL, request/job correlation in audit rows, scrubbed JSON logs).
• Focused batch: test_observability_metrics test_config_secrets test_settings_dump_redaction test_job_leases test_dispatch_outbox test_admin_api test_admin_jobs_api test_auth_contract test_api test_queue_routing test_transport_policy test_job_processor test_members_api → 602 passed, 0 failed (19 min; host load was 100–300 from other sessions).
• ruff and mypy clean on the new modules; gitleaks (v8.28.0, repo config) on the changed files: no leaks. Secret-shaped test strings are built at runtime.
• CI is billing-blocked: per the owner's decision this PR merges on local evidence and is not CI-validated.

Review fixes (26f9755, Integrator review of 25 findings)

• D1 bounded scrape: each source runs on a worker pool with a deadline (3 s, two stages); a late source is reported (homayar_dependency_up=0, homayar_metrics_collection_errors_total{source}) and not restarted while still running. Own DB pool with connect_timeout=2; object-store probe with 2 s connect/read timeouts and no retries. /metrics is async and renders through a CapacityLimiter(1), so it never holds API threads. Tests: hung dependency; down database + object store answered in < 9 s.
• D2 --workers 2: with PROMETHEUS_MULTIPROC_DIR counters/histogram are summed across workers by MultiProcessCollector; process_* only in single-process mode. compose.cloud.yaml sets it for the api service (now its own tmpfs /run/homayar-metrics, round 2). Test: several worker processes summed.
• D3: counter test compares deltas (b3344b3).
• D4 logs: SQLAlchemy errors → class, SQLSTATE, constraint/table/column only (no parameters, no DETAIL). UNHANDLED_ERROR logs the route template inside the request's correlation scope. Tests with a registered user's e-mail, a phone number and a share token in the path.
• D5: METRICS_ALLOW_UNAUTHENTICATED refused with DEPLOYMENT_MODE=cloud (staging/production); on-prem use logged at startup by name, never value.
• D6: only /metrics bypasses TrustedHostMiddleware (scrape via api:8000); every other path still checked (tested). METRICS_ENABLED, METRICS_BEARER_TOKEN, METRICS_CACHE_SECONDS in the compose.cloud.yaml anchor (docker compose config OK). compose.onprem.yaml waits for PR Lane C: recovery objectives, encrypted consistent backup, isolated restore and gate (HOM-153, HOM-116, HOM-117) #42.
• D7: auth/audit failure counters removed here and from the docs; HOM-114 adds them with emitters.
• Should-fix: P0 wave: voice admission, local AI runtime, Persian speech, locations, voicemail #10/Phase 1 infrastructure fixes and HOM-172 normalization speed-up #16 scrub coverage, Website: one cinematic 3D scenario per page, with no framed sections #17 bounded input, Console: remove legacy call-center pages and dead code (HOM-173) #19 correlation id on 500, Console: HeroUI v3 foundation with brand theme, RTL locale and shared primitives #20 per-attempt ids documented, Backend: demo workspace seed for the product console (HOM-175) #22 token stripped, Production: mandatory TLS on internal hops, with audited same-host opt-outs on-premise (HOM-161) #24 real registered e-mails, PII: postal keyword with harakat/tatweel, text-pii/4 (HOM-171) #25 sweeper dead-letter runbook, README: platform feature and deployment map #12 build_info unknown until HOM-28 (documented).
• Follow-up (documented as a known gap): uvicorn/nginx access logs print raw paths.

Local evidence for 26f9755: PostgreSQL 16 (54334) + Redis 7.4 focused batch test_observability_metrics test_config_secrets test_settings_dump_redaction test_job_leases test_dispatch_outbox test_admin_api test_auth_contract test_api test_queue_routing test_transport_policy test_job_processor → 610 passed, 0 failed (14 min). SQLite test_observability_metrics → 68 passed, 1 skipped. ruff check clean on changed files; gitleaks on changed files: no leaks. Owned files added in this round: infra/compose/compose.cloud.yaml (anchor + api PROMETHEUS_MULTIPROC_DIR, Integrator-approved exception).

Review fixes round 2 (e0b577e, Integrator delta review of df191e6)

• E1 server traceback: uvicorn logs an escaped exception again on uvicorn.error with the full traceback through its own handler. ServerLogFilter on uvicorn's loggers scrubs those records, replaces the traceback with the exception summary (class, SQLSTATE, constraint) and adds request_id=. Test: a real uvicorn server, a route raising an IntegrityError whose parameters and DETAIL carry an e-mail and a phone number, and a share token in the path. Every handler is captured (uvicorn's formatter, the root JSON handler, stdout and stderr) and none of them contains the e-mail, phone, token, SQL, DETAIL or a traceback. With the filter disabled, the test fails.
• E2 metric writes: every write goes through WriteGuard; a failure is logged as METRICS_WRITE_FAILED at most once a minute and never fails the request. PROMETHEUS_MULTIPROC_DIR gets its own 32 MB tmpfs, /run/homayar-metrics, apart from the /tmp upload spool; the sizing is documented. Test included.
• E3: counter tests read through in_process_exposition(). The whole observability file passes in both modes (74 passed each).
• Should-fix:
  • S3 probe: total_max_attempts=1;
  • cut after scrubbing (URL credentials found in linear time, input bounded at 16000);
  • ADR and release.py docstring no longer claim a build step before HOM-28;
  • dispatch age documented without a label;
  • the audit tasks wait for HOM-115;
  • the on-prem deferral is documented.

Local evidence for e0b577e: the same 11-file PostgreSQL batch, 615 passed, 0 failed. ruff clean; gitleaks clean on the changed files. Not CI-validated.

Security review notes

• /metrics is never public: not routed by nginx; token compared with secrets.compare_digest; refuses in staging/production without a token unless explicitly opted out; the token is a SecretSetting (redacted in dumps, _FILE support).
• No high-cardinality or identifying labels: fixed vocabularies only; failure_code is mapped to a bounded category; provider endpoint/model never exposed (tested). Tenant impact is a count.
• Cross-tenant reads only through the existing read-only service scopes (recovery on processing_jobs, dispatch on job_dispatches) and only aggregate columns; provider_health is platform state. No policy changed.
• Heartbeats store names, timestamps, exception class names and integer counters only (test: an exception message with a password is not stored).
• Request correlation id: client value accepted only if it matches the existing ^[A-Za-z0-9._:-]{8,80}$, otherwise replaced.
• Logs: scrubbing is defence in depth; no new logging of headers or bodies; database errors never carry bound parameters.
• A hung dependency cannot stall the API: scrape work is on its own pool, one render at a time.

[Kohandezh]

Integrator merge evidence (HOM-113), head 0b26c42:

• Independent review rounds D1–D7 and E1–E3 are resolved. The delta since 43bc48a is the env.py logger fix (disable_existing_loggers=False) and an nginx catch-all test adjusted for Remove the legacy public site [3/3] #75. Both were reviewed.
• CI run 39173320113: 7 of 7 checks pass, including the backend tests and migrations job.
• The env.py change also fixes main's red test_tombstone_reapply (an order-dependent logger disable).
  Merging.