auth_failures_total is a metric the runtime does not emit, and observability.mdx alerts on it

Found while working #70, which deleted the other phantom-metric alert in the same code block (kernel_cache_misses_total). Filed rather than folded in: unlike that one, the correct shape here is not pinned by any evidence, so it needs a call rather than a transcription.

What is there

content/docs/operate/observability.mdx, in "Useful starter alerts":

      - alert: ObjectOSAuthFailureSpike
        expr: rate(auth_failures_total[5m]) > 5
        for: 10m
        annotations:
          summary: "Sustained auth failure rate — check for misconfiguration or attack"

auth_failures_total is emitted by nothing. On objectstack@origin/main:

git grep auth_failures_total over the whole repository returns zero files.
packages/observability/src/semconv.ts is the canonical registry — it exists, in its own words, "so hosts can wire alerts/dashboards against a stable namespace and so call sites don't sprinkle string literals through the code base". It declares exactly 13 names, and no auth metric is among them:

http_requests_total          http_request_duration_ms      http_request_errors_total
storage_operations_total     storage_operation_duration_ms storage_errors_total
cache_lookups_total          cache_writes_total            cache_errors_total
job_schedule_failures_total  registry_lookups_total        registry_lookup_duration_ms
registry_source_fetches_total

The first alert in the same block, ObjectOS5xxSpike, uses http_requests_total and is fine — which is what makes this one costly. A reader copies the whole block, one alert works, and the other silently never fires. A Prometheus rule on a metric that is never scraped does not error; rate() over an absent series yields no samples and the alert simply stays green forever. That is worse than no alert, because the operator believes auth failures are covered.

Why this is a call, not a transcription

Two defensible fixes, and picking between them is a product decision:

Delete the alert. Consistent with what Six pages outside Deploy still describe the retired multi-Environment runtime #70 did to the kernel alert, and honest. Costs the page its only security-facing alert, and "auth failure rate" is also listed as a signal to track in the table above it and in resources/faq.mdx ("What metrics should I monitor?").
Ask the runtime to emit one. Auth failure rate is a genuinely useful signal, SEMCONV is the declared place for it, and the framework already instruments HTTP, storage, cache, jobs and the registry. That is an objectstack card, not an objectos one — this repository would then transcribe the name once it exists.

There is a cheap third option worth considering while deciding: express the alert against a metric that does exist, by counting 401/403 responses out of http_requests_total{status=~"40[13]"}. That covers most of what the original alert was watching without waiting for a new metric. Whether it covers enough is the part that needs a real answer — a failed password on a session endpoint and a rejected API key do not necessarily land on the same status code.

⛔ Do not resolve this by leaving the alert and adding a note that the metric may not exist. An alert that cannot fire, annotated as such, is still an alert a reader copies.

Verified against: packages/observability/src/semconv.ts and a repository-wide git grep auth_failures_total on objectstack@origin/main.

Related: #70.

Generated by Claude Code

Claim: PM loop round 6
Session: session_01CJPxtTxoxTUnjNdTbiEaRa
Branch: claude/issue-87-phantom-auth-metric
Worktree: objectos-issue-87
Domain: domain:devx (objectstack-ai/objectstack#9551, Option B). Dispatched autonomously, 原话: 「你自主派发」.
File surface: content/docs/operate/observability.mdx, content/docs/resources/faq.mdx (English only)
Container & model: M, mode:subagent, model: opus
Clause-②: no
Serial constraints cleared: batch siblings #80 (glossary.mdx), #84 (configure/runtime.mdx), #88 (quickstart.mdx) — disjoint.

faq.mdx is in the surface because I checked: line 164 still lists "auth failure rate" among the metrics to monitor, so fixing only the alert would leave the FAQ recommending a signal that does not exist. #86 rewrote the neighbouring sentence in that answer today — re-read it before editing.

Premise independently confirmed

I re-ran the measurement rather than taking the card's word, with the reverse control a zero-hit needs:

auth_failures_total   0 hits
SEMCONV              15 hits   ← control: the grep and the repository are fine

packages/observability/src/semconv.ts declares 13 names and no auth metric among them. The card is right.

Splitting the decision — this card is dispatchable, one half of it is not

The card frames this as a product call and says picking between "delete" and "ask the runtime to emit one" is a decision. That framing bundles two questions, and only one of them is mine:

"What should these pages say today, given what the runtime actually emits?" — a documentation question, and this card's job.
"Should the runtime emit an auth-failure metric?" — a product/roadmap question about objectstack, and not for this repository or this seat to answer.

So the dispatch resolves the first and routes the second: the dev fixes the docs against what exists today and files an objectstack card for the missing metric rather than deciding it. If the runtime later gains the metric, transcribing the name back here is a one-line follow-up.

⛔ The card's own prohibition stands and is carried into the dispatch: do not resolve this by leaving the alert with a note that the metric may not exist. An alert that cannot fire, annotated as such, is still an alert a reader copies.

Generated by Claude Code

{
  "issue": 87,
  "status": "done",
  "branch": "claude/issue-87-phantom-auth-metric",
  "pr": "https://github.com/objectstack-ai/objectos/pull/98",
  "premise_still_valid": true,
  "summary": "Deleted the ObjectOSAuthFailureSpike alert from observability.mdx plus the two other places that recommended the same phantom signal — the 'Auth failures' row in the signal table above the alert block, and 'auth failure rate' in faq.mdx's 'What metrics should I monitor?'. The dispatch's third option (express the alert against http_requests_total{status=~\"40[13]\"}) was measured rather than adopted, and it fails on three independent counts: the sign-in endpoints do not feed that counter at all (http_requests_total has exactly one emitter, instrumentRouteHandler, applied by a Proxy dispatcher-plugin.ts builds over a LOCAL server binding and never re-registers, while plugin-auth mounts its routes on the raw Hono app via getRawApp()); a rejected API key is indistinguishable from no credential (api-key.ts validates fail-closed, so both land on the same anonymous 401); and 401/403 are dominated by ordinary authorization denials and anonymous traffic. So deleting was the better answer. Each page now carries prose in #86's established shape saying no auth-failure series exists and why 401/403 counting is not a substitute, pointing at the Console's `Auth` audit view for sign-in activity — deliberately scoped to activity, not failures, because plugin-audit's AuthSessionAuditEvent is a closed login|logout union with a required real sys_user id and records successful sessions only. The second question (should the runtime emit one?) was filed, not answered: objectstack-ai/objectstack#9623, unassigned, with the zero-grep evidence, the 13-name SEMCONV registry, the three reasons the HTTP-status substitute fails, and the case for where it would belong.",
  "tests": "All on fe78131, the head of the branch, tree clean. HEAVY (under flock /tmp/os-heavy-verify.lock, -E 99): `pnpm turbo run type-check --force --concurrency=2` -> '@objectos/docs:type-check: cache bypass, force executing 8c984641fb51da8e' ... 'Tasks: 1 successful, 1 total / Cached: 0 cached'; re-run on the final tree after restoring next-env.d.ts, same digest, same result. `pnpm turbo run build --force --concurrency=2` -> 'cache bypass, force executing ade11cd4db19e8fa' ... 'Tasks: 1 successful, 1 total / Cached: 0 cached', 550+ doc paths prerendered. `pnpm turbo run test` -> 'WARNING No tasks were executed as part of this run. / Tasks: 0 successful, 0 total' — the docs app declares no test script, which is what CI does too; read as 'nothing to run', not as a pass. TRANSLATION (all three, as CI invokes them): check-translation-ownership.mjs --actor os-zhuang --files changed.txt -> exit 0, '0 translation artifact(s) and 2 other file(s)'; check-translations.mjs -> exit 0, 'checkmark translations gate passed' (observability + faq siblings now listed stale in all 6 locales — reported-not-blocking by design, per AGENTS.md); check-translation-output.mjs --self-test -> exit 0, '20 case(s) on locale zh-Hans, every rule demonstrated able to fail'; check-translation-output.mjs --files changed.txt -> exit 0, 'translation output gate passed (110 pre-existing finding(s) reported)', none of the 110 in these two pages. RENDERED HTML, real dev server (next dev on :3000), not inferred from source: /docs/operate/observability -> 200, 130966 bytes; parsed the served HTML — the alert `pre` block contains ObjectOS5xxSpike alone, the signal table parses to exactly 4 rows with no auth row, and grep -c over the served HTML gives 0 for both auth_failures_total and ObjectOSAuthFailureSpike. /docs/resources/faq -> 200, 135456 bytes; the answer renders as '5xx rate, p95 latency, readiness (/api/v1/ready), queue depth. There is no auth-failure metric to monitor — review sign-in activity in the Console's Auth audit view instead.', and 'auth failure rate' greps to 0. No ablation applies — this is a docs-only change with no gate or guard to mutate. Control-byte self-scan (grep -naP over both edited files and the filed issue body) clean. PREMISE re-measured independently on objectstack@origin/main b057e53f4 with the reverse control a zero-hit needs: `git grep auth_failures_total` -> 0 files, control `git grep -l SEMCONV` -> 15 files. Trap hit and handled: running the dev server rewrote the TRACKED generated file apps/docs/next-env.d.ts (import path flips from ./.next/types to ./.next/dev/types); restored with git checkout before pushing, so it is not in the PR — left in, it would have put the dev-mode typegen path into a tree CI type-checks in build mode. CI on the PR was not waited for, per the 2026-08-10 ruling; PR is draft and mergeable_state was 'unstable' (checks in flight) at report time.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as objectstack-ai/objectstack#9623: the mandated split-out half of this card — decide whether SEMCONV should declare an auth-failure metric; unassigned, with the zero-grep evidence, the 13-name registry, and why 401/403 out of http_requests_total cannot stand in for one",
    "filed as #96: the same faq.mdx sentence still recommends 'queue depth', which is also emitted by nothing — the only greps for queue_depth/queue_size are placeholder names inside metrics-registry unit tests, while the control job_schedule_failures_total resolves to both a declaration and a live emitter. Same defect class as this card, different call, so filed rather than folded in",
    "filed as #97 (label `finding`): AGENTS.md documents the docs dev server on http://localhost:3001, but apps/docs `dev` is bare `next dev` and binds 3000 — measured, a fetch to 3001 returns nothing while 3000 returns 200",
    "reported not edited: glossary.mdx, quickstart.mdx and configure/runtime.mdx are owned by sibling cards this round — untouched. Their PRs (#92, #93, #95) landed on main during this task and do not touch either of my two files; my branch merges cleanly",
    "noted, not filed (belongs to objectstack#9623's body, where it is recorded): http_requests_total covers only dispatcher-registered routes — packages/rest resolves http.server itself at rest-api-plugin.ts:129, so the REST data API is outside the counter too. Worth its own look by whoever takes #9623"
  ]
}

Generated by Claude Code

ACCEPT — #98.

The split held

The 裁决 said the product half is not this repository's to answer. It was filed as objectstack-ai/objectstack#9623, unassigned, carrying the evidence rather than a conclusion. That is the outcome I wanted: the docs defect is fixed today, and "should the runtime emit an auth-failure metric" is sitting where someone can actually decide it.

The third option was tested and refuted, at mechanism level

I offered http_requests_total{status=~"40[13]"} as a lead and told the dev to test its own caveat. It went further than the caveat and found a structural reason the substitute cannot work:

http_requests_total is emitted through instrumentRouteHandler, applied at dispatcher-plugin.ts:710 to dispatcher-registered routes;
plugin-auth mounts its routes on the raw Hono app via getRawApp(), bypassing that path entirely.

So the sign-in endpoints do not feed the counter at all — the substitute would have been an alert on a series that structurally excludes the events it claims to watch. Plus the two failures I did flag: a rejected API key is indistinguishable from no credential (both fail closed to an anonymous 401), and 401/403 are dominated by ordinary authorization denials against users who authenticated fine.

I verified the call sites and the declaration myself. Worth stating what level that verification is at: it is code reading, not a runtime measurement — the same kind of evidence that led me to rule wrongly about /health on #88 earlier today. The difference is which way an error would cut: here the conclusion is "do not add this alert", so a wrong reading costs a missing alert rather than a wrong instruction to a reader. I am accepting it on that asymmetry, not on the belief that code reading is sufficient in general.

The replacement is scoped honestly

Deleting a phantom alert and leaving the reader nowhere to go just invites them to invent another one. This points at the Console's Auth audit view — and deliberately says sign-in activity, not failures, because AuthSessionAuditEvent is a closed login|logout union with a required real user id, so it records successful sessions only. Not overselling the alternative is the part that makes it useful.

Where I would have pushed harder

The FAQ sentence now reads "5xx rate, p95 latency, readiness (/api/v1/ready), queue depth" — and queue depth is also a phantom, filed as #96 (the only greps resolve to placeholder names inside metrics-registry unit tests). So the page this card was about still recommends a metric that does not exist, in the same sentence the card fixed.

Filing it is defensible: it needs its own call, exactly as the auth one did, and the dev applied the card's own reasoning consistently. But the defect class here is "this page recommends a signal that isn't there", and the card closes with one instance of it still on the page. I would have taken it folded in. #96 should be worked soon rather than aging — it is not an independent finding so much as the unfinished half of this one.

A near-miss worth recording

Running next dev rewrote the tracked generated file apps/docs/next-env.d.ts, flipping its import path to the dev-mode typegen directory. The dev caught it and restored it before pushing. Left in, it would have committed a dev-mode path into a tree CI type-checks in build mode — green locally, and a trap for whoever next ran the build. Catching a self-inflicted contamination from your own verification step is easy to miss precisely because it comes from the tool you were using to be careful.

Other checks

Item	Reading
Scope	2 files, exactly the declared surface; sibling cards' pages untouched
CI	`build` and `Ownership & freshness` both green
`--force`	`cache bypass, force executing` on both tasks
Premise	Re-measured independently with the reverse control: `auth_failures_total` → 0 files, control `SEMCONV` → 15
Rendered	Read from a real dev server, not inferred: 0 occurrences of `auth_failures_total` or `ObjectOSAuthFailureSpike` in the served HTML, signal table down to 4 rows

Also filed #97 — AGENTS.md says the docs dev server is on port 3001; apps/docs's dev script is bare next dev and binds 3000. Measured, not assumed.

Landing

Direct merge, same recorded deviation: build required, ci.yml still without merge_group: (#76), #77 open with its base ten commits behind main.

Generated by Claude Code

Navigation Menu

Repository navigation

`auth_failures_total` is a metric the runtime does not emit, and observability.mdx alerts on it #87

What is there

Why this is a call, not a transcription

Premise independently confirmed

Splitting the decision — this card is dispatchable, one half of it is not

The split held

The third option was tested and refuted, at mechanism level

The replacement is scoped honestly

Where I would have pushed harder

A near-miss worth recording

Other checks

Landing

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

auth_failures_total is a metric the runtime does not emit, and observability.mdx alerts on it #87

Description

What is there

Why this is a call, not a transcription

Activity

os-zhuang commented on Aug 18, 2026

Premise independently confirmed

Splitting the decision — this card is dispatchable, one half of it is not

os-zhuang commented on Aug 18, 2026

os-zhuang commented on Aug 18, 2026

The split held

The third option was tested and refuted, at mechanism level

The replacement is scoped honestly

Where I would have pushed harder

A near-miss worth recording

Other checks

Landing

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`auth_failures_total` is a metric the runtime does not emit, and observability.mdx alerts on it #87