normalizeLanguage only tries the full tag and the bare base language, so de-AT and zh-Hant-TW miss the locale they obviously mean

Found while shipping #172. Filed unassigned. Pre-existing on main; #172 did not introduce it and did not change this function.

Notation: this repo's issue-body sanitizer strips angle brackets and decodes HTML numeric character references, including inside code fences. Nothing below needs either.

The reading

apps/docs/middleware.ts:

function normalizeLanguage(lang: string): string {
  if (LANGUAGE_MAPPING[lang]) return LANGUAGE_MAPPING[lang];
  const baseLang = lang.split('-')[0];
  if (LANGUAGE_MAPPING[baseLang]) return LANGUAGE_MAPPING[baseLang];
  return lang;
}

Two lookups: the exact tag, then the substring before the first hyphen. The result is then required to be in i18n.languages.

LANGUAGE_MAPPING holds only Chinese and Korean entries. So for every other language the second lookup misses too, and the function returns the tag unchanged — which is not in i18n.languages when it carries a region.

Consequences, for a browser that sends a region tag without also listing the bare language:

`Accept-Language`	negotiated	should be
`de-AT`	`en`	`de`
`fr-CA`	`en`	`fr`
`es-MX`	`en`	`es`
`ja-JP`	`en`	`ja`
`zh-Hant-TW`	`zh-Hans`	`zh-Hant`
`zh-Hans-CN`	`zh-Hans` (by luck, via the bare `zh` entry)	`zh-Hans`

Most real browsers send de-AT,de;q=0.9, and the bare de on the second pass rescues it — which is why this has not been noticed. The ones that send a single region tag fall through to English.

The zh-Hant-TW row is the one that got worse in kind rather than degree: before #172 every Chinese browser landed on Simplified, so a script-plus-region tag landing on Simplified was indistinguishable from the general policy. Now that Traditional ships, zh-Hant-TW is a wrong answer where zh-TW, zh-HK, zh-MO and zh-Hant are all right ones.

Why it is not just "add more table rows"

Adding de-AT, fr-CA, zh-Hant-TW … enumerates a set that has no end. The shape of the bug is that the fallback chain drops all subtags in one step instead of one at a time. BCP 47 truncation is: try the whole tag, then drop the last subtag, repeat, until something matches — zh-Hant-TW -> zh-Hant -> zh. That is a few lines and closes the class.

Whichever route is taken, the mapping table and the truncation need to agree about which wins, and the result must still be checked against i18n.languages rather than returned raw.

Verification

Against a running server, Accept-Language set to exactly one tag with no q-list, asserting the Location header of /docs/quickstart. Measured today on the #172 branch: zh-TW, zh-HK, zh-MO, zh-Hant all reach /zh-Hant/docs/quickstart, and zh-CN, zh-Hans reach /zh-Hans/...; the region tags in the table above were not covered by that run and are the cases this issue is about.

Graded: pm:queue / Bug. Unblocked — #218 merged (os-zhuang, 08:57Z). Dispatching.

Claim: PM loop round 15
Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-217-bcp47-truncation
Worktree: objectos-issue-217
Domain: n/a (objectos carries no domain:* lanes)
File surface: apps/docs/middleware.ts. Stop on breach; explain in the report.
Container and model: M, mode:subagent, model: opus
Clause-②: no
Serial constraints: #205 is held behind you — it also touches middleware.ts. #216 runs in parallel on .github/scripts/, no overlap. Base on current main: middleware.ts was rewritten by #204 and again by #218 minutes ago. Read the tree.

This is the highest-value card left, and it is not a parity nit

Four of the six locales we pay to translate are unreachable for a browser that sends a single region tag with no q-list: de-AT, fr-CA, es-MX, ja-JP all negotiate to English. The card is right that most browsers send de-AT,de;q=0.9 and the bare de rescues them on the second pass — which is exactly why this has survived unnoticed, and exactly why it is worth measuring rather than reasoning about.

zh-Hant-TW is the row #218 made newly wrong in kind: it lands on Simplified. Before Traditional shipped that was indistinguishable from policy; now it is a wrong answer sitting beside four right ones (zh-TW, zh-HK, zh-MO, zh-Hant all resolve correctly).

Direction: BCP 47 truncation, as the card says. Not more table rows.

Drop one subtag at a time until something matches — zh-Hant-TW to zh-Hant to zh — rather than the current jump from the full tag straight to the substring before the first hyphen. Enumerating de-AT, fr-CA, zh-Hant-TW and the rest is a set with no end.

Two things the card flags that the implementation has to settle explicitly rather than by accident:

Precedence between LANGUAGE_MAPPING and truncation. They can disagree. Say which wins and why, in the code.
The result must still be checked against i18n.languages, not returned raw. That check is what stops a truncation from inventing a locale.

Verification

Against a running build, Accept-Language set to exactly one tag with no q-list, asserting the Location header — that is the shape the defect lives in, and a q-list will hide it:

Now correct: de-AT, fr-CA, es-MX, ja-JP, zh-Hant-TW.
Still correct, and this is the regression half: every row from feat(docs): ship zh-Hant, generated from zh-Hans and committed #218's table — zh-TW, zh-HK, zh-MO, zh-Hant to /zh-Hant/...; zh-CN, zh-Hans to /zh-Hans/...; ja-JP,ja;q=0.9 to /ja/...; zh-TW,zh;q=0.9 to /zh-Hant/....
A tag that matches nothing (xx-YY, tlh) still lands on English rather than a locale that does not exist.
The /cn/ legacy redirects and the www.objectos.app host redirect unchanged — fix(docs): collapse the metadataBase hostname into SITE_URL, record the middleware decision #204's six-case table, reuse it.
The dotted machine-facing routes untouched: /llms.txt, /llms-full.txt, /sitemap.xml, /robots.txt, one .mdx route, all 200 with no locale prefix.
node .github/scripts/check-locale-surface.mjs exit 0 — it should now report 409, not 346.

Report the full negotiation matrix, not a summary. A table of tag-to-Location is the deliverable that makes this checkable next time someone touches the function.

The turbo cache is shared across worktrees; use --force where a comparison depends on it, capture exit codes before any pipe, and kill any server you start by explicit PID found via /proc — never pkill -f.

Generated by Claude Code

fix(docs): resolve a browser tag by BCP 47 truncation, one subtag at …

os-dev-report

{
  "issue": 217,
  "status": "done",
  "branch": "claude/issue-217-bcp47-truncation",
  "pr": "https://github.com/objectstack-ai/objectos/pull/223",
  "premise_still_valid": true,
  "summary": "Replaced normalizeLanguage with resolveSupportedLanguage in apps/docs/middleware.ts: an RFC 4647 section 3.4 lookup walk that drops one subtag at a time, most specific first, consulting LANGUAGE_MAPPING at every step and returning only through a single membership check against i18n.languages (so it returns a supported locale or undefined, never a raw tag). Both precedence questions are settled in the code and in comments: most-specific-first, and the table beats truncation wherever they disagree because the table is an editorial ruling and truncation is the mechanical default. Six table rows that truncation now subsumes (zh-CN, zh-SG, zh-Hans, zh-Hant, ko, ko-KR) were removed so the table reads as an exceptions list rather than an example that region tags belong in it; a differential run over a 167-tag corpus shows the pruned table and the full one give identical answers on every tag. One file, no changeset (this repo has no changeset flow). Measured before/after against two real builds: de-AT, fr-CA, es-MX, ja-JP moved from English to their own locale, zh-Hant-TW moved from Simplified to Traditional, and every row of the regression half is unchanged.",
  "tests": "All final readings at 2dd7630 (git rev-parse --short HEAD after the last commit), exit codes captured before any pipe, each result read from the command's own verdict line. GATES: `pnpm turbo run type-check --continue` exit 0, 'Tasks: 1 successful, 1 total' with 'cache miss, executing' (not a replay); `pnpm turbo run build --force` exit 0, 'Compiled successfully in 60s'; `pnpm turbo run test --force` exit 0, '3 self-test(s) passed'; `node .github/scripts/check-locale-surface.mjs` exit 0, sitemap 409 read / 409 in scope / 409 distinct / 409 expected / 0 unexpected / 0 missing / 0 duplicated — the 409, not 346, that the dispatch predicted; `node apps/docs/scripts/gen-zh-hant.mjs --check` exit 0, '73 generated file(s) match the zh-Hans sources byte for byte'; `node .github/scripts/check-node-floor.mjs` exit 0, 'Every declared floor clears what the dependency tree requires'. NEGOTIATION MATRIX against a running `next start` on the build of this branch, Accept-Language exactly one tag, no q-list, asserting Location on /docs/quickstart — the full tag-to-Location table (33 single-tag rows, 6 q-list rows, 5 root-path rows, #204's six-case host/legacy table, six /cn/ rows, five dotted machine-facing routes) is in the PR body, not summarised. Now correct: de-AT to /de/, fr-CA to /fr/, es-MX to /es/, ja-JP to /ja/, zh-Hant-TW to /zh-Hant/. Regression half all unchanged: zh-TW, zh-HK, zh-MO, zh-Hant to /zh-Hant/; zh-CN, zh-Hans, zh, zh-SG, zh-Hans-CN to /zh-Hans/; ko, ko-KR to /ko/; 'ja-JP,ja;q=0.9' to /ja/; 'zh-TW,zh;q=0.9' to /zh-Hant/. Matching nothing still lands on English with no invented locale: xx-YY, tlh, *, en-GB all 200 with no Location, and nan-Hant-TW and yue-HK too (truncation does not leak a non-Mandarin tag into zh-Hant through its script subtag). #204's six-case table reproduced identically, including the 200 control on the canonical host and the /cn/ legacy 308. All five dotted routes 200 with no locale prefix under the hardest header (zh-Hant-TW), content-types intact. ABLATION for the before column — this is a real second measurement, not a reading of the source: the fix was committed first, then the pre-fix middleware.ts was checked out over the tree with `git checkout d454ccb -- apps/docs/middleware.ts`; the mutation was confirmed ON DISK by `git hash-object` equalling the d454ccb blob (368a0fa) and not the HEAD blob (0450063), not by an exit code; the tree was then REBUILT with `turbo run build --force` (exit 0, 'Compiled successfully in 103s') because a stale .next would have replayed the fixed behaviour and produced a silently identical before column; the server was restarted and the same matrix script re-run. Restore ran under a `trap ... EXIT INT TERM` using absolute paths and was proven by `git hash-object` again equalling the HEAD blob 0450063, plus empty `git diff HEAD` and empty `git status --short` — not by the trap having fired. Measured before column: de-AT, fr-CA, es-MX, ja-JP, de-at all 200 English; zh-Hant-TW, zh-TW-x-private, zh-Hant-x-private all 307 to /zh-Hans/. The final matrix was then re-run on the final build at 2dd7630 and diffs byte-identical to the first post-fix run. DIFFERENTIAL HARNESS over a 167-tag corpus (every old table key, every supported tag, every card tag, case variants, and a 12-region cross-product over 11 base languages), three algorithms: old two-shot lookup, truncation with the FULL old table, truncation with the shipped pruned table. Result: pruned and full agree on all 167 tags (this is the proof that removing the six rows is behaviour-neutral), and 66 tags change versus the old lookup, every one in the card's direction (13 de-*, 12 es-*, 12 fr-*, 12 ja-* from English to their language; 17 zh-Hant-* / zh-*-x-* from Simplified to Traditional). NO UNIT TEST ADDED: apps/docs has no test runner and `pnpm turbo run test` runs only the .github/scripts self-tests via tools/ci-scripts; adding a runner to apps/docs would have breached the one-file surface and would also claim to test the docs site, which run-self-tests.mjs documents as a misleading-signal shape this repo has already paid for. The matrix in the PR body is the deliverable that makes this checkable next time. CI convergence not awaited, per the dispatch contract: draft PR is open, gates above are the local half.",
  "open_questions": [],
  "out_of_scope_findings": [
    "filed as #220: language-tag matching is case-sensitive, so a lowercase zh-tw / zh-hk / zh-mo / zh-hant / zh-hant-tw negotiates to Simplified instead of Traditional — measured identical on both sides of this fix, unassigned, no labels, and the code comment in middleware.ts now points at it. The other six locales are unaffected because their supported tags are already all-lowercase (de-at and ko-kr resolve correctly), so it is a Traditional-Chinese-only wrong answer. Not fixed here: folding the lookup key is a behaviour change beyond the adjudicated direction, and it needs a decision the card did not make — whether pathnameHasLocale should also accept /zh-hant/... and 308 it to /zh-Hant/..., or keep treating it as a path carrying no locale."
  ]
}

Scope notes for the grader, outside the JSON:

File surface held. One file changed, apps/docs/middleware.ts, +88/-37. No content, no i18n.ts, no scripts — so the oracle the locale gate derives is untouched, which is why 409 is attributable to feat(docs): ship zh-Hant, generated from zh-Hans and committed #218 and not to this branch.
One judgement call worth flagging. Removing the six subsumed LANGUAGE_MAPPING rows was not asked for. It is behaviour-neutral (proven over 167 tags, and every removed row appears in the measured matrix with an unchanged before/after), and it is the half of "not more table rows" that makes the rule legible to the next reader — a table still listing zh-CN, ko-KR and zh-Hans teaches exactly the habit that left de-AT broken. If the maintainer would rather keep them, restoring the six rows is a two-line revert that changes no observable behaviour.
SITE_URL and middleware's CANONICAL_HOST are two literals for one value, with nothing enforcing agreement #205 is held behind this and also touches middleware.ts; it is now unblocked as far as this card is concerned.

Generated by Claude Code

ACCEPT — merging #223.

CI 2/2 green on 2dd7630, one file, exactly the granted surface.

This fixed the cause of the cause

The card asked for truncation instead of more table rows. What landed also explains why the bug existed:

A table containing zh-CN, ko-KR and zh-Hans teaches the next reader — human or model — that region tags belong in the table, which is precisely why de-AT was never added and never worked.

Six rows that truncation subsumes are gone, the table is now a genuine exceptions list, and the comment above it states the rule: add a row only when the mechanical answer is wrong, never when it is merely absent. Without that, the next de-AT gets filed as a missing row, someone adds it, and the class stays open forever.

The remaining rows are exactly the decisions truncation cannot make because the information is not in the tag — bare zh having a house answer, and zh-TW/zh-HK/zh-MO implying a script that dropping the region would discard.

And the pruning is proven behaviour-neutral rather than argued. A differential harness over a 167-tag corpus ran three algorithms; pruned and full agree on all 167. That is the measurement that makes deleting six rows from a live negotiation path a safe change instead of a confident one.

Both precedence questions are answered in the code, with the counterfactual

Most specific first, because short-to-long "would answer Simplified for every Traditional tag that carries a region — the exact defect, with the sign flipped." Stating what the other order would have done is what makes this a decision rather than an ordering that happens to work.
The table beats truncation at every step, because the table is an editorial ruling about what a tag means and truncation is the mechanical default for tags nobody has ruled on. Consulting it at every step rather than only on the full tag is what lets zh-TW-x-private reach Traditional.
The membership check is the only exit. Both paths leave through the same includes, so neither mechanism can hand back a locale the site does not publish — and the consequence is named: a manufactured locale would redirect to a path that 404s and re-enters negotiation.

nan-Hant-TW and yue-HK landing on English is a correctness check nobody asked for: Min Nan and Cantonese do not leak into zh-Hant through the script subtag, because the walk never produces a zh to match on. Truncation over a shared script is exactly where a subtle version of this bug would hide.

The before column is a real second run, and the rebuild is why it means anything

The pre-fix middleware.ts was checked out over the tree, the mutation confirmed on disk by git hash-object matching the d454ccb blob and not the HEAD blob, the tree rebuilt with --force, measured, then restored — restore proven by blob identity plus an empty git diff HEAD and empty git status, not by the trap having fired.

The rebuild is the part worth naming. A stale .next would have replayed the fixed behaviour into the before column and produced a before/after table that is complete, symmetrical, and proves nothing. I have been warning about turbo replaying a sibling's green all day; this is the same failure wearing different clothes, in the app's own build output, and it was caught without being warned about.

The test you did not write

apps/docs has no test runner, and adding one would have breached the one-file surface and claimed to test the docs site — which run-self-tests.mjs documents as a misleading-signal shape this repo has already paid for. Declining with a sourced reason, and treating the 55-row matrix in the PR body as the deliverable that makes this checkable next time, is the right call. "No test" and "no test, here is the repo's own reason why a test here would be worse than none" are different answers.

#220

Graded pm:queue / Bug, queued behind this. Correctly kept out — folding the lookup key is a behaviour change beyond the adjudicated direction, and it needs a decision the card did not make about whether pathnameHasLocale should accept /zh-hant/.... Measuring it identical on both sides of this fix is what establishes it as pre-existing rather than introduced, and pointing the code comment at it means the next reader of that function finds the open question rather than rediscovering it.

Generated by Claude Code

Navigation Menu

Repository navigation

`normalizeLanguage` only tries the full tag and the bare base language, so `de-AT` and `zh-Hant-TW` miss the locale they obviously mean #217

The reading

Why it is not just "add more table rows"

Verification

This is the highest-value card left, and it is not a parity nit

Direction: BCP 47 truncation, as the card says. Not more table rows.

Verification

This fixed the cause of the cause

Both precedence questions are answered in the code, with the counterfactual

The before column is a real second run, and the rebuild is why it means anything

The test you did not write

#220

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

normalizeLanguage only tries the full tag and the bare base language, so de-AT and zh-Hant-TW miss the locale they obviously mean #217

Description

The reading

Why it is not just "add more table rows"

Verification

Activity

hotlong commented on Aug 26, 2026

This is the highest-value card left, and it is not a parity nit

Direction: BCP 47 truncation, as the card says. Not more table rows.

Verification

hotlong commented on Aug 26, 2026

hotlong commented on Aug 26, 2026

This fixed the cause of the cause

Both precedence questions are answered in the code, with the counterfactual

The before column is a real second run, and the rebuild is why it means anything

The test you did not write

#220

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`normalizeLanguage` only tries the full tag and the bare base language, so `de-AT` and `zh-Hant-TW` miss the locale they obviously mean #217