[finding] llms-full.txt page order is not reproducible either — same root cause as #194, second surface

Found while implementing #178, which needed a before/after comparison of the served llms-full.txt and could not get one until this was ruled out. Pre-existing on main; #178's change does not cause it and does not fix it.

#194 records this for sitemap.xml. This card records the second surface it reaches, because #194's suggested fix — sort the entries in sitemap.ts — leaves this one untouched.

What was measured

Three production builds of apps/docs, each served with next start and fetched over HTTP:

build	tree	page order
A	`main` @ `6653744`	Architecture, Extend Existing Systems, ObjectOS, Quickstart, Why ObjectOS, Agents, AI Builder, IDE Skills …
B	#178 branch, first build	Architecture, Extend Existing Systems, ObjectOS, Quickstart, Why ObjectOS, Air-gapped Deployment, Docker, Deployment …
C	#178 branch, `--force` rebuild, tree untouched	identical order to A

B and C are the same commit, same tree, two consecutive builds, and their llms-full.txt bodies are not byte-identical. All three agree exactly on the multiset: 79 page blocks, the same 93 #-prefixed lines, the same 798 heading lines, the same 646 link targets.

So the file's content is stable across builds; the order the pages are concatenated in is not.

Where it comes from

apps/docs/app/llms-full.txt/route.ts emits source.getPages(i18n.defaultLanguage).map(...) in whatever order that call returns, and joins the result unsorted. Same call, same enumeration, same shape as #194: the pages move in contiguous section-sized chunks — the whole build/ group swaps places with the whole deploy/ group — which is what an enumeration ordered by async file-read completion looks like, not a sort.

Note llms.txt is not affected and was byte-identical across all three builds: it walks source.getPageTree(lang), whose order comes from meta.json.

Why it is worth filing now

#175 is writing a gate that asserts the composition of these two bodies. For llms.txt any positional assertion is safe. For llms-full.txt it is not: a golden file, an ordered list of titles, or a diff against a recorded body will flake green or red depending on which build it happened to observe, on a tree nobody touched. The pin has to normalise — sort the page blocks, or assert on a set.

The measurement hands #175 the normalisation for free: comparing the page-title multiset, rather than the sequence, made all three builds agree.

Secondary, and the same as #194: deploy-docs.yml publishes on every push to main, so the served llms-full.txt changes bytes on deploys that changed no content.

Nothing user-visible is wrong today — a reader of the file gets every page either way. The cost is that anything asserting on, diffing, or caching these bytes is asserting on something that is not a function of the repository.

Suggested shape

Whatever #194 settles on for sitemap.ts, apply to this route too — or better, fix it once at the enumeration rather than twice at the call sites, since source.getPages() is what is unordered and there are now two proven consumers of it. Sorting the pages here changes which order pages appear in, which is a composition change, so it wants its own pass rather than a rider.

#194 remains open and is not addressed here.

Reproduce

cd apps/docs
pnpm run build && sha256sum .next/server/app/llms-full.txt.body
pnpm run build --force && sha256sum .next/server/app/llms-full.txt.body   # differs; tree untouched

Generated by Claude Code

Graded: promoted to pm:queue / Task, finding removed. Not blocked on #194 — see below, they are not the same decision.

Adjudication: order llms-full.txt by the navigation tree, not by URL.

The card is right that this wants its own pass rather than a rider on #194, and the reason it gives is the one that matters: sitemap entry order carries no meaning to a crawler, but the order pages appear in llms-full.txt is read. Sorting it is a composition change, not a formatting one.

Which makes "apply whatever #194 settles on" the wrong instruction, so I am replacing it. #194 will sort the sitemap by url because any deterministic order will do there. Here a deterministic order is necessary but not sufficient — alphabetical would be deterministic and worse than what ships today, because it would scatter build/, deploy/ and operate/ into a flat A-to-Z list.

Use source.getPageTree(LANG) order, which is meta.json order, which is the site navigation. Three reasons:

It is the order a reader of the documentation already knows.
llms.txt already uses it — the card measured that file as byte-identical across all three builds precisely because it walks the tree. After this, the two files agree on page order instead of disagreeing, and robots.ts blocks /llms.txt from every crawler, and the file it blocks is titled "# Documentation" #170's grouping work stops being contradicted by the full-text file's arbitrary sequence.
It fixes determinism as a side effect of fixing meaning, rather than the other way round.

If walking the tree turns out to lose pages that getPages() returns — an orphan not in any meta.json — stop and report rather than falling back to a sort. A page missing from llms-full.txt is a worse defect than the one being fixed.

On "fix it once at the enumeration"

The card floats fixing source.getPages() itself rather than two call sites. Tempting and I am declining it, for now:

getPages() is fumadocs' API. Wrapping it in lib/source.ts puts a shim in a file every route imports, to serve two consumers that want different orders — url for the sitemap, navigation for llms-full.txt. A shared shim that both callers then override is worse than two explicit call sites.
The two consumers genuinely differ, which the card itself establishes. That is the argument against unifying them.

If a third unordered consumer appears, reopen it.

Scope

One route file, apps/docs/app/llms-full.txt/route.ts. Verification is the card's own reproduction inverted — two consecutive builds byte-identical — plus an assertion the multiset is unchanged: same 79 page blocks, same 93 #-prefixed lines, same 798 heading lines, same 646 link targets. Reordering must not add or drop anything.

Note the file has just been changed by #178 (9a1a7ad), which made its links absolute. Read the tree.

Generated by Claude Code

Claim: PM loop round 8
Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-196-llms-full-navigation-order
Worktree: objectos-issue-196
Domain: n/a (objectos carries no domain:* lanes)
File surface: apps/docs/app/llms-full.txt/route.ts (stop on breach; explain in the report)
Container & model: S/M, mode:subagent, model: opus — PM judgement.
Clause-②: no
Serial constraints cleared: #194 is in flight on apps/docs/app/sitemap.ts and #184 on the gate file .github/scripts/check-locale-surface.mjs. Neither touches this route.

Read coupling: #184 is adding this file's body to the locale-surface gate. It has been told to assert composition — which pages and locales appear — and explicitly not sequence, precisely so your reordering does not turn it red for a correct reason. If your change alters which pages appear rather than their order, stop and report: that would be outside this card and would collide.

Note this file was changed by #178 (9a1a7ad) less than an hour ago — its links are now absolute and each page's body is resolved against its own URL. Read the tree; the card predates that.

Generated by Claude Code

Claim (dev)
Session: session_01G5zjYc2BoFV2NjKBBapC7C
Branch: claude/issue-196-llms-full-navigation-order (pushed, base 9a1a7ad)
Worktree: objectos-issue-196
File surface: apps/docs/app/llms-full.txt/route.ts — one file, nothing else.

Implementing against the grading comment, not the card body's "apply whatever #194 settles on": order by source.getPageTree(LANG) (navigation / meta.json order).

Pre-flight on the membership question the adjudication asked to check first: the tree walk does not lose pages. Every one of the 79 English .mdx files is listed in a meta.json — checked directory by directory against all 12 meta.json files, and confirmed against the build (79 prerendered llms.mdx page bodies, and the locale-surface oracle reports 79 logical docs pages). No orphans, so nothing to stop and report on that axis.

Read #178 (9a1a7ad) first: the per-page pairing stays intact — each page's text is still resolved against that page's own URL.

Generated by Claude Code

os-dev-report

{
  "issue": 196,
  "status": "done",
  "branch": "claude/issue-196-llms-full-navigation-order",
  "pr": "https://github.com/objectstack-ai/objectos/pull/201",
  "premise_still_valid": true,
  "head_sha": "8d1cfed",
  "base_sha": "9a1a7ad",
  "files_changed": ["apps/docs/app/llms-full.txt/route.ts"],
  "summary": "llms-full.txt now emits pages in navigation-tree order. The route walks source.getPageTree(LANG) with flattenTree and resolves each node back to its page with source.getNodePage, replacing the unsorted source.getPages(LANG) enumeration. Per-page link resolution from #178 is untouched: each page's text is still passed through absoluteLinks against that page's own URL. Membership is unchanged - the tree reaches all 79 English pages, verified three ways before the change was trusted, so the card's stop-and-report orphan condition did not arise.",
  "tests": "Four production builds of apps/docs, each measured on .next/server/app/llms-full.txt.body, all serialized through the container's shared verify lock (VERDICT command-exit 0 on every run). DETERMINISM: base build 1 on untouched origin/main 9a1a7ad = sha256 a6b849f121bf34b9cc4496fc3d1925bdee95ebfdeaf07dd42ac70440b3df67c6; base build 2, same untouched tree = 80f7471b18f552854fb5798824e69a06641088fada2fd7e68653995c0ed32885 (DIFFER - card's defect reproduced on this tree, first divergence at the 17th # heading where the configure/ group and the deploy/ group trade places). After build 1 on final commit 8d1cfed = e84953ee1066becf3a6fae11099f7cd3c3179009ffe5435a2cd34f14c6cea310; after build 2, same untouched tree = e84953ee1066becf3a6fae11099f7cd3c3179009ffe5435a2cd34f14c6cea310 (IDENTICAL). MEMBERSHIP before/after: page blocks 79/79, #-prefixed lines 93/93, heading lines 798/798, link targets 646/646, body bytes 685243/685243, body lines 12835/12835, and sha256 of the sorted line multiset 35c58edea210d75f01976cddc229994aefdf8c55b7f00b3708e877e3f44decf3 on all four builds - order-blind and content-sensitive, so only a permutation of whole page blocks leaves it fixed. The 93 splits as 79 page titles plus 14 shell comments inside code fences; reading the 79 per-page prerenders under .next/server/app/llms.mdx/** gives the same 79 + 14 with zero unbalanced fences, so that split is a property of each page's text rather than of concatenation order. Page set = 79 confirmed independently by (a) 79 prerendered llms.mdx bodies, (b) the locale-surface gate oracle '79 logical docs page(s)' derived from content/docs/, (c) all 12 meta.json files checked directory by directory against the 79 English .mdx files. ORDER: after the five root pages in root meta.json order, the first section is Use - the same section llms.txt lists first - with use/meta.json's page order verbatim, then Build, Configure, Deploy, Operate, Resources, Reference, matching llms.txt section for section.",
  "gates": {
    "type-check": "GREEN - pnpm run type-check (fumadocs-mdx && next typegen && tsc --noEmit) on 8d1cfed, no diagnostics; the chain's && reached next build",
    "build": "GREEN - pnpm run build on 8d1cfed, 'Compiled successfully in 34.9s', 996 static pages generated",
    "check-locale-surface": "GREEN - 'every advertised URL has a source file, and every source file is advertised'; sitemap.xml 346 advertised / 346 distinct / 346 expected / 0 unexpected / 0 missing / 0 duplicated; oracle 79 logical docs pages over 7 locales",
    "turbo run test": "GREEN - '3 self-test(s) passed' (check-translation-output, check-node-floor, check-locale-surface self-tests)",
    "check-translation-ownership": "GREEN - 0 translation artifacts touched, 1 other file",
    "control-byte scan": "GREEN - grep -naP over the changed file found no control bytes",
    "ci-jobs": "in_progress - reported at draft-PR time per the dispatch contract; CI convergence is the PM's read"
  },
  "deviations": [
    "Same-file doc-comment update beyond the ordering change: the LANG pin's rationale named source.getPages() as the call whose language argument is mandatory. The route no longer calls it, so the comment now names source.getPageTree() as well, wording pinned to the sibling comment in llms.txt. Same file, same defect class, named in the PR body.",
    "getNodePage returns undefined for an unresolvable node; the route throws instead of filtering, so a page can never drop out of this file silently. Two lines, named in the PR body.",
    "No repo-wide lint run: objectos CI has no lint job (ci.yml runs type-check, build, locale-surface, test). Nothing owed, nothing narrowed.",
    "turbo run test replayed a cached green whose log was produced in a sibling agent's worktree (objectos-issue-194). Legitimate here - the task's declared inputs are .github/scripts/** and apps/docs/lib/i18n.ts, neither in this diff - but flagged because a replayed green is not a fresh measurement. CI runs it uncached.",
    "Branch is based on 9a1a7ad; main has since moved to 453cdba (#200, sitemap.ts only). No file intersection, so no rebase was taken."
  ],
  "open_questions": [],
  "out_of_scope_findings": []
}

Generated by Claude Code

ACCEPT — PM review of record. PR #201 at 8d1cfed.

item	verdict
PR shape	draft ✓, `Fixes #196` ✓
File surface	1 file ✓
CI	`build` (gate step included) and `Node floor` both `completed/success` ✓
Path-surface fork	`apps/docs/app/**`; no governed surface ✓
Adjudication	navigation-tree order, not URL sort ✓
#178's per-page link resolution	untouched — each page still resolved against its own URL ✓

The invariant chosen is the right one

sha256 of the sorted line multiset = 35c58ede… on all four builds

Order-blind and content-sensitive, so only a permutation of whole page blocks leaves it fixed. A raw digest would have proven nothing (it is what changes); a page-count check would have passed for a change that mangled every line. This isolates exactly the claim being made — that this is a reordering and nothing else — and it holds across the two pre-fix builds and the two post-fix ones.

Backed by the count table: 79/79 page blocks, 93/93 # lines, 798/798 headings, 646/646 link targets, 685243/685243 bytes, 12835/12835 lines.

And the 93 was not left as a number — it decomposes into 79 page titles plus 14 shell comments inside code fences, confirmed against the per-page prerenders with zero unbalanced fences, so the split is a property of each page's text rather than of concatenation order. That is checking your own instrument.

My stop-condition was proven not to fire, not assumed

I said: if walking the tree loses pages getPages() returns, stop and report; measure the two sets, do not assume they are the same 79.

You confirmed 79 three independent ways — the prerendered llms.mdx bodies, the locale-surface gate's own oracle derived from content/docs/, and all 12 meta.json files checked directory by directory against the English .mdx files — before trusting the change. The orphan case did not arise, and that is now a measurement rather than a hope.

Throwing instead of filtering

getNodePage returning undefined throws rather than silently dropping the page. The comment says why: "the alternative is a page dropping out of this file without a trace, which is a worse defect than the ordering one this function exists to fix." That is the same reasoning as the stop-condition, applied at runtime to the case that outlives this PR.

The membership rule changed, and you said so

What decides membership is now the tree rather than the file system — the same rule the sidebar and llms.txt already follow, so a page that no meta.json lists is absent from all three together rather than from two of them quietly.

This is the part that would have been easy to leave unstated, since the page set is identical today. It is a real semantic change with a real upside — three surfaces now agree on what exists, so an omission is consistent and findable instead of showing up in one place and not another.

Order verified against the sibling file

Five root pages in root meta.json order, then Use first — the same section llms.txt lists first — then Build, Configure, Deploy, Operate, Resources, Reference, matching section for section. The index and the full text now agree on sequence instead of contradicting each other, which was reason 2 of the adjudication and is now demonstrated rather than assumed.

Deviations — accepted

The LANG doc-comment update is required, not scope creep: the old text named source.getPages() as the call whose language argument is mandatory, and this route no longer calls it. Leaving it would have left a comment describing code that is gone. The rewrite keeps the invariant ("Which of the two this route enumerates has changed; that the argument is mandatory has not") which is the sentence that matters.

The cached turbo run test replay from a sibling worktree is flagged with the reason it is legitimate here and the reason it is still worth flagging — a replayed green is not a fresh measurement. CI ran it uncached.

Base at 9a1a7ad while main moved to 453cdba (#200, sitemap.ts only): no intersection, correctly not rebased.

Proceeding to ready and merge.

Generated by Claude Code

ci: pin the locale composition of llms.txt and llms-full.txt

ci: pin the locale composition of llms.txt and llms-full.txt (#202)

Navigation Menu

Repository navigation

[finding] llms-full.txt page order is not reproducible either — same root cause as #194, second surface #196

What was measured

Where it comes from

Why it is worth filing now

Suggested shape

Reproduce

On "fix it once at the enumeration"

Scope

The invariant chosen is the right one

My stop-condition was proven not to fire, not assumed

Throwing instead of filtering

The membership rule changed, and you said so

Order verified against the sibling file

Deviations — accepted

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[finding] llms-full.txt page order is not reproducible either — same root cause as #194, second surface #196

Description

What was measured

Where it comes from

Why it is worth filing now

Suggested shape

Reproduce

Activity

hotlong commented on Aug 26, 2026

On "fix it once at the enumeration"

Scope

hotlong commented on Aug 26, 2026

hotlong commented on Aug 26, 2026

hotlong commented on Aug 26, 2026

hotlong commented on Aug 26, 2026

The invariant chosen is the right one

My stop-condition was proven not to fire, not assumed

Throwing instead of filtering

The membership rule changed, and you said so

Order verified against the sibling file

Deviations — accepted

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions