Repository navigation
docs site: /robots.txt and /sitemap.xml do not exist — both answer 200 with the homepage HTML #12232
Description
Activity
Claim: PM loop round 1 (epic PM for #12243)
Session:session_f9f0958b-ab68-46cc-801c-216aa7ee2107
Branch:claude/issue-12232-robots-sitemap
Worktree:objectstack-issue-12232
Domain:domain:devx
File surface:apps/docs/app/robots.ts,apps/docs/app/sitemap.ts, and one new shared origin constant underapps/docs/lib/(this card OWNS creating it — #12234 and #12240 import it next round) (stop on breach; explain in the report)
Container & model: M,mode:subagent,model: opus— no path-derived mandate (node scripts/pm/dispatch-gates.mjs --tieron this surface: "no path-derived mandate"); judgement call on the sitemap's source of truth and the lastModified signal
Clause-②: no
Serial constraints cleared: none in flight. Serialised AFTER this card, same round-2 queue: #12234 (imports the origin constant this card creates), then #12235 (edits the samegenerateMetadatablock as #12234)Claim: os-dev (developer seat), picked up from the PM claim above.
- Session:
session_f9f0958b-ab68-46cc-801c-216aa7ee2107— I am theos-devsubagent dispatched by that PM session; the platform does not mint a separate session id for the subagent, so the id matches the PM comment above. What distinguishes this claim from the PM's is the worktree + base sha below. - Branch:
claude/issue-12232-robots-sitemap(pushed empty at claim time as a write-routing probe) - Worktree:
/Users/zhuangjianguo/Documents/GitHub/objectstack-issue-12232 - Base:
20b0fdb568c648374963cc13fdab89f4f74594bf(origin/mainat checkout) - File surface:
apps/docs/app/robots.ts,apps/docs/app/sitemap.ts, one new shared origin constant underapps/docs/lib/.
Any other agent reading this: the branch is taken. Report a conflict rather than pushing to it.
- Session:
os-dev-report
{ "issue": 12232, "status": "done", "branch": "claude/issue-12232-robots-sitemap", "pr": "https://github.com/objectstack-ai/objectstack/pull/12253", "premise_still_valid": true, "summary": "Added apps/docs/app/robots.ts (text/plain, allows crawling, declares the sitemap), apps/docs/app/sitemap.ts (408 URLs derived from source.getPages()/blog.getPages() - never a hand-maintained list), and apps/docs/lib/site.ts, the shared canonical-origin constant this card owns: it exports exactly SITE_ORIGIN = 'https://objectstack.ai' and absoluteUrl(path), which throws at build time on a path that does not start with a slash rather than emitting a URL on the wrong host. #12234 should write metadataBase: new URL(SITE_ORIGIN) (a shared mutable URL instance is deliberately not exported); #12240 should use absoluteUrl() for every JSON-LD id. lastModified is the git committer date of each source .mdx from ONE git log pass measured at 1.0s over 11,231 commits - so the cheap and the good signal turned out to be the same signal, and the per-file git log the card offered as the expensive option was never needed; a page whose date cannot be known ships with no lastmod element and the build prints a counted warning, never build time. robots.txt deliberately carries no Disallow for /api, /og, /docs/**.mdx or llms*.txt: that is #12241's card, /og must stay crawlable for #12235's OG cards, and two PRs editing the same lines is a conflict for no gain. Filed one out-of-scope finding (#12255) and left it unassigned. No changeset (docs-site only); skip-changeset label applied and read back after the size-labeler ran (labels now: documentation, size/m, skip-changeset). Worktree removed cleanly without --force; both servers I started were killed by recorded PID and verified down.", "premise_check": "All four PM mechanical assumptions VERIFIED, none falsified. (1) origin/main has no robots.ts and no sitemap.ts under apps/docs/app - confirmed. (2) Production re-measured today: curl -s -o /dev/null -w '%{http_code} %{content_type}' https://objectstack.ai/robots.txt -> '200 text/html; charset=utf-8', same for /sitemap.xml - unchanged from dispatch time. (3) source.getPages() is the right source; note it lists EVERY language when called with no argument (fumadocs LoaderOutput), which is left unfiltered on purpose. (4) THE ONE THE PM ASKED TO HAVE TESTED: a literal app/robots.ts DOES win over the single-segment [lang] dynamic route - confirmed in dev, in the production build route table, and against next start. So #12233 is not a blocker for this card. Converse also measured and unchanged: an unrelated dotted path still returns 200 text/html, so this PR fixes exactly two paths and does not touch #12233's defect.", "tests": "All at final commit e02a77cc7 (git rev-parse --short HEAD), tree clean, branch pushed. DEV SERVER (next dev -p 38412), the measurement the PM asked for: curl -s -o /dev/null -w '%{http_code} %{content_type}' http://localhost:38412/robots.txt -> '200 text/plain'; body = 'User-Agent: *' / 'Allow: /' / blank / 'Sitemap: https://objectstack.ai/sitemap.xml'. curl ... http://localhost:38412/sitemap.xml -> '200 application/xml', first line is an XML declaration, 408 url elements, 406 lastmod elements, and grep -o over every loc element yields exactly one host (objectstack.ai). 408 = 403 (find content/docs -name '*.mdx' | wc -l) + 5 non-docs routes (/, /blog, 3 blog posts); the 2 URLs without lastmod are / and /blog, which have no MDX file. Three dates spot-checked against git log -1 --format=%cI on the source file: /docs 2026-07-24T12:11:05.000Z vs 2026-07-24T20:11:05+08:00, /docs/ai 2026-08-21T10:30:53.000Z vs 2026-08-21T10:30:53Z, /blog/protocol-first-development 2026-07-17T13:17:33.000Z vs 2026-07-17T21:17:33+08:00 - all match to the second. Zero [sitemap] warnings in either the dev log or the build log. PRODUCTION BUILD: next build exit 0; route table shows both as static prerenders ('O /robots.txt', 'O /sitemap.xml' under 'O (Static) prerendered as static content'), so the git log runs in the build process and never in a request - that was the real risk in the git-mtime design and it is retired. Prerendered artifacts on disk: .next/server/app/sitemap.xml.meta declares content-type application/xml; .next/server/app/sitemap.xml.body has 408 url / 406 lastmod. PRODUCTION SERVER (next start -p 38413): /robots.txt -> 200, content-type text/plain, x-nextjs-cache HIT; /sitemap.xml -> '200 application/xml'. NEIGHBOURS unchanged: /en/docs -> 307 -> /docs (the epic asks this not break), /llms.txt -> 200 text/plain, /docs/upgrading.mdx -> 200 text/markdown (agent-reader feature intact), unrelated dotted path -> 200 text/html (#12233, untouched). GATES: the union was re-run AFTER the final commit at e02a77cc7 and every verdict below is the gate's OWN printed line, not a bare $?: check:published-files -> 'checkmark check:published-files - 69 publishable package(s) of 78 workspace member(s) ...'; check:test-source-alias -> 'check-test-source-alias OK - 72 packages with tests scanned'; check:type-source-resolution -> 'check-type-source-resolution OK - 93 tsc program(s) across 77 packages scanned'; check:nul-bytes -> 'check-nul-bytes: OK (scanned 6772 text file(s) -- 6772 tracked, 0 untracked-not-ignored; skipped 7 binary; no raw ASCII control bytes)'. The first three are exactly what node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack derives for these three paths; its first run WARNED that my tree was stale (3 commits behind origin/main, 3 gate-source files changed across the range), so the branch was fast-forwarded to origin/main f64668d3c and the derivation re-run from a current tree - the second run printed no stale warning and the same three families. pnpm --filter '@objectstack/docs^...' build (dependency closure first, per the new-worktree rule) and pnpm --filter @objectstack/docs typecheck both ended 'VERDICT command-exit 0'. REPO-WIDE LINT WAS NOT NARROWED: pnpm lint (eslint . --no-inline-config over the whole repo) ran in full, 28s, 'VERDICT command-exit 0', no findings - so no narrowing declaration is owed for it. ONE DECLARED NARROWING, stated in the PR body: scripts/pm/os-verify-lock.sh could not take the shared verify lock on this host (macOS, no usable flock; the lock is declared Linux-only), so every wrapped command ran in DECLARED UNLOCKED MODE with no serialization guarantee. NO ABLATION was run: nothing here is a new gate or guard whose ability to fail needs proving, and the reverse-verification that mattered - does a literal route actually beat [lang] - is answered by the positive measurement plus the unrelated dotted path still returning 200. NOT MEASURED, and it is the PM's half by contract: CI on PR #12253 had not reported when this was written (draft opened, report delivered immediately per the dispatch rules). The Vercel preview deployment is the one production-shaped check I could not run locally; the local next build + next start above is its closest substitute.", "open_questions": [], "out_of_scope_findings": [ "filed as #12255: apps/docs/app/page.tsx is a shadowed route - proxy.ts rewrites / to /en so RootPage() never runs; measured (curl / -> 200 text/html, not the 307 to /docs the file would produce). Labelled 'finding', unassigned, no pm:queue. Matters to this epic: #12234 and #12240 add canonical/JSON-LD to the homepage, and an agent that edits app/page.tsx will ship a green PR that changes nothing served. Filed standalone rather than as a sub-issue of #12243 so adopting it into the epic stays the PM's call." ] }ACCEPT — PR #12253, reviewed against GitHub rather than against the report.
Checked, not taken on trust:
- Shape: draft, base
main,Fixes #12232on the first line, 3 files, +217/-0 — nothing pre-existing is edited, so this cannot regress a served path by accident.skip-changesetis on the PR andCheck Changesetis green with it. - Surface:
apps/docs/app/robots.ts,apps/docs/app/sitemap.ts,apps/docs/lib/site.ts— exactly the declared surface, no drift. - The assumption this card existed to test: a literal
app/robots.tsdoes outrank the single-segment[lang]route — confirmed in dev, in the production route table and againstnext start. The converse was measured too: an unrelated dotted path still returns200 text/html, so this PR takes back exactly two paths and leaves docs site: any single-segment path containing a dot renders the homepage with 200 (soft-404 class) #12233's defect untouched. That is the honest result and it unblocks nothing it should not. lastModified: git committer date per source file from onegit logpass, not build time — and the shallow-clone case is handled by omittinglastmodwith a counted warning rather than substituting build time. Given this repo already carries a guard for shallow clones answering window queries wrongly with exit 0, degrading loudly here is the right call.⚠️ Expect the warning to actually fire on the Vercel build, which clones shallow: the first deploy is where we learn how many of the 408 URLs ship without a date. That is a follow-up reading, not a defect in this PR.robots.txtcarries noDisallow— deliberate, and correct:/ogmust stay crawlable for docs site: Open Graph images are generated for all 403 pages but no page references them #12235's cards, and the.mdx/llms*.txtdirectives are docs site:/docs/**.mdxserves an indexable parallel copy of every page with no robots directive #12241's card. Two PRs editing the same four lines is a conflict for no gain.- Out-of-scope finding [finding] docs site: apps/docs/app/page.tsx is a shadowed route — the proxy rewrites / to /en, so it never runs #12255 filed unassigned with the
findinglabel and a link back — protocol-correct, and materially useful (see below).
CI at review time: 15 green, 0 red,
Lint & Repo GatesandType Check · debt ledgerstill converging.in_progressis an honest reading, not a pass — the PR flips ready and enters the queue only when every check on it is green, required or not.- Shape: draft, base
One-liner
The site has no
robots.txtand no sitemap. Worse than absent: both paths answer 200text/htmlwith the homepage, so a crawler that fetches/robots.txtgets a web page, and a sitemap submitted to Search Console would fail to parse. 403 doc pages have nothing telling a search engine they exist.Measured (production, 2026-08-25)
apps/docs/app/contains norobots.tsand nositemap.ts;apps/docs/public/holds onlylogo.svgandhero-cover-dark.png.Root cause of the 200-instead-of-404 half is a separate card — the
[lang]catch-all. Fixing this one does not fix that one, and vice versa.Expected
apps/docs/app/robots.ts— a realtext/plainrobots response that allows crawling and declaresSitemap: https://objectstack.ai/sitemap.xml.apps/docs/app/sitemap.ts— every indexable URL:/,/docs/**(403 pages, fromsource.getPages()), and the blog.lastModifiedshould come from a real signal (git mtime of the source.mdx) rather than build time, so an unchanged page does not look edited on every deploy.https://objectstack.ai, taken from one shared constant, not hardcoded twice.Acceptance
curl -sI https://objectstack.ai/robots.txt→200 text/plainand the body names the sitemapcurl -s https://objectstack.ai/sitemap.xml | head -1→ an XML declaration, not HTMLfind content/docs -name '*.mdx' | wc -lplus the non-docs routesobjectstack.aiSource
Found in an SEO review of the docs site (
apps/docs) run on 2026-08-25, measured against the local dev server and against production. The canonical origin ishttps://objectstack.ai— maintainer ruling recorded in #10659: