Sitemap and llms.txt disagree about hidden pages #1770

Follow-up noted in #1769. Not urgent, but it needs a decision rather than a fix, so recording it here.

The discrepancy

The two indexes we publish disagree about what counts as a public page:

The three-page difference is exactly the pages marked display: "hidden" in their _meta.ts:

Verified against the live site:

sitemap=1  llms.txt=0   /getting-started/install-memgraph/rocky
sitemap=1  llms.txt=0   /getting-started/install-memgraph/amazon-linux
sitemap=1  llms.txt=0   /database-management/experimental-features

All three return 200 and render publicly — they are hidden from the sidebar, not unpublished.

Why they differ

scripts/generate-llms-txt.mjs reads the _meta.ts files, so it honours display: "hidden" and matches the sidebar. next-sitemap walks build output and has no visibility into _meta.ts, so it lists everything.

Neither is wrong on its own terms. They simply encode different answers to a question nobody has decided.

The actual question

Should a page hidden from the sidebar be advertised to search engines?

Right now we say yes to Google and no to LLM agents. Whichever answer is right, the two indexes should probably agree.

Worth noting the three pages are not a uniform group: rocky and amazon-linux look like install pages kept reachable for existing users, whereas experimental-features is the kind of page one might deliberately keep out of search results. The answer may differ per page, in which case _meta.ts is the wrong signal to key off and an explicit list is better.

Options

1. Exclude them from the sitemap — add an exclude list to next-sitemap.config.js. Simple, but the list is maintained by hand and will drift from _meta.ts.
2. Generate the sitemap from the same page walk — scripts/generate-llms-txt.mjs already builds an ordered, _meta.ts-aware tree of every page. Emitting sitemap.xml from it would keep the two indexes aligned by construction and could replace next-sitemap for the docs. Larger change.
3. Include them in llms.txt — if hidden pages are considered public, drop the display: "hidden" filter from the generator instead.

No action needed until someone decides which way it should go.

Context

• feat: Generate llms.txt and Markdown versions of every docs page #1768 added llms.txt and the per-page Markdown output
• chore: Stop tracking the generated sitemap #1769 untracked the generated sitemap