How should I ... do ... robots.txt? Sitemap?

[kylehodgson]

I spent some time on a quick node script that pulls in paths entries and my dynamicPaths generator function from observable.config.jsand hands them to npm'ssitemap` library. Now I have a nice little sitemap.xml, and I'd like to reference it in my robots.txt. Which is easy! Except, how do I deploy either of these?

I saw documentation somewhere about adding "external" files not managed by observable, and I did add this to head in observable.config.js:

<a href="/robots.txt" rel="external">robots.txt</a>
<a href="/robots.txt" rel="external">sitemap.xml</a>

npm run build doesn't copy robots.txt to the dist folder. I can do this as part of the CI pipeline of course - but I wanted to see if I was just missing something?

[Fil]

One way to do this is to use the dynamicPaths option to add references to files that must be built at a given URL. For example, see https://github.com/Fil/pangea/blob/c7b2ee2fec2aead7b2741e7256556c16b4c04ebc/observablehq.config.ts#L16C17-L16C29

See #1199 for a related discussion.

[kylehodgson]

I ended up doing it just after build. I did try it with a postBuild task but my CI process (Cloudflare pages) wasn't triggering it, so I just stuffed it in to build.

Package.json:

"build": "observable build && node src/sitemapper.js | npx sitemap > dist/sitemap.xml && cat src/robots.txt > dist/robots.txt"

Sitemapper.js:

import * as of from "../observablehq.config.js";

function formPath(path) {
    return  "https://myiste.tld" + path
}

const site_pages = [];

for await (const value of of.default.dynamicPaths()) {
  site_pages.push(formPath(value));
}

of.default.pages.forEach(node => {
    node.pages.map(subpage => subpage.path.substring(0,4)!=='http' && site_pages.push(formPath(subpage.path)) );
});

process.stdout.write(site_pages.join("\n"));

the npm sitemap package can do much fancier things, of course, but this was enough for my needs.