fix: respect explicit Accept: text/html from AI bot user agents

[abhay-codes07]

Summary

Bot user agents currently get the markdown twin even when they explicitly ask for Accept: text/html. The AEO spec says the opposite should happen, so this fixes the adapters to honor an explicit Accept while keeping the existing bot behavior for everything else.

Closes #69.

Changes

• Add shouldServeMarkdown(accept, isBot) to @dualmark/core. It serves markdown when RFC 7231 negotiation already picks markdown, or when the request is from a known bot and markdown is at least as acceptable as HTML. It checks the second case by negotiating with markdown listed first, so a tie (wildcard or empty Accept) resolves to markdown for bots, while an explicit Accept: text/html keeps the bot on HTML.
• Use the helper in the cloudflare, deno, fastly, netlify, nextjs, sveltekit, vercel and nuxt adapters, replacing the bot.isBot || fmt === "markdown" check that ignored the Accept header for bots.
• Add a regression test in each adapter: a GPTBot/1.0 UA with Accept: text/html now receives HTML, not a noindex markdown body.
• Add unit tests for the helper in @dualmark/core.

Behavior is unchanged for Accept: */*, no Accept, and Accept: text/markdown, so the existing conformance scores and bot handling stay the same.

Type

• Bug fix (non-breaking)
• New feature (non-breaking)
• Breaking change
• Spec change (requires bump in AEO_SPEC_VERSION)
• Docs / examples / tooling only

Verification

• bun run build passes
• bun run test passes (541 tests across the packages)
• bun run typecheck passes
• If touching examples: ran dualmark verify against the example end-to-end (no examples touched)
• If touching /spec/: ran sync-spec and committed the mirror (no spec files touched)

Changeset

• Added a changeset (@dualmark/core minor, adapters patch)

---

cc @thepushkaraj @aagarwal1012 for a look when you have a minute(file change might look scary but indeed needed have a look). Happy to adjust the approach if you would rather keep the decision logic inside each adapter instead of a shared core helper.

[vercel]

@abhay-codes07 is attempting to deploy a commit to the Dodo Payments Team on Vercel.

A member of the Team first needs to authorize it.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[thepushkaraj]

@dodo-squirrels review

[dodo-squirrels]

Review: looks good, approving the approach

I reviewed the change end-to-end and it's well-implemented and correct.

Core logic — shouldServeMarkdown(accept, isBot) in packages/core/src/negotiation.ts is a clean encoding of AEO spec §5. It serves markdown when (1) RFC 7231 negotiation already picks markdown, or (2) the request is from a known bot and markdown is at least as acceptable as HTML (checked by negotiating with markdown listed first so ties resolve to markdown). An explicit Accept: text/html where HTML strictly outranks markdown keeps the bot on HTML.

Edge cases verified:

• Accept: */*, text/*, and no Accept → still markdown for bots (unchanged behavior).
• Accept: text/markdown → markdown for any UA (unchanged).
• Browser-like Accept: text/html,...,*/*;q=0.8 from a bot UA → now correctly stays on HTML.
• Unacceptable types (e.g. Accept: image/png) are still handled by the pre-existing fmt === null && accept 406 guard, which runs before shouldServeMarkdown in every adapter — so no misrouting there.

Consistency — all 8 edge/framework adapters (cloudflare, deno, fastly, netlify, nextjs, nuxt ×2, sveltekit, vercel) were updated identically and each passes the raw accept header. The Astro adapter is correctly excluded since it does no UA-based negotiation.

Versioning — changeset is appropriate: @dualmark/core minor (new public export), adapters patch (bug fix).

Validation — checked out the branch and ran the suites locally:

• Full test suite: all 24 test tasks pass (incl. the new per-adapter and core shouldServeMarkdown tests).
• Typecheck: all 25 tasks pass, 0 errors.

No changes required. LGTM.

[dodo-squirrels]

Review: fix: respect explicit Accept: text/html from AI bot user agents

Verdict: LGTM — approving. One minor edge-case behavior change noted below (non-blocking).

Reviewed end-to-end: new @dualmark/core primitive + all 8 adapters, exercised the core logic against a full truth table.

Spec alignment

content-negotiation spec §5 is explicit: UA-based markdown is an OPTIONAL extension that "MUST still respect explicit Accept: text/html from a bot UA (i.e. UA detection MUST NOT override an explicit Accept)." On main, every adapter used bot.isBot || fmt === "markdown", so a crawler sending a bot UA and Accept: text/html (e.g. a preview/link-unfurl fetch) was handed text/markdown + X-Robots-Tag: noindex on the canonical URL — a real spec violation. This fixes it.

shouldServeMarkdown logic — verified

The two-line implementation is subtle but correct:

1. negotiateFormat(accept) === "markdown" → default order prefers HTML on ties, so this is true only when markdown strictly wins (covers Accept: text/markdown for any UA).
2. bot branch: negotiateFormat(accept, ["markdown", "html"]) → markdown listed first, so a tie (wildcard / no Accept) resolves to markdown for bots, while explicit text/html (HTML strictly outranks, or markdown unacceptable) keeps the bot on HTML.

I exercised the full matrix on the branch:

Accept	bot	result
text/markdown	any	md	✓
"" / */* / text/*	bot	md	✓
text/html	bot	HTML	✓ the fix
text/html,…,*/*;q=0.8 (browser-like)	bot	HTML	✓
"" / */* / text/html	human	HTML	✓
text/html;q=0.5, text/markdown;q=0.5	bot	md	✓
same tie	human	HTML	✓

Adapter integration — verified

All 8 adapters (cloudflare, deno, fastly, netlify, nextjs, sveltekit, vercel, nuxt ×2 sites) now route through shouldServeMarkdown, and each ships the same regression test. The pre-existing if (fmt === null && accept) → 406 guard runs before shouldServeMarkdown in the edge adapters, so the "Accept excludes everything" case still correctly 406s rather than reaching this function.

Minor note (non-blocking)

In the Nuxt middleware the 406 guard is format === null && !botInfo.isBot, so a bot sending an Accept that excludes both types (e.g. Accept: image/png) now falls through to HTML SSR instead of the old behavior of serving markdown. That's arguably an improvement (no noindex markdown for a mismatched Accept) and still spec-defensible, but it is a behavior change on that edge — worth a one-line test if you want to pin it.

Tests

• Full suites green locally across all affected packages: core 185, cloudflare 24, deno 24, fastly 20, netlify 28, nextjs 51, sveltekit 20, vercel 29, nuxt 50.

Clean primitive, consistent rollout, good test coverage, changeset with correct semver bumps (core minor for the new export, adapters patch). Nice work.