feat: add option to perserve variable syntax in plain text compiler

[HugoHSun]

	Fix CX-98

🧰 Changes

Adds a preserveVariableSyntax option to the plain() text compiler so it can output variables in {user.key} format instead of resolving them to bare key names.

When we index docs for search, we want to keep the variable syntax intact so the frontend can interpolate them with actual user values at display time (e.g. showing "Welcome Owlbert" instead of "Welcome name" in search results).

Both MDX expressions ({user.name}) and legacy <Variable> tags get unified into the same {user.key} output format. Also added a bunch of edge case tests.

Related:

• ReadMe: readmeio/readme/pull/17331
• Gitto: readmeio/gitto/pull/1833

🧬 QA & Testing

npm link @readme/markdown
npx tsx -e "
    const RDMD = require('@readme/markdown');
    const hast = RDMD.hast('{user.name}');
    console.log('default:', RDMD.plain(hast));
    console.log('preserved:', RDMD.plain(hast, { preserveVariableSyntax: true }));
"

Expected output:

default: name
preserved: {user.name}

• Broken on production.
• Working in this PR app.

[dannobytes]

is this fallback technically new or existing behavior in our system?

i'm a lil cautious and worried about having this be our default behavior. from a user and privacy perspective, i would expect user variables to never become publicized, since user var names are technically "private" or internal information. this currently reveals unresolved user var names and lets them show up in search results.

so if a user var is undefined, i'd expect the plain text compiler to simply omit it

    const md = 'Hello {user.name} from {user.company} and good bye';

    expect(plain(hast(md), { variables: { name: 'Owlbert' } })).toBe('Hello Owlbert from and good bye');

[HugoHSun]

Hey! This is actually existing behaviour that predates this PR. The view mode already falls back to the capitalised variable name when rendering docs, and search indexing / table of contents do the same with the key name. This PR doesn't change any of that, it just adds the preserveVariableSyntax option on top for the search indexing path.

I think it's less of a privacy concern since the markdown source is always visible in the frontend anyway, and variable names tend to be pretty generic stuff. Also we can update the behaviour when interpolating in the frontend (readmeio/readme/pull/17331), like omitting it if unresolved at rendering time

[dannobytes]

thank you for adding tests

[dannobytes]

can we give txt some before and after text to check that it correctly interpolates?

Suggested change

	const txt = '{user.name}';
	expect(plain(hast(txt), { preserveVariableSyntax: true })).toBe('{user.name}');
	const txt = 'Hello {user.name} and good bye';
	expect(plain(hast(txt), { preserveVariableSyntax: true })).toBe('Hello {user.name} and good bye');

[dannobytes]

same here, to add before + after text

[dannobytes]

this is a duplicate test of L119, is probably unnecessary and can be removed

[dannobytes]

Suggested change

	/** When true, outputs variables using `{user.key}` syntax instead of resolving
	* to values or bare key names. Used by search indexing so the frontend can
	* interpolate variables at display time. */
	/**
	* When true, outputs variables using `{user.key}` syntax instead of resolving
	* to values or bare key names. Used by search indexing so the frontend can
	* interpolate variables at display time.
	*/

[rafegoldberg]

This PR was released!

🚀 Changes included in v13.3.0