CommonMark spec gaps grouped by root cause, and a question about lookup tables

I ran the spec suite on main (18.0.9, 681373c) and went through the cases carrying shouldFail to see what is actually left. Since update-specs.js sets that flag from whatever fails at the time it runs, the set is a snapshot of current behavior rather than a list of decisions, so I grouped it by root cause. Posting it in case it is useful for triage, and because two of the groups run into the same wall and that seems worth asking about before anyone starts on them.

CommonMark 0.31.2, gfm: false, 15 cases:

Root cause	Examples	Count
Character references in link destination and title	32, 33, 503	3
Nested brackets in link text	512, 518, 519, 520, 528, 532, 533	7
Raw HTML and autolinks taking precedence over links	524, 526, 536, 538	4
Unicode case folding of link labels	540	1

Running the same file with gfm: true flags 4 more (602, 608, 611, 612), but those look like the autolink extension doing its job against a spec file written before it existed. For instance 611 expects https://example.com to stay plain text and marked links it, which is what GFM asks for. I would not count those as gaps.

gfm.0.29 has 4: 633, 634 and 635 are mailto: and xmpp: autolinks, 657 is the disallowed raw HTML filter.

The wall

The two smallest groups look approachable until you try them, and both need a lookup table, which sits badly with marked having no runtime dependencies.

Character references, current output on the left:

[foo](/f&ouml;&ouml; "f&ouml;&ouml;")
  <p><a href="/f&ouml;&ouml;" title="f&ouml;&ouml;">foo</a></p>
  <p><a href="/f%C3%B6%C3%B6" title="föö">foo</a></p>

[link](foo%20b&auml;)
  <p><a href="foo%20b&auml;">link</a></p>
  <p><a href="foo%20b%C3%A4">link</a></p>

Leaving the entity in place works for text content because the browser resolves it, but a destination has to be decoded first and then percent encoded, and getting there means the HTML5 named reference table.

Case folding is the same shape. Example 540 wants [ẞ] to match [SS], which needs full folding to ss. I tried the usual shortcuts and neither gets there:

'ẞ'.toLowerCase()                    // 'ß'
'SS'.toLowerCase()                   // 'ss'
'ẞ'.toUpperCase().toLowerCase()      // 'ß'

The question

Would you take a PR that adds a table for either of these, and if so is there a size budget worth knowing about? Or would you rather they stay as known gaps and keep the dependency-free footprint as it is?

Happy to do the work on whichever you would accept. If the answer is neither, that is a useful answer too, and it might be worth a note somewhere so the next person does not rediscover it.

For case folding there is already a PR at #3997.

For HTML entities I would rather see this as an extension so people can opt in to including a large table that doesn't change the output displayed in a browser.

I’ll take the independent “raw HTML and autolinks taking precedence over links” group (CommonMark examples 524, 526, 536, and 538). I’ll leave character references, Unicode folding, and nested-bracket handling to their existing work, add focused spec regressions, and keep the parser change limited to this precedence boundary. AI disclosure: OpenAI Codex will assist with implementation and validation; I will review and own the resulting contribution.

I will take the remaining independent GFM protocol-autolink group (GFM 0.29 examples 633–635: mailto:/xmpp: plus the optional XMPP resource). I will keep this separate from #4066 and leave the bare-email domain fix in #4063 untouched. AI disclosure: OpenAI Codex will assist with implementation and validation; I will review and own the resulting contribution.

I will take the remaining independent GFM tagfilter gap (GFM 0.29 example 657, Disallowed Raw HTML). I will keep it separate from #4066 and #4067 and limit the change to the nine tag names defined by the GFM extension. AI disclosure: OpenAI Codex will assist with implementation and validation; I will review and own the resulting contribution.

I'll take the nested-brackets-in-link-text group (512, 518, 519, 520, 528, 532, 533), since it's the one left over after the other claims here.

Keeping it clear of the rest: nothing to do with character references or case folding, separate from evgenyponomarev's precedence and autolink groups, and separate from my own #4053, which only touches autolink destinations.

Being upfront about the shape of it, since this group is the one nobody has picked up: CommonMark handles these with a delimiter stack that lets an inner link win and turns the outer brackets back into literal text, and the inline lexer here matches links with a regex instead. So this may not come out as a contained change. If it turns out to need more restructuring than you'd want for seven spec examples, I'll say so here rather than push a PR that trades a parser rewrite for them.

@UziTech

@UziTech on the entities one — you said you'd rather have it as an opt-in extension than a table in core. That sounds like a reasonable place to land, and nobody seems to be building it, so a couple of questions before anyone does:

Where would you want it to live? Something under markedjs alongside the other extensions, or a package outside the org that just gets pointed at from the docs?
Would it hook the tokenizer, so token.href comes out already decoded and percent-encoded, or the renderer, so only the emitted attribute changes? The tokenizer version fixes 32, 33 and 503 as the spec has them, but it also changes what other extensions see on the token.
Is a full HTML5 named-reference table what you had in mind, or would the subset that can appear in a destination be enough?

Happy to build whichever shape you'd take. If the answer is that it should stay a documented gap, that's fine too, and worth noting in this issue so the next person doesn't re-derive it.

Reporting back on the nested-brackets group rather than opening a PR, as promised. It needs the delimiter stack, and I don't think it's worth it for seven examples. Writing down what's underneath so the next person doesn't have to re-derive it.

There are two separate failures in the group.

Depth. _inlineLabel allows one level of nesting — \[(?:\[\s\S]|[^\[\]\])*\] matches a [...] with nothing bracketed inside it. So 512 and 528 ([link [foo [bar]]], two levels) never match as links and fall through to text. Adding a second level to the pattern would fix those two and leave three levels broken; CommonMark doesn't cap the depth, so there's no depth that finishes it.

Precedence. 518, 520, 532 and 533 want the inner link to win and the outer brackets to go literal:

[foo [bar](/uri)](/uri)  ->  [foo <a href="/uri">bar</a>](/uri)

Currently they produce nested anchors, <a href="/uri">foo <a href="/uri">bar</a></a>, which is invalid HTML regardless of what the spec asks for. That may be worth its own issue even if this group stays open.

The rule producing the expected output is that when a ] closes a link, every [ opener before it is deactivated, so the leftmost-innermost one closes first and the outer never gets a chance. That's inherently a left-to-right scan over a stack of openers. The regex here starts at the leftmost [ and takes the longest match, which is the opposite end, so it isn't a matter of tuning the pattern — state.inLink doesn't help either, since suppressing the inner link makes the outer one win, which is backwards.

So both halves land on the same change: matching links with a scan over [/] positions instead of inline.link and inline.reflink. That touches how every inline link is found, so it isn't a contained diff, and the payoff is seven spec examples that all involve brackets nested inside link text.

Unclaiming it. If you'd rather have it anyway, or want the nested-anchor output looked at on its own, say so and I'll pick it back up.

Thanks for writing this up here instead of dropping it quietly. Our numbers differ on the group though, so worth reconciling before anyone acts on it.

Four of the seven fail on main at e250e31, not six. 518, 519 and 532 already pass:

512 fail   518 pass   519 pass   520 fail   528 fail   532 pass   533 fail

#4064 has been open since yesterday for the depth half and turns 512, 520 and 528 green, with the spec suite at 1783 passing. You are right that a regex cannot follow arbitrary nesting, and the PR says the same thing; its claim is narrower, that two levels is where the suite stops asking, with a third and a fourth flipping no further example.

That leaves 533, and its failure is not precedence. The spec wants [foo bar <a href="/uri">baz</a>] and what comes out keeps the asterisks as literal text. #4059 fixes that one and has been approved since the 19th.

On the nested anchors, I could not get any. 518 and 532 come out as the spec asks, and the other two fail in unrelated ways: 520 gives ![[<a href="uri1">foo</a>](uri2)](uri3), and 533 is the asterisks above. So there may be nothing to file separately there.

My fault for the collision. I should have claimed the group here when I opened #4064 rather than leaving it to be found.

My take on the CommonMark spec is that it is more a spec for how CommonMark works rather than how markdown should work. The purpose of markdown should be to have a human readable text document with simple formatting that can convert to html for a web page. That means since a browser displays ö and ö the same it should not matter which is output by the markdown. Personally I think html entities are a relic of a past without unicode and displaying the unicode character is the way it should be. I also don't believe most people who use marked care if the spec is matched as long as the output (rendered html) is what they expect.

That being said, I understand the need for a spec and consistency between parsers. Which is why I believe any part that follows the spec but degrades the normal use of marked should be opt-in (i.e. an extension). Currently @markedjs only hosts extensions that work specifically like past options. Those extensions don't get fixes or updates for the way they should work since there are users who rely on how they used to work. I would like to see extensions be external from MarkedJS so there can be more choice and it is easier for better variants to win. If a user sees a @markedjs/extension they will typically think it is the best without even looking for some external extension that could be better. We have a Known Extensions section in our docs to highlight external extensions.

I would also like to point out that the MarkedJS team is very small, and we work on it in our free time, so spreading out the extensions helps us not get overloaded. We highly appreciate the the work you all are doing to help improve marked for everyone. Thanks! 💯

Thanks for grouping these by root cause. It made this easy to check against a second harness, and the comparison turned up something I think is worth knowing before anyone decides about the tables.

I ran the raw spec.json for 0.31.2 through the spec repository's own test/normalize.py against marked 18.0.11, gfm: false, and got 28 failures, not 15. That is not a disagreement about the corpus. Both harnesses read the same file. It is the comparison.

runTests and update-specs.js both go through htmlIsEqual, which constructs @markedjs/html-differ with only ignoreSelfClosingSlash: true and ignoreComments: false. So ignoreWhitespaces stays at its default of true, and the differ also decodes character references before comparing. Two classes of spec deviation therefore cannot set shouldFail, and cannot fail the spec suite either.

On 18.0.11:

comparison	fails
`htmlIsEqual`, what the suite uses	11
`normalize.py`, what the spec ships	28

The 17 in the gap:

class	count	examples
whitespace inside `<pre>`	10	2, 6, 126, 130, 133, 144, 237, 318, 636, 637
HTML5 named references left literal	7	25, 26, 27, 28, 30, 39, 40

Whitespace inside <pre> is significant by definition, and normalize.py is explicit about it: it collapses whitespace everywhere except inside <pre>, which is the one place the differ's global setting still applies. Three of them:

ex 2    want  <pre><code>foo\tbaz…       got  <pre><code>\tfoo\tbaz…
ex 126  want  <pre><code></code></pre>   got  <pre><code>\n</code></pre>
ex 318  want  <pre><code>b\n\n\n</code>  got  <pre><code>b\n\n</code>

The differ reports all three as equal. 318 is a missing newline in code block content rather than formatting.

The entity group is the one you already describe, but in text content rather than in link destinations: &Dcaron;, &HilbertSpace;, &DifferentialD;, &ClockwiseContourIntegral; and &ngE; come out literal. You note that leaving them in place works because the browser resolves them, which is true for display, and the spec still asks for them to be resolved. The differ decodes the actual side before comparing, so it agrees with that reading by construction rather than by decision.

One result goes the other way and it is good news. 518, 519, 532 and 533 pass on 18.0.11 under both comparisons. They sit in your nested brackets group and carry shouldFail in the committed fixture. Since update-specs.js only rewrites that flag when it is run, four of those seven look like stale flags from an older build rather than open gaps, which would make the real group 3 rather than 7.

On your actual question I have no standing, since I do not maintain this. But one thing may matter to the decision: the 10 whitespace cases need no lookup table at all. They are currently invisible rather than known, which is a different problem from the wall you describe, and probably a cheaper one.

Reproducing this takes about thirty seconds and needs nothing of mine:

curl -sSo spec.json https://spec.commonmark.org/0.31.2/spec.json
curl -sSo normalize.py https://raw.githubusercontent.com/commonmark/commonmark-spec/master/test/normalize.py
npm i marked@18.0.11
node -e '
const {marked}=require("marked"), fs=require("fs");
const spec=JSON.parse(fs.readFileSync("spec.json","utf8"));
fs.writeFileSync("out.json",JSON.stringify(spec.map(s=>({
  example:s.example, expected:s.html, got:marked.parse(s.markdown,{gfm:false})}))));
'
python3 -c '
import json
from normalize import normalize_html
rows=json.load(open("out.json"))
bad=[r for r in rows if normalize_html(r["got"])!=normalize_html(r["expected"])]
print(f"{len(rows)-len(bad)}/{len(rows)} pass, {len(bad)} fail")
print(", ".join(str(r["example"]) for r in bad))
'

markdown-it in its commonmark preset passes 652/652 under the same normalizer, so the 28 are not the normalizer being unreasonable.

Two numbers, two measurements: 15 is what marked's harness flags, 28 is what the spec's reference comparison finds. They should not be quoted interchangeably, and I would not have spotted the difference without your table to compare against.

For provenance: this came out of exit0.run/0021, a public registry I run where a benchmark number only counts once a stranger has reproduced it. Which is the reason the reproduction above does not go through it.

@exit0-run

Update, since this has aged out. I ran it on 18.0.9 and two releases have landed since.

On 18.0.11 the count is 11, not 15:

Root cause	Examples	Count
Character references in link destination and title	32, 33, 503	3
Nested brackets in link text	512, 520, 528	3
Raw HTML and autolinks taking precedence over links	524, 526, 536, 538	4
Unicode case folding of link labels	540	1

518, 519, 532 and 533 are fixed, and their flags are already gone, removed by #4051 and #4059. @exit0-run was right that those four pass on 18.0.11. The one thing to correct is that they no longer carry shouldFail there, so there is nothing stale left to clean up. My 15 was an 18.0.9 number, and the gap between the two counts is those two PRs rather than anything about the comparison.

The point about the differ is separate from the counting, and it is the useful one. htmlIsEqual builds html-differ with only ignoreSelfClosingSlash and ignoreComments, so ignoreWhitespaces stays at its default of true. Whitespace inside pre/code is significant, so mismatches there are invisible rather than known, which is a different situation from the groups above.

That turned up a real one. An empty fenced code block renders as <pre><code>\n</code></pre> instead of <pre><code></code></pre>, and examples 126, 130, 144 and 237 have been reporting as passing the whole time with no shouldFail flag against them. Fix in #4073. The spec suite cannot protect that fix for the same reason it never caught the bug, so the tests there assert exact output.

Example 318 is the same blind spot but a different bug. A fenced block in a list item expects b\n\n\n and gets b\n\n, and the code token is already short before the renderer sees it, so that one is in the lexer. Leaving it out of #4073 and noting it here.

The question in the original post still stands for the two groups that are left, character references (3) and case folding (1). Both still need a table.

Correction accepted: I was wrong to call those four stale flags. #4051 and #4059 removed them, so there is nothing left to clean up there, and the 15 to 11 difference is those two PRs rather than anything about the comparison. Thanks for pinning it down.

#4073 covers 126, 130, 144 and 237, and 318 is noted as a lexer bug. That leaves five of the ten whitespace cases untriaged, and they fall into three groups rather than five separate things.

Tab expansion, examples 2 and 6. Opposite symptoms, and they may be one bug seen from both sides.

ex 2   md    "  \tfoo\tbaz\t\tbim\n"
       want  <pre><code>foo\tbaz\t\tbim\n</code></pre>
       got   <pre><code>\tfoo\tbaz\t\tbim\n</code></pre>

ex 6   md    ">\t\tfoo\n"
       want  <blockquote><pre><code>  foo\n</code></pre></blockquote>
       got   <blockquote><pre><code>foo\n</code></pre></blockquote>

Two keeps a leading tab the indent should have consumed. Six drops two spaces the tab should have left behind.

Fence indent stripping, example 133. The fence opens with three spaces, so each content line should lose up to three and no more.

ex 133 md    "   ```\n   aaa\n    aaa\n  aaa\n   ```\n"
       want  <pre><code>aaa\n aaa\naaa\n</code></pre>
       got   <pre><code>aaa\n aaa\n  aaa\n</code></pre>

The four space line is handled. The two space line keeps both.

Hard break continuation, examples 636 and 637. One bug, two spellings.

ex 636 md    "foo  \n     bar\n"
ex 637 md    "foo\\\n     bar\n"
       want  <p>foo<br>bar</p>
       got   <p>foo<br> bar</p>

Leading whitespace on the line after a hard break should go. One space survives in both.

All five sit in the same blind spot as the ones you just fixed, so the same approach applies: the spec suite will report them as passing whatever the renderer does, and only a test asserting exact output holds the fix in place.

Happy to open a PR for the hard break pair if that is useful, since it looks self contained. The tab ones I would leave to somebody who knows that code, because 2 and 6 pulling in opposite directions suggests the fix is in the expansion itself rather than at either end.

Went through the remaining five. Two of them are not bugs, which seems worth saying before anyone spends time on them.

636 and 637 are fine as they are. I did implement the fix: the br rule is /^( {2,}|\)\n(?!\s*$)/, so the next line's leading spaces stay on the following text token. Adding * to the rule turns foo bar into foo bar. But measuring it over the whole spec gives zero change to either the differ set or the exact set. That whitespace is in paragraph text, where HTML collapses it and normalize.py treats the two as equal, so marked was already correct and the differ is right to ignore it here. I reverted it.

That is the line I would draw through the group: whitespace inside pre/code is significant and the differ genuinely cannot see it, which is where 126, 130, 144, 237 and 133 were hiding. Whitespace in ordinary text is not, and there the differ is doing the right thing.

133 is fixed in #4074. indentCodeCompensation only stripped lines indented at least as much as the fence, so a line indented less kept all of it.

2 and 6 are real, and bigger than they look. Both land inside pre/code, so they count. The cause is that tab expansion only happens in pedantic mode: Lexer.ts:113 replaces every tab with four spaces behind this.options.pedantic, and normal mode has no tab handling. Doing it properly means expanding to the next stop at a multiple of four for block structure only, preserving tabs in content, and allowing a tab to be consumed partially, which is what example 6 needs where the blockquote marker takes part of one and the rest becomes content indentation. That is column-aware block parsing rather than a local fix, so it wants a decision from the team first, same as 318.

Correction to something I wrote earlier and which has been repeated since: I called 318 a lexer bug and that was wrong. Details are in #4074, along with the measurement showing why the obvious fix for it makes things worse.

On 636 and 637 I get a different measurement, and I would rather show it than argue it.

ex 636  want '<p>foo<br>bar</p>'   got '<p>foo<br> bar</p>'   equal: False
ex 637  want '<p>foo<br>bar</p>'   got '<p>foo<br> bar</p>'   equal: False

That is normalize_html from commonmark-spec/test/normalize.py on both sides, marked 18.0.11, gfm off.

I think I can see where we diverged. You describe the fix as turning foo bar into foo bar, five spaces. By the time the normalizer has run, that run is already one space: it collapses runs of whitespace outside pre. What is left is a single space after  , and the normalizer keeps it, because collapsing a run is not the same as stripping whitespace that begins a text node after a tag. So on the exact set your fix should move these two from failing to passing rather than changing nothing.

Your semantic point is right and I am not disputing it. That space does not change what a browser renders, and marked's output is fine as HTML. The awkward part is that the spec's own reference comparison disagrees with the browser here, so normalize.py counts them while a reader would never notice. That is a gap between the two, not a defect in marked, and it is probably worth writing down rather than fixing.

Which makes your line through the group the right one, just drawn one step further along: whitespace inside pre/code is significant and the differ cannot see it, which is where 126, 130, 144, 237 and 133 were hiding. Whitespace in ordinary text is insignificant to a browser, and the differ is right to ignore it, but the spec normalizer still does not.

The rest I take as settled. Thank you for the 318 correction, and for saying it plainly; I had repeated your lexer framing without checking it.

@UziTech

@UziTech sorry for the slow reply, and thanks, that answers it. An external extension rather than one under @markedjs is not where I expected to land, but the point about the prefix reading as an endorsement makes sense, and so does not carrying a table in core for output a browser already renders right. I will leave the entity work alone, and case folding already has #3997.

The whitespace half of the gap is well covered above, so I will only add the other half, which nobody has taken apart yet. Of the 7 reference cases, only 25 is about HTML5 only names. 26, 27, 39 and 40 are numeric: #, &#X22;, 
 and 	 all come out literal. 28 and 30 are undefined names, where &MadeUpEntity; should come out as &MadeUpEntity; and marked leaves the ampersand bare.

Six of the seven need no table, then. It does not change the answer though. A browser renders all six the same either way, so they cost the reader what the named ones cost, which is nothing. The question I opened with gets the same no across the whole group, and that is a cleaner place to leave it than I expected.

Small correction to my own first post while I am here. The 15 was 18.0.9 and the fixture is at 11 now: 518, 519 and 532 were stale flags, and 533 went green when #4059 merged on the 24th.

Ran the two tab cases, since they are the ones from the whitespace group still without a PR. Three arms, marked 18.0.11, all 652 examples, compared with the spec's normalize.py.

arm	gfm off	gfm on
as shipped	28	32
every tab to four spaces, what the pedantic path does	31	35
tab stops of four, leading whitespace only	27	31

Turning the pedantic replacement on unconditionally does not fix 2 or 6, and costs three examples. It breaks 1 and 3 because a literal tab inside code content has to survive, and it breaks 7 on the indent width.

ex 1   md     "\tfoo\tbaz\t\tbim\n"
       want   <pre><code>foo\tbaz\t\tbim\n</code></pre>
       naive  <pre><code>foo    baz        bim\n</code></pre>

Expanding to the next stop of four across leading whitespace only is free. It fixes 2 and moves nothing else, and the set is identical in both runs.

ex 2   md     "  \tfoo\tbaz\t\tbim\n"
       want   <pre><code>foo\tbaz\t\tbim\n</code></pre>
       got    <pre><code>\tfoo\tbaz\t\tbim\n</code></pre>

6 does not come with it, and that is the part that matches your "bigger than they look". Its tab sits after the block quote marker rather than at the start of the line:

ex 6   md     ">\t\tfoo\n"
ex 7   md     "-\t\tfoo\n"

The stop has to be counted from the column after the marker is consumed, so no preprocessing pass reaches it. 2 is a preprocessing fix and 6 is not, which is a cleaner split than I expected before measuring.

Worth pinning 7 while anyone is in there. It passes today and its content is right, foo on both sides. Its only deviation is a newline between <li> and <pre>, which sits outside pre where the comparison collapses it. But it is the same shape as 6, a tab after a block marker, and the naive arm renders its content with three spaces instead of two. That makes it the natural regression check once tab handling moves into block parsing.

That split is cleaner than the one I had in my head, and the middle row is the part I
had wrong. I had lumped all the tab cases together as "needs block parsing" and left
them alone on that basis. Measuring the three arms separately is what pulls 2 out of
that group, and it only shows up because you ran the naive arm as a control rather
than assuming it.

The 31 in the naive arm is the useful number. Expanding every tab looks like the
obvious fix right up until you notice it has to destroy the tabs inside code content
to get there, and example 1 is the shortest possible demonstration of that.

Your point about 6 counting the stop from after the marker is consumed is what makes
it a different job rather than a harder version of the same one. A preprocessing pass
sees >\t\tfoo before anything knows > is a marker, so the column it would need is
not available yet.

I can take the preprocessing half if nobody else is on it. That is 2 only, leading
whitespace expanded to the next stop of four, with 7 as the regression check the way
you suggested, and 6 left for whenever tab handling moves into block parsing. Happy
to leave it to you instead if you would rather carry it through, since you did the
measuring.

One thing worth writing down somewhere for the next person: the reason this group sat
untouched is that marked's own htmlIsEqual normalises whitespace away, so every one
of these reports as passing in the suite. The differ cannot see the thing being
measured. That is why the counts here only make sense against normalize.py.

Took a run at the preprocessing half and it does not survive contact with marked's own
suite. Posting the negative result rather than the PR.

Your numbers reproduce exactly. Leading whitespace expanded to the next stop of four,
scored against normalize.py over all 652:

arm	gfm off	gfm on
as shipped, 18.0.11	28	32
leading whitespace only	27	31

2 drops out and the remaining sets are identical in both runs, which is what you said.

The problem is that a whole-source pass cannot tell which lines are inside a fenced
block, so it expands those too. test/specs/new/tabs_code fails:

in    ``````\ttab```
want  <pre><code>\ttab</code></pre>
got   <pre><code>    tab</code></pre>

Upstream keeps the tab, the patched build does not. Same root cause as example 1 in
your naive arm, just reached through a fence instead of an indent.

The uncomfortable part is that the 652 do not catch it. Both of us measured this arm as
moving nothing else, and it does move something else. Only marked's own fixture sees it,
which is the mirror image of the whitespace group, where the 652 see failures the suite
normalises away.

So 2 lands on the same side as 6 after all. It is not that the column is unavailable
before block parsing, it is that whether the line is code at all is unavailable. That
makes the split cleaner than the one I posted: none of the tab cases are preprocessing
work, and 7 stays the right regression check for whoever does it inside the block parser.

Happy to be wrong about this if you can see a preprocessing shape that skips fences.

Your result reproduces here, and the answer to the fence question is no. Worth showing both.

The regression. Running all five fixture groups through @markedjs/testutils, so the comparison is the suite's own, the leading whitespace arm regresses exactly one fixture, new/tabs_code, and nothing else. Reporting that as a delta rather than a count, because getTests hands back specs without the shouldFail flags, so my absolute numbers are not the suite's.

Why the 652 could not see it. I checked that from my side rather than assume I had read them carelessly: no example in CommonMark 0.31.2 contains a fence marker and a tab. Thirteen contain a tab at all, eleven of them in Tabs, plus 13 in Backslash escapes and 82 in Setext headings. The corpus has no case of that shape, so it was not there to be found.

A fence aware pass does clear both instruments, and is still wrong. Tracking an open fence with ^ {0,3}(`{3,}|~{3,}) and leaving every byte between it and its close alone:

arm	652, gfm off	652, gfm on	marked's fixtures
as shipped	28	32	baseline
leading whitespace only	27	31	regresses `new/tabs_code`
fence aware	27	31	regresses nothing

It fixes 2, moves nothing else in either run, and tabs_code passes. Two instruments agreeing is, after yesterday, exactly the point where I should keep going rather than stop. Writing \t for a literal tab:

in            "    \ttab\n"

upstream      <pre><code>\ttab
markdown-it   <pre><code>\ttab
fence aware   <pre><code>    tab

An indented code block, no fence anywhere, and the same destruction of a tab in code content, reached through an indent instead. Neither instrument contains it.

A fence opened on a list item line is the other one:

- ```
  \ttab
  ```

A list marker followed by backticks never matches an opener at the start of a line, so the pass expands the content. marked already disagrees with the reference there, so it does not surface as a regression, but the pass is reaching into code content again.

So your conclusion holds, and I think it gets sharper. The predicate a preprocessing pass would need is not "inside a fence", it is "is this line code content", and that is the container stack: fences, indented blocks, and both of those nested in lists and block quotes. No whole source pass has it. 2 belongs with 6, and 7 stays the regression check for whoever does it inside the block parser.

Take the block parser half if you want it. I have no claim on it, and you have the cause localised already.

@exit0-run

Took the block parser half. Splitting the report, because the two examples came out very differently.

2 is fixed, and it was not the hard problem

PR at #4080. It needed no column arithmetic at all. codeRemoveIndent is /^(?: {1,4}| {0,3}\t)/gm and alternation takes the first branch that matches, so ··\tfoo matched the two spaces and left the tab behind. Putting the tab branch first fixes it, and no space-only input changes because that branch cannot match without a tab.

That also explains the asymmetry you and I both noticed without explaining: example 1 (\tfoo…) passes because with no leading space the space branch matches nothing and the tab branch gets its turn. Same rule, opposite outcome, purely from branch order.

Whole spec through normalize.py, gfm off: 28 to 27, and example 2 is the only one that moves.

6 I implemented and am not sending

I want to show it, because the measurements say ship it and the measurements are wrong, which is the third time that has happened in this thread.

blockquoteSetextReplace2 is /^ {0,3}>[ \t]?/gm, so the marker eats one whole tab where the spec says it consumes one column and leaves the rest. marked already has a column aware expandTabs(line, indent) in helpers.ts, used by the list tokenizer, so the change is small: expand the whitespace run after the marker before stripping it.

It works. Example 6 passes exactly, and:

	spec fails
18.0.11	28
with 2 only	27
with 2 and 6	26

Full spec suite green, full unit suite green. Nothing in 652 regresses.

It is still wrong. Six hand written blockquote inputs against the reference parser, before and after:

input	reference	18.0.11	with the change
`>\t\tfoo`	`<pre><code>··foo`	`<pre><code>foo` ✗	`<pre><code>··foo` ✓
`>\t\tfoo\tbar`	`<pre><code>··foo\tbar`	`<pre><code>foo\tbar` ✗	`<pre><code>··foo\tbar` ✓
`>\tfoo`	`<p>foo</p>`	`<p>foo</p>` ✓	`<p>··foo</p>` ✗
`> >\t\tfoo`	`<pre><code>foo`	`<pre><code>foo` ✓	`<pre><code>··foo` ✗
fence in a blockquote, tab indented content	`<pre><code>\tfoo`	`<pre><code>\tfoo` ✓	`<pre><code>··foo` ✗

Two fixed, three broken, and the spec corpus contains none of the three. Same shape as the tab and fence case: the inputs that would catch it are not in the file.

The fence one is the one that decides it. > ```` / > \tfoo/> ```` has a tab that is code content, and expanding the run after the marker rewrites it to spaces. That is your container stack point arriving exactly where you said it would, one layer lower than a preprocessing pass.

The other two come from a different cause, and it is structural. After the marker is stripped, the remaining text is handed to blockTokens and re-parsed from column 0, but its real origin is column 2. Any tab expansion at that layer is computed against the wrong origin, so getting example 6 right means over expanding everything else by the offset. The fix has to be a column offset carried into the nested parse, not a rewrite of the line.

One extra bug, free

> \tfoo is a blockquote, one space, then a tab that reaches column 4. Two columns of content indent, so it is a paragraph. marked makes it an indented code block:

want  <blockquote><p>foo</p></blockquote>
got   <blockquote><pre><code>foo\n</code></pre></blockquote>

Present on 18.0.11, unrelated to my change, and it is not in the corpus either. Noting it rather than fixing it, since it is the same missing offset.

Unclaiming 6. The offset is a real piece of work and somebody who knows the blockquote and list nesting should decide whether it is wanted before it gets written. @exit0-run, thanks for handing it over, and the reproduction above is all yours if you want it.

@UziTech

Ran your five blockquote inputs against commonmark 0.31.2, the reference the spec ships, on 18.0.11 built from df57534. Both findings hold: >\t\tfoo gives foo where the reference gives ··foo, and > \tfoo gives an indented code block where the reference gives a paragraph.

The fence case turned out to be worth more than a prediction, so I went looking for it in the list container instead of the blockquote.

- ```
  \tfoo
  ```

reference   <pre><code>\tfoo
marked      <pre><code>····foo

marked already ships the bug you argued would be introduced. Not in the blockquote, in the list, on main, today.

The mechanism is Tokenizer.ts:338:

nextLineWithoutTabs = nextLine.replace(this.rules.other.tabCharGlobal, '    ');

tabCharGlobal is /\t/g, so every tab on a continuation line becomes four spaces with no reference to the column it sits at, and line 372 puts that string into itemContents. item.text is what blockTokens re-parses at line 440, so the tab is already gone before the fence tokenizer sees the line.

What makes this useful for the decision is the contrast inside the same tokenizer. Line 296 is the column-aware call:

let line = expandTabs(cap[2].split('\n', 1)[0], cap[1].length);

expandTabs takes the starting column, and this is its only call site in the codebase. It runs on the item's first line with the marker width, and nothing after that line gets the same treatment. So the list already carries both approaches side by side, and the split falls exactly where your table falls. Every indent shape that broke your blockquote patch comes out right in the list:

input	reference	marked
`-\t\tfoo`	`<pre><code>··foo`	matches
`-\tfoo`	`<li>foo</li>`	matches
`- \tfoo`	`<li>foo</li>`	matches
`- -\t\tfoo`	`<pre><code>foo`	matches
`1.\t\tfoo`	`<pre><code>·foo`	matches

Only the fence one diverges.

So I read your conclusion as strengthened rather than weakened. The line rewrite is enough for the indent shapes, which is why the list gets those right, and it is not enough for content tabs, which is why the list gets that one wrong. Neither container carries an offset into the nested parse: line 205 is blockTokens(currentText, tokens, true) and line 440 is blockTokens(item.text, []), both starting from column zero.

That also means there is a list-side bug to file whatever happens to example 6, and it is not in the corpus either, same as the three you found. I have not opened one, because it belongs next to whatever gets decided about the offset rather than on its own. Say the word if you would rather it stand alone and I will write it up.

@UziTech, on your earlier point about output over spec conformance, the fence case is the one that survives it. A tab inside a fenced code block is content, so rewriting it changes what the browser renders, not just what a spec comparison reports.

Navigation Menu

Repository navigation

CommonMark spec gaps grouped by root cause, and a question about lookup tables #4050

The wall

The question

2 is fixed, and it was not the hard problem

6 I implemented and am not sending

One extra bug, free

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CommonMark spec gaps grouped by root cause, and a question about lookup tables #4050

Description

The wall

The question

Activity

UziTech commented on Aug 11, 2026

evgenyponomarev commented on Aug 21, 2026

evgenyponomarev commented on Aug 21, 2026

evgenyponomarev commented on Aug 21, 2026

Kjubikstronk commented on Aug 21, 2026

Kjubikstronk commented on Aug 21, 2026

Kjubikstronk commented on Aug 21, 2026

luantaraschi commented on Aug 21, 2026

UziTech commented on Aug 22, 2026

exit0-run commented on Aug 29, 2026

Kjubikstronk commented on Aug 30, 2026

exit0-run commented on Aug 30, 2026

Kjubikstronk commented on Aug 30, 2026

exit0-run commented on Aug 30, 2026

luantaraschi commented on Aug 30, 2026

exit0-run commented on Aug 31, 2026

Kjubikstronk commented on Aug 31, 2026

Kjubikstronk commented on Aug 31, 2026

exit0-run commented on Aug 31, 2026

Kjubikstronk commented on Sep 1, 2026

2 is fixed, and it was not the hard problem

6 I implemented and am not sending

One extra bug, free

luantaraschi commented on Sep 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions