Description
Describe the feature
Custom extensions always take priority over default tokens. A way to specify where in the parsing sequence an extension should run so certain default tokens don't get overridden.
Why is this feature necessary?
Sometimes there are ambiguous cases in Markdown where the result depends on which token gets parsed first. Example, tables without a starting pipe:
<div> header
|:---:|
cell
This is not be parsed as a table, because HTML is parsed first. However if someone writes a custom table extension for example, suddenly tables take priority and this is rendered as a table. The user would want the extension to execute before the default table tokenizer but after the HTML tokenizer.
Describe alternatives you've considered
It is possible to work around this by, in the extension tokenizer, pre-parsing the string and checking if any higher-priority tokens appear. However this adds extra steps and results in the same line potentially being parsed multiple times.
I'm picturing something like breaking the lexer.blockTokens() and lexer.inlineTokens() functions into an array of tokenizers and calling each one in sequence, and a user making an extension would somehow be able to inject their extension into this array (added to the front by default), though I imagine calling each tokenizer from an array instead would cause some slowdown.
In fact I think that was the original approach to the Extensions feature when I was making it, some kind of list of tokenizers, but the slowdown was too much. But now that it's been out for a while I am starting to run up against this limitation more and more.