Idea: Make it easy to plug-in Prism or VS Code/TextMate grammars.

[joshgoebel]

Brilliant idea or terrible idea? Would this enhance the ecosystem or destroy it?

[foo123]

Check out this addon https://github.com/foo123/highlightjs-grammar which works for all syntax highlighters and enables you to define a grammar specification for a language and use it as is in every highlighter of your choice (including highlightjs and prism)

[joshgoebel]

and enables you to define a grammar specification for a language

I think that is the hard part unless you plan to publish and maintain a huge repository of grammars yourself. This is why (most) people come to a library I think, for the supported languages. :-) For most people saying "you just have to write your own grammar" is almost as bad as "you just have to write your own parser". :-)

What we'd gain (if say we supported Prism grammars) is instant support for all the languages that Prism supports that we do not. The question is might that hurt us by causing fewer people to want to developer Highlight.js grammars in the first place...

[foo123]

@yyyc514 of course you have a point in that grammars need to be written themselves, else it would be good if a repository of grammars exists. I dont mind having such a repository (maybe in the future) and the repos themselves have already made grammars for some popular languages (eg php, python, css, javascript, html, etc.) as examples on how to create fully-functional grammars.

But again the point of the grammar addons started through my own use cases where I had to highlight custom languages or dsl's that were not supported by any known language. And it would be a pain in the neck if I had to create parsers all the time, especialy if the parser was tied to a specific highlighter and had to make a new one for another highlighter. So highlighting custom languages was the starting point of the grammar addons.

Of course creating a grammar requires a minimal knowledge (eg BNF specifications) but again it is very minimal knowledge than writing a full parser and having to delve into the internals of a syntax highlight framework. So I think it is a win-win situtation.

Again all these are optional, only for those who benefit from such an approach, the others can use what exists already as before. This is only an addon, it is not necesarily a replacer of existing frameworks or languages.

[joshgoebel]

BNF spec... how deep does that get you? Are you really just lexing mostly or is it contextual enough to know you're inside a class, that this is a function call, not a variable definition (they can look the same in C, etc), etc?

I'd be curious to see a BNF for JSX or something complicated and see how well it worked compared to say what we currently have. Probably things we could learn.

[foo123]

The parsers created by the grammar addons are similar to PEGs which are mildly stronger than context-free parsers. But there are other features (like grammar actions) which enable strong context-sensitive parsing (not simply context-free).

Take a look at the xml grammar which not only highlights xml properly but detects errors, duplicate attributes, duplicate ids and so on..

The grammar reference is here

It is highly versatile, and eases the burden a lot in creating highly functional and feature-free parsers with only minimal knowledge (and in fact knowledge about a language that is needed anyway, eg what is a keyword, an identifier, a comment for that language and so on..).

Of course someone may need a feature not supported by grammar specification or even not possible at all by a grammar specification (and in that case would need to write his own highly detailed parser), but I doubt one will be found.

[joshgoebel]

Take a look at the xml grammar which not only highlights xml properly but detects errors, duplicate attributes, duplicate ids and so on..

Yeah, but that's the easy stuff (in my book). We could support the same if we just had callbacks inside a mode that let you do things like save a little state and dynamically apply the rule or skip it, etc...

I'm talking about harder stuff like:

#1987

[foo123]

If it can be modelled in a PEG grammar it can be done by the addon.

BNF spec... how deep does that get you? Are you really just lexing mostly or is it contextual enough to know you're inside a class, that this is a function call, not a variable definition (they can look the same in C, etc), etc?

Yes you can know if you are inside a class or function definition and so on, through grammar actions which create contexts. See for example the scoped grammar (further down the page) at this repo

[joshgoebel]

@foo123 Have you written a tokenizer/parser yourself for your grammar files? Or is all you have a bunch of conversation scripts to convert them into various grammar files recognized by other engines?

[foo123]

@yyyc514 My grammar addons have a full-blown PEG-like parser of their own.
In fact they all have the same generic parser and only the interface code changes to adapt to a specific editor/highlighter

[joshgoebel]

My grammar addons have a full-blown PEG-like parser of their own.

If that's true then you don't really need 99% of HLJS. You could use merely the new TokenTree plus HTMLRender... or perhaps only a slightly modified HTMLRender... that's only 42 lines of code.

First turning your PEG grammars into valid HLJS modes seems like a lot of hard work with very little upside... why not just use a plugin instead that replaces our parser with your own? You abort the actual parsing with a before callback and then in your after callback you'd have access to the emitter and could use it's simple API paired with your parsers output stream.

I'm not 100% sure all the pieces are in place to support that, but I'd love to see it tried. I'm very curious to see what a 3rd party plugin/grammar would look like that wasn't actually native HLJS based.