(Julia) Improved highlighter

[fredrikekre]

Hi, I have improved (IMO) the Julia highlighter to detect more patterns than what the current implementation does. I presented all of the changes here: https://fredrikekre.se/posts/highlight-julia/ and the code can be found on this branch: https://github.com/fredrikekre/highlight.js/tree/fe/julia-unleashed.

I would like to upstream this work to this repository but I am unsure if it would be accepted wholesale and wanted to discuss it here first. Some of the changes are uncontroversial I believe, some changes are pure bugfixes and some changes might be too "bold" to upstream. I can obviously separate things into separate pull request too.

The first thing I wanted input on is what I called contextual highlighting of types. The section in the post discusses some pros and cons of this, but I quite like it and this is the behavior in many text editors I have used (Sublime Text, VSCode). Would this be something that could be accepted upstream?

The second thing thing is function calls which with my changes are highlighted no matter where they are defined (e.g. in the base language or external package). I think this is nice (obviously) and I think it makes sense in Julia, since there usually is not so important to differentiate builtin functions from functions defined in packages or user code. This is also the reason that quite a lot of functionality in Julia is implemented in external packages, since user-code is as optimized as builtin code. Due to this I don't think keeping a list of builtin function is very useful, the list would be pretty short and not detect much of the code you write anyway. As with contextual highlighting of types, this is what both Sublime Text and VSCode does for Julia code.

I am of course happy to hear your thoughts about the other changes too, but, as I said, I believe they are mostly uncontroversial.

Thanks.

[joshgoebel]

We're open to a lot of this I think. I read the blog post (but didn't look too carefully at the code yet). Much of it aligns with the higher fidelity initiative and such. There are some technical issues (you can't use look behind yet - it's not supported by all major browsers). That might make a lot of what you're trying to do much harder, we'll have to see. There are some tricks I can share for some things. And you've taken some liberties with your usage of CSS classes (partly cause your a bit ahead of our theme support) but we can sort that out.

Please open a PR so I have the GitHub annotation tools available to me and then we can taker a closer look at what it possible and what isn't.

[joshgoebel]

The second thing thing is function calls which with my changes are highlighted no matter where they are defined (e.g. in the base language or external package).

We haven't really done this in the past. If we highlight function dispatch at all it's because the function is a built-in... we do want to highlight dispatch in general in the future (for languages where it's easily doable)... so then it becomes a question of whether there is value in built-ins having a different coloring (as you're asking)... this might end up being something we just let the themes decide... but I think that's something that will play out real-time as we're figuring it out. :)

[joshgoebel]

This large of a PR (it's almost a grammar reboot) will probably be quite a slog though (and can be overwhelming)... (see the ridiculously long threads on the LaTex and Mathematica reboots)... is there any chance of breaking it down into say the ~5 big chunks and then we could work on them as smaller pieces to get them merged faster? You could perhaps start with a bulleted list here of what the ~5 pieces might be (what would be easy to break out and submit piecemeal)... and then we could decide which one to start with?

[fredrikekre]

Thanks for the input, I will try to break it up into smaller pieces.

There are some technical issues (you can't use look behind yet - it's not supported by all major browsers).

Ah, that's too bad, IIRC I used that quite a bit, but I am sure it can be rewritten not to use it...

We haven't really done this in the past. If we highlight function dispatch at all it's because the function is a built-in... we do want to highlight dispatch in general in the future (for languages where it's easily doable)... so then it becomes a question of whether there is value in built-ins having a different coloring (as you're asking)... this might end up being something we just let the themes decide... but I think that's something that will play out real-time as we're figuring it out. :)

Alright, so that would mean keeping a list of built_ins and tag other function calls with e.g. hljs-call or something like that? I can understand why one would keep such a list for built_ins, but it seems like it would be pretty difficult to keep it up-to-date given that new functions are added removed from languages at times (at least more often than, say, function call syntax changes). And in the case of Julia, just because a function is called sum it does not have to mean it is the built-in sum function. I am guessing this is true in other languages too with similar namespacing capabilities. Keeping a list can also be "wrong" sometimes if you have e.g. sum = 2 in the code.

[joshgoebel]

but it seems like it would be pretty difficult to keep it up-to-date

We've made do just fine for years. :) When someone notices an issue, we fix it. :) And keywords (and builtins) are a LARGE part of how auto-detection works. So as long as we have auto-detect they aren't likely to go anywhere - regardless of how we decide to color them. In fact, ceasing to color them differently could actually make it less likely that grammars would get updated with correct keywords over time... So that's a factor. I'd probably still side in favor of whatever the "best" highlighting seems to be, but still it's a thought.

just because a function is called sum it does not have to mean it is the built-in sum function.

Perhaps this is more of a Julia only argument but I don't find it persuasive at all. This applies to many languages. I can redefine setTimeout in JavaScript also but 99% of the time it's a built-in.

Keeping a list can also be "wrong" sometimes if you have e.g. sum = 2 in the code.

In many languages it's clear this isn't a function dispatch, so it wouldn't be highlighted as a built-in. sum = vs sum(), etc.

---

My mind is far from made up here yet. Just giving you some food for thought. At a grammar level we (as a cohesive library) often prefer uniformity in output and are a hesitant to hear "my language is truly different than all your other languages and should be an exception to rules every other language follows". :-) Lets take this a step at a time and see how it goes. I want to play around with it (with Julia and other languages). You're getting in on the ground floor. These are things we haven't done in the past, so there are still lots of unanswered questions. #2500 #2521

Other engines focus more on "what it is" vs "what it does" (GitHub does this):

function add(a,b) {...}
add(2,3)

IE, here add would be a function-name... so it doesn't matter if we're defining it or calling it. If we went that way we'd likely end up with title-function or similar. There is also a chance that we don't get to decide, that this a theme concern. You could easily have dispatch and dispatch.built_in and the theme decides whether there is value in highlighting the built-ins differently than the user defined. I see a lot of merit in that approach. We tend to dislike grammar authors "playing God" with highlighting vs simply telling us what things are and then letting users/themes determine what the final output actually is.

TextMate grammar of course knows that it's both a entity.name.function and meta.definition.function or meta.function-call... so separate namespaces for "what it is" vs" what it does"... that's probably a bit out of scope for us, but it does have it's advantages.

[joshgoebel]

Ah, that's too bad, IIRC I used that quite a bit, but I am sure it can be rewritten not to use it...

There are several established patterns, and there are things like this coming along: #2834

The problem in the past has been many things are possible, just very ugly... and hence frowned upon (or impossible to read)... :-)

[fredrikekre]

Keeping a list can also be "wrong" sometimes if you have e.g. sum = 2 in the code.

In many languages it's clear this isn't a function dispatch, so it wouldn't be highlighted as a built-in. sum = vs sum(), etc.

Ah, but that is not how it works right now, is it? Maybe I misremember. Tagging as built-in if the name is also a function call seems like a good approach then, and the theme can decide to show call+built-in differently from call.

At a grammar level we (as a cohesive library) often prefer uniformity in output and are a hesitant to hear "my language is truly different than all your other languages and should be an exception to rules every other language follows". :-) Lets take this a step at a time and see how it goes. I want to play around with it (with Julia and other languages). You're getting in on the ground floor. These are things we haven't done in the past, so there are still lots of unanswered questions. #2500 #2521

I totally understand this, and as I said from the beginning, I understand that many of the changes might be too "bold" for this project.

[joshgoebel]

Ah, but that is not how it works right now, is it? Maybe I misremember.

Depends on the grammar, some have this "improvement", some do not. Typically it comes in the form of detecting dispatches and swallowing them (without assigning a class)... this prevents them from being highlighted as keywords. Closest example handy:

      // hack: prevents detection of keywords in some circumstances
      // .keyword()
      // $keyword = x
      {
        variants: [
          { begin: '\\.' + IDENT_RE },
          { begin: '\\$' + IDENT_RE }
        ],
        relevance: 0
      },

[joshgoebel]

Of course once we start highlighting function dispatch this problem should solve itself.

[joshgoebel]

@fredrikekre Happy to work on this with you whenever you have time but it doesn't require holding an open issue so I'm going to close this for now. Hope you are well and enjoying the holidays.

[weiyang2048]

Hi all,

If I want to change the style of say, Julia comments, with CSS, I wonder what is the correct way to do it.

For python and javascript, the following works for me, but it doesn't change the style for Julia comments.

.hljs-comment, .hljs-quote { color: green !important; font-style: italic !important; }

[joshgoebel]

Is there some weird difference in the generated HTML? Not sure why Julia should be different than any other grammar regarding CSS.