FR: syntax inclusions when using the LaTeX package 'robust-externalize'

[kiryph]

Is your feature request related to a problem? Please describe it.

I would like to have syntax highlighting for parts in a different language than LaTeX when using the LaTeX package "robust-externalize" (ctan, github).

Example file: https://github.com/jlelong/vscode-latex-basics/blob/main/test/colorize-fixtures/robust-externalize.tex

The test file misses \usepackage{robust-externalize} which is probably important for vimtex. Hence I added it manually:

VimTeX misses highlighting for c, bash and python.

Describe the solution you'd like
A similar solution to the way support was added in LaTeX Workshop in its Textmate LaTeX grammar:

jlelong/vscode-latex-basics#79

which renders the file in Visual Studio Code as following:

To recap the most important things:

1. 'robust-externalize' allows user to define custom presets for any language. This means the inclusion mechanism must work similar to fenced code in markdown files by matching the specified language.
2. The environments are CacheMeCode, PlaceholderPathFromCode, PlaceholderFromCode, SetPlaceholderCode.
3. The naming scheme of presets or placeholders should follow this
   • <lang> (example python)
   • <lang> <variant> (example python print code and result)
   • <version> <lang> <variant> (example my python matplotlib)
   • __<LANG>_<STRING>__
4. The names occur as the first argument in the curly braces after the opening of an environment.

Describe alternatives you've considered

Switching editor or using tree-sitter injections. Both are not viable. The latter does not work with vimtex.

[lervag]

I think we can add some basic support here, but this package seems extremely hard to support in a robust (pun intended) fashion. For instance, it seems hard to determine the nested code "filetype", especially with the Placeholder variants. And from the package docs I see that there are a lot of additional commands and environments.

[lervag]

I've added some very minimal and subpar implementation now. It's not good enough, but it is a start. I'm not quite sure what we should deem a "good enough" solution here, but I believe it may need an option for user configuration to e.g. specify custom presets.

Also, the SetPlaceholderCode environments and similar seem to be very hard to properly detect a syntax for. The name __PYTHON_TEMP is not something we can use to detect the syntax, since it is something you could change to whatever you wanted.

[kiryph]

Thanks.

However, it does NOT work yet.

UPDATE I missed the word NOT. Sorry for the confusion.

I've pulled in your recent commit:

❯ cd ~/.local/share/nvim/lazy/vimtex
❯ git log -n 1
commit fdb5618e05964e1cfcfd3bc1f2463ef3148ae8ff (HEAD, origin/master, origin/HEAD)
Author: Karl Yngve Lervåg <karl.yngve+git@gmail.com>
Date:   Wed Jul 10 18:10:46 2024 +0200

    feat(syntax): add wip support for robust-externalize

    refer: #2977

I am not sure what is wrong. I am using Neovim 0.10 and have disabled tree-sitter for latex files.

Feedback to the commit fdb5618

There is also an official preset for gnuplot which should be supported out of the box and added to

  for [l:preset, l:target] in [
        \ ['c', 'c'],
        \ ['bash', 'bash'],
        \ ['python', 'python'],
        \ ['my python', 'python'],
        \]

And then there is the question:

How should a user add custom presets?

I have created following presets

1. gap (www.gap-system.org) for including generated LaTeX math from GAP package typeset. Details here How to add GAP as a custom language when generating math output? leo-colisson/robust-externalize#7
2. gap chroma using chromacode for syntax highlighting of a gap code listing. Details here: Tree-sitter based syntax highlighting using the cli tool chromacode leo-colisson/robust-externalize#48

[lervag]

However, it does NOT work yet.

Hmm; on my end, with the minimal example I added, it does work for the things that I intended. Here's a screenshot:

But yes, it is not good enough yet.

I am not sure what is wrong. I am using Neovim 0.10 and have disabled tree-sitter for latex files.

Neither am I. On my end, it works both with minimal config and with my personal config.

There is also an official preset for gnuplot …

What is the name of that preset?

And then there is the question:

How should a user add custom presets?

Yes, that's one of the main important questions. I believe we should add something like this:

let g:vimtex_syntax_packages = {
      \ 'robust_externalize': {
      \   'custom_presets': [
      \     ['my python', 'python'],
      \     ['foo', 'bash'],
      \     ['some tikz preset', '@texClusterTikz'],
      \     ['a latex math preset', '@texClusterMath'],
      \   ]
      \ },
      \}

The idea is that the presets can be configured as a pair of "preset name" to "syntax". The "syntax" should either be something like bash or python, which refer to the nested filetype, or @cluster, which refers to an existing cluster like the @texClusterTikz.

For the latter to work, i.e. tikz, I think it may be useful or perhaps even necessary to load the tikz package explicitly when we use the robust externalize package.

Also, I think we should support some of the common presets directly. Both the "trivial" ones like "c", "bash", "python", etc., and the builtin presets that you mentioned. But for this, it would be nice to get a suggestion for what these are/should be.

[kiryph]

However, it does NOT work yet.

Now, it works:

I am not sure what was wrong. Maybe I did not really restart nvim. On a fresh morning, things can be different...

There is also an official preset for gnuplot …

What is the name of that preset?

It is simply called gnuplot and exists since version 2.3.

The idea is that the presets can be configured as a pair of "preset name" to "syntax".

I think this idea is good to allow the full degree of freedom for the preset names as robust-externalize.

Also, I think we should support some of the common presets directly. Both the "trivial" ones like "c", "bash", "python", etc., and the builtin presets that you mentioned. But for this, it would be nice to get a suggestion for what these are/should be.

I would say the trivial ones are when the preset name matches the name of the vim filetype.

So in your current list

  for [l:preset, l:target] in [
        \ ['c', 'c'],
        \ ['bash', 'bash'],
        \ ['python', 'python'],
        \ ['my python', 'python'],
        \]

only the last one would be needed.

About builtin presets: There are the trivial (match vim filetype) ones

• python,
• bash,
• latex,
• gnuplot.

The other builtins are

• tikz
• verbatim text

Additionally, I would find it useful to support preset names <vim filetype><space><variant> with a simple regex \S+ to avoid in many circumstances to configure anything.

Examples are

• python matplotlib (example given in the official documentation)
• python print code and result (example given in the official documentation)
• gap chroma (which would become for other languages python chroma)

I could imagine myself to rename python matplotlib simply to matplotlib and would use your suggested customization

let g:vimtex_syntax_packages = {
      \ 'robust_externalize': {
      \   'custom_presets': [
      \     ['matplotlib', 'python'],
      \     ['sage', 'python'],
      \     ['sympy', 'python'],
      \     ['some tikz preset', '@texClusterTikz'],
      \     ['a latex math preset', '@texClusterMath'],
      \   ]
      \ },
      \}

[kiryph]

And here the usual 50ct regarding tree-sitter 😄:

A long term goal for VimTeX could be to add dedicated support for Neovim by using syntax highlighting and syntax-based functionality through tree-sitter. Afaik, vim-matchup did this.

I have a few languages where better/maintained tree-sitter parsers than vim regex based syntax files exist. Also contributing to tree-sitter appears more worthwhile because a larger user base profits compared to improving vim syntax files.

This becomes apparent in this context where one injects/includes different kind of languages which means more syntax files are involved and increases the workload to have maintained syntax definitions for more languages.

[lervag]

Sorry for the delay - vacation time and stuff.

However, it does NOT work yet.

Now, it works: …

Great, glad to hear it!

There is also an official preset for gnuplot …

What is the name of that preset?

It is simply called gnuplot and exists since version 2.3.

Ok, cool - I'll add that one to the default list.

Also, I think we should support some of the common presets directly. Both the "trivial" ones like "c", "bash", "python", etc., and the builtin presets that you mentioned. But for this, it would be nice to get a suggestion for what these are/should be.

I would say the trivial ones are when the preset name matches the name of the vim filetype.

So in your current list

  for [l:preset, l:target] in [
        \ ['c', 'c'],
        \ ['bash', 'bash'],
        \ ['python', 'python'],
        \ ['my python', 'python'],
        \]

only the last one would be needed.

Ah, well, that's actually harder than it seems. When creating the rules, we sort of need to know in advance what the target or preset is. Thus, it must either be parsed up front or predefined. I'm doing the latter right now, because it is easy. For the minted package, I've parsed the document for these types of pairs, but it is not so trivial.

So, if it is possible to make a predefined list that would work in 95 % of cases, then I would prefer to do that. And then allow to change the list to make it possible to solve the remaining 5 %.

The idea is that the presets can be configured as a pair of "preset name" to "syntax".

I think this idea is good to allow the full degree of freedom for the preset names as robust-externalize.

About builtin presets: There are the trivial (match vim filetype) ones …

Ok - does this mean c is not part of the builtin presets?

I guess I should also add support for latex, tikz, and verbatim text in some way. It would have to be implemented in a slightly different way.

Additionally, I would find it useful to support preset names <vim filetype><space><variant> with a simple regex \S+ to avoid in many circumstances to configure anything.

Hmm, I think this should already work. The regex I use currently looks like this:

  \ 'start="\\begin…{' . l:preset . '[ ,}]"'

I.e., the preset "python" → target "python" would already match the python matplotlib.

And here the usual 50ct regarding tree-sitter 😄:

A long term goal for VimTeX could be to add dedicated support for Neovim by using syntax highlighting and syntax-based functionality through tree-sitter. Afaik, vim-matchup did this.

To be honest, I don't really think that VimTeX itself should provide any treesitter highlighting. I've noticed that there is a lot of progress on the latex parser. I would not be sad to see that the latex parser becomes so good that I can start to consider to deprecate the VimTeX syntax highlighting.

I have a few languages where better/maintained tree-sitter parsers than vim regex based syntax files exist. Also contributing to tree-sitter appears more worthwhile because a larger user base profits compared to improving vim syntax files.

Yes, I agree with the sentiment. However, I don't know how to do the treesitter stuff yet, and time is limited.

This becomes apparent in this context where one injects/includes different kind of languages which means more syntax files are involved and increases the workload to have maintained syntax definitions for more languages.

Yes, also agreed. I definitely think that treesitter would make life better also for VimTeX. And I would not mind this future! But for now, I'll still maintain and also accept some continued evolution of the current syntax engine.

[lervag]

To summarize, the current tasks are:

• Add configuration option.
• Support the other builtin latex, tikz and verbatim presets.

Let me know if you think I missed something.

[lervag]

Ok, I've improved things now, please test. It should be possible to customize as explained in the updated docs.

I would appreciate that additional improvements follow in new feature requests, preferably with a simple example that I can add to the test file. This just to make it easier for me, so I don't have to read so much text to realize the context if I can't follow up immediately.

[kiryph]

Hi Lervag,

thanks for all the updates and feedback.

So, if it is possible to make a predefined list that would work in 95 % of cases, then I would prefer to do that. And then allow to change the list to make it possible to solve the remaining 5 %.

This sounds good to me.

Ok - does this mean c is not part of the builtin presets?

Yes.

Ok, I've improved things now, please test.

I am on it.

I would appreciate that additional improvements follow in new feature requests, preferably with a simple example that I can add to the test file. This just to make it easier for me, so I don't have to read so much text to realize the context if I can't follow up immediately.

I will do this.

[kiryph]

Feedback of my first tests:

Customization in Neovim with package manager Lazy.nvim in Lua works:

require('lazy').setup({

  {
    'lervag/vimtex',
    config = function()
      vim.g.vimtex_view_method = 'sioyek'
      vim.g.vimtex_syntax_packages = {
        robust_externalize = {
          presets = {
            {'bash', 'bash'},
            {'python', 'python'},
            {'tikz', '@texClusterTikz'},
            {'latex', 'TOP'},
            {'gap', 'gap'},
          }
        }
      }
    end,
  },

I noticed here that I had to repeat all defaults. If this bothers me, I will raise a separate issue.

Regarding the syntax highlighting, I miss tree-sitter syntax highlighting for the included languages in Neovim. But this has already been discussed and answered.

[lervag]

Feedback of my first tests:

Customization in Neovim with package manager Lazy.nvim in Lua works:

require('lazy').setup({

  {
    'lervag/vimtex',
    config = function()
      vim.g.vimtex_view_method = 'sioyek'
      vim.g.vimtex_syntax_packages = {
        robust_externalize = {
          presets = {
            {'bash', 'bash'},
            {'python', 'python'},
            {'tikz', '@texClusterTikz'},
            {'latex', 'TOP'},
            {'gap', 'gap'},
          }
        }
      }
    end,
  },

I noticed here that I had to repeat all defaults. If this bothers me, I will raise a separate issue.

Regarding the syntax highlighting, I miss tree-sitter syntax highlighting for the included languages in Neovim. But this has already been discussed and> Customization in Neovim with package manager Lazy.nvim in Lua works: …

I also use lazy - it's good! Notice, though, that you should use the init function, not the config function. This is because VimTeX really demands that most of your options are defined before VimTeX is loaded. config does not guarantee this (for some reason). Thus, you should write:

require('lazy').setup({
  {
    'lervag/vimtex',
    init = function()
      vim.g.vimtex_view_method = 'sioyek'
      -- …
    end,
  },
})

I noticed here that I had to repeat all defaults. If this bothers me, I will raise a separate issue.

Yes, I was thinking of that already and was not sure if that would be annoying. If it really does not make sense to override the defaults, then perhaps the option should only be for adding new presets. We could also consider the "left-hand side" to be unique keys. That is, we could use a dictionary, and if you don't override the defaults they will remain active. But it would allow overriding them.

Regarding the syntax highlighting, I miss tree-sitter syntax highlighting for the included languages in Neovim. But this has already been discussed and answered.

Yes, unfortunately this is impossible as long as VimTeX uses the legacy syntax engine. Sorry. answered.