[feat] add lsp/diagnostics support for embedded languages

[jmbuhr]

• get embedded language chunks base on treesitter AST
• create hidden buffer with only the code of one language
• attach lsp server to this buffer
• synchronize position in qmd buffer with hidden R/python/julia buffer
• send lsp requests from the hidden buffer
• display results in qmd buffer
• make autocommands for this

current best idea: improve https://github.com/AckslD/nvim-FeMaco.lua to use these language buffers

[jmbuhr]

Let's follow up on the discussion at quarto-dev/quarto-vim#11 (comment) here.

TLDR: we need to fork the treesitter markdown grammar to create a quarto/pandoc grammar and also invent a new plugin for embedded languages.

• I didn't know you could set multiple filetypes, thanks @adrianocorrea ! That might be an angle. Though I do think it is not the final solution, as we wouldn't want e.g. R snippets interrupting while typing in markdown. And setting the filetype to R would also attach the LSP client, which would see all the markdown and just produce a lot of read squiggly lines. This is different from e.g. python.django, because django is syntactically valid python, but markdown is neither R nor python.
• Snippets are already included in https://github.com/jmbuhr/quarto-nvim-kickstarter/tree/main/snips but down the line we will probably want our own filetype for qmd that extends markdown (more on this down below) and then we can add these to friendly snippets.
• There is no need to add additional vim-regex syntax highlighting, as this is already covered by treesitter, however, there is room for improvement. So let's identify upstream projects we can contribute too in order to improve the experience.

(neo)vim does not have a dedicated mode for buffers with multiple languages (like emacs polymode). Instead, we will have to rely on the same trick that the quarto VS Code extension uses: We create a (hidden) buffer for each embedded language that contains just the code in that language with all other lines set to whitespace (this way line number still match). Then an lsp client attaches to this buffer and the challenge is now to synchronize between completions, hovers etc. between the hidden buffer and the quarto buffer (see this script for current experiments with this: https://github.com/quarto-dev/quarto-nvim/blob/dev/examples/run.lua note: R and P are custom functions I use for neovim developement: https://github.com/jmbuhr/.dotfiles/blob/main/nvim/.config/nvim/lua/jmbuhrconfig/utils.lua)

Now about potential upstream contributions:
We currently use the markdown treesitter grammar: https://github.com/MDeiml/tree-sitter-markdown
but this is focused on the commonmark spec and potentially github flavored markdown. Quarto uses pandoc markdown, so if we fork the markdown grammar and extend it to pandoc markdown we will also help pandoc users.
This will then automatically give us better syntax highlighting, formatting options etc.
For example, with the markdown spec we already recognize code blocks, which applies the correct highlighting, but this fails sometimes depending on how code block attributes are set.

This grammar is also used for example by the recommended commenting plugin (https://github.com/numToStr/Comment.nvim), which set's the comment string depending on the language context provided by treesitter.

And here is the thing. It would be the most elegant solution if we had a general and abstract implementation of exactly what the commenting plugin does: provide a different context for the document depending on the language type provided by treesitter. I would imagine a quarto document as having windows through which you can see into other buffers with the embedded languages. In it's most general form, this would allow us to get all the goodies you get from single file-type buffers such as lsp, completion, diagnostics, hover etc. in one quarto buffer. But I don't know how to build these windows, yet, which is why the last two bullet points are not checked.

On a side note, this would essentially be emacs polymode in neovim, so it might be useful for non-quarto users as well.

[jmbuhr]

Plugins with which we have to interface:

• completion plugin https://github.com/hrsh7th/nvim-cmp: swtich to a different buffer internally when within a code block
• diagnostics: get all virtualtext from a different buffer for code blocks and apply to quarto buffer
• hover, and other lsp shortcuts: switch to the language buffer internally. e.g. get the line number of a "go to definition" but apply the jump on the open quarto buffer

[jmbuhr]

https://github.com/jose-elias-alvarez/null-ls.nvim might be what we need

[jmbuhr]

diagnostics added via #6

[anand-bala]

@jmbuhr Thanks for the fantastic work. This plugin may be helpful for this integration: https://github.com/AckslD/nvim-FeMaco.lua

It creates a floating window for the code block, sets the correct file type, and synchronizes the contents into the current buffer.

[jmbuhr]

Thank you! This looks amazing and already so close to what we need! I will see what is missing (not sure if it takes into account that code blocks might refer to code in earlier blocks for example) and how I can contribute there :)