Add support for the new semantic tokens protocol

[matklad]

Just a heads up that LSP is currently adding a new API for semantic highlighting:

https://github.com/microsoft/vscode-languageserver-node/blob/master/protocol/src/protocol.semanticTokens.proposed.ts

Note this API is significantly different from the one currently implemented in lsp-mode. The new API is proposed by Microsoft team, and subjectively have higher chance of being stabilized. It might be a good idea to try it while it is in the proposed phase, to make sure it works OK for emacs.

At the moment, rust-analyzer and, I think TypeScript are servers that supports these API.

[matklad]

Also see microsoft/vscode-languageserver-node#576 for currently unresolved synchronization issues.

[yyoncho]

Thank you @matklad!

@sebastiansturm do you know if clangd is planning to implement that spec? I guess you will be interested in implementing that once clangd follows up?

[sebastiansturm]

I'll have to check if clangd support is planned or already underway, but if this new protocol is already supported by the TypeScript server I can use that for testing. I'll have to take a look at the proposal, but judging by the description of SemanticTokensRangeParams.range, tokens can now be requested for specific regions instead of being fed to the client at the server's leisure, which seems a lot more convenient to me

[tschuett]

clangd: https://reviews.llvm.org/D76663

[sebastiansturm]

nice, that's even better than using TypeScript for testing. Will rebuild clangd from trunk some time the next few days and have a go at it

[tschuett]

clangd: https://reviews.llvm.org/D77225

[sebastiansturm]

I pushed some preliminary semanticTokens support to my fork of lsp-mode, will use this over the coming week, then add modifiers, maybe semanticTokenEdits support and minor stuff such as missing docstrings.

[sebastiansturm]

tried this with a trunk build of clangd

[yyoncho]

@sebastiansturm do you have an update on that? I think this is about to be released in 3.16.

[sebastiansturm]

I have to admit I haven't followed up on my own todo list, though I've been using my fork as a daily driver with no issues for about a month now (using trunk builds of clangd). Will prepare a PR this evening