Support custom semantic tokens & modifiers

[radeksimko]

The feature is effectively gated on the server capability, so client and server part can be released independently. Our extension would just continue using the fallback token types and that one modifier until/unless the server claims support of the custom ones.

Related PRs:

• terraform-ls: internal/lsp: Provide (opt-in) custom semantic tokens & modifier terraform-ls#833
• hcl-lang: Support custom modifiers via BlockSchema & LabelSchema hcl-lang#106
• terraform-schema: schema: Add custom token modifiers terraform-schema#99

Follow-up issues:

• superType doesn't work as fallback within semanticTokenTypes - i.e. semanticTokenScopes are required
• modifiers for attributes
• modifiers for traversals

---

Possible UX

I may not have picked the best colour combinations, but the point I'm trying to make here is not the colours, but the ability to assign colours to the relevant constructs in a meaningful way.

{
  "name": "Example Theme",
  "semanticHighlighting": true,
  "semanticTokenColors": {
    "hcl-blockType.terraform-resource": {"fontStyle": "bold", "foreground": "#fff"},
    "hcl-blockType.terraform-data": {"fontStyle": "bold", "foreground": "#fff"},
    "hcl-blockType.terraform-module": {"fontStyle": "bold", "foreground": "#3d5aff"},
    "hcl-blockLabel.terraform-module.terraform-name": {"foreground": "#bac4ff"},
    "hcl-blockLabel.terraform-type": {"foreground": "#844FBA", "fontStyle": "bold"},
    "hcl-blockLabel.terraform-resource.terraform-name": {"foreground": "#a589b3"}
  }
}

Background

This enables the wider ecosystem of themes by providing custom token types & modifiers for highlighting in a way, which:

1. doesn't hamper UX with default themes
2. allows custom themes to be more specific
3. doesn't hamper other language clients which may not have the same themes as VSCode
4. allows us to make changes in both server and client independently

Language server is mainly responsible for associating all the custom token types and modifiers to the relevant pieces of code, assuming that the client communicates via client capabilities that it supports these custom types & modifiers. This allows the server to still provide sensible semantic highlighting to clients which do not support these custom types & modifiers.

We could use these types & modifiers in our static grammar too, but I have not explored that possibility too deeply.

• One clear benefit would be that themes would be applied immediately (before LS starts)
• The grammar may not be as accurate as semantic highlighting
• Use cases involving both custom themes and environment without language server are likely to be very few, if any
• There is maintenance cost associated with this

so given the above I'm leaning more towards keeping this mostly driven by the language server, at least for now.

The token types and modifiers are prefixed because apparently this is a global namespace and we don't want to end up in a conflict with any other extension.
microsoft/vscode#103097

[hashicorp-cla]

All committers have signed the CLA.

[radeksimko]

The scopes here just follow VSCode's defaults per https://github.com/microsoft/vscode/blob/02e8bd149ca3fb68e7f26ebec33f920ba0feef96/src/vs/platform/theme/common/tokenClassificationRegistry.ts#L541-L582

[radeksimko]

As per conversation in microsoft/vscode#145312 we could make these more specific, e.g. by appending .hcl to all scopes, but I'm not sure what is the best long-term strategy for token/scope "parity", so I just mostly mimicked the existing behaviours for semantic tokens for now.

[dbanck]

LGTM and works as expected; so much room for possibilities now!

Just a minor style suggestion, but feel free to ignore and merge anyway.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.