[themes] Explore theming support for semantic colors

[aeschli]

• define a set of Token Classifications that can be used for semantic highlighting
• evaluate actual color/style values based on the current text mate theme
• allow themes to define such colors and styles

[aeschli]

related: #32813

[kanlukasz]

* allow themes to define such colors and styles

Do you mean that there will be option to set color variables and use them with themes making?
I mean something like in Sublime Text

Here is example what i talking about:
https://stackoverflow.com/questions/56473901/is-it-possible-to-use-color-variables-in-the-visual-studio-code-when-creating-th

Or maybe it's possible right now but i don't know how?

[aeschli]

This is a proposal on how to represent the classification that a semantic highlighter computes (TokenTypes and TokenModifiers) and how themes and users can associate the style (TokenStyle). I'm still editing this proposal, but feedback and questions are welcome!

Semantic highlight range provider

A semantic highlight range provider takes a document (and potentially other arguments such as the current viewPorts) and returns a list of classified source ranges. The proposal is that each range can be classified with a TokenType and any number of TokenModifiers.

interface SemanticHighlightRangeProvider {
   provideHighlightRanges(doc,...): [ Range, TokenType, TokenModifier[] ][];
}

Note: This proposal focuses just on the classification types. The full SemanticHighlightRangeProvider API with a proper return type and optimizations such as compact token representation or ranges grouped by line will be part of a different proposal.

TokenTypes and TokenModifiers

A TokenType describes the symbol (class, field, variable...) at the location. The TokenModifiers provide additional information on the symbols (static, declaration, write access...).
TokenType and TokenModifiers are identified by a string id.

The purpose of splitting the classification into type and modifier is to simplify the theme definition (described later) and to make it easier for extensions to contribute additional classifications.

Predefined TokenTypes and TokenModifiers

VSCode will define a set of predefined (standardized) TokenTypes and TokenTypeModifiers, but extensions can add more if necessary:

Predefined TokeTypes:

• comments, strings, numbers, keywords, regexp, operators

• namespaces

• types, structs(Type), classes(Type), interfaces(Type), enums(Type), parameterTypes(Type)

• functions, macros(Function)

• variables, constants(Variable), parameters(Variable), properties(Variable),

• labels

Predefined TokeTypes Modifiers:
declaration : For all symbol declarations
documentation: For all occurrences of symbols in documentation
member: for member types, member variables (fields) and member functions (methods)
static for static (member types), member variables and member functions
modification for references where a variable is modified
async for declarations or references for functions that are async
deprecated: for annotating deprecated symbols
abstract: for annotating abstract functions.

TokenTypes and TokenModifiers Defaults

Each TokenType and TokenModifier come with a default style which is used when the actual theme does not define a style for it.

The default style can be specified as either

• a set of textMateScopesToProbe that are ran against the text mate theme rules the current theme defines to find a matching style.
• a predefined (static) style
• a reference to a different token-type to inherit the style of the other token type
• a function of one or more token-types

interface TokenTypeRegistry {
    registerTokenType(id, default: { textMateScopesToProbe: string[], value: TokenStyleValue })
}

type TokenStyleValue: = TokenStyle | TokenType | FunctionOf(TokenStyleValue)

Theme Definition

The active color theme (or user setting) will define colors and styles (TokenStyle) for TokenTypes and TokenModifiers or combination thereof.

interface TokenStyle {
   foreground?: Color | ColorFunction
   style?: bold | italic | underline
}

{
   "tokenStyles": {
       "(typenType|*)[.tokenTypeModifier]*": TokenStyle,
       //...
    }

• The first segment of a rule is the tokenType or '*', followed by modifiers. The order of the modifiers is not relevant.
• A rule matches if the token-type (e.g. const) matches or if the rule starts with * and when all modifiers of the rule are matched.
• The rule specificity is defined by the number of segments matching. That means
  const < *.declaration < *.declaration.exported < const.declaration.exported.
  (still to be decided: order when multiple rules have the same number of segments matching)
• The final style can be the result of multiple rules. For each style property (foreground, underline, bold, italics) the style of the most specific rule is applied. If a rule does not define a style, the rule is ignored for the evaluation of that style.
• The foreground color can be a combination of colors if ColorFunctions ('darker()') are used.

{
        "tokenStyles": {
            // all consts are blue and underlined
            "const": { "foreground": "#0000ff", "underline": true },
            // all declarations (types, functions, vars,....) are bold
            "*.declaration": { "bold": true },
            // all declaration that are also exported (public) are not bold, but underlined
            "*.declaration.exported": { "bold": false, "underline": true },
            // consts declarations that are exported are bold, underlined (inherited) and blue (inherited)
            "const.exported.declaration": { "bold": true },

            // all async decls and refs are rendered with a slightly darker color
            "*.asyncReturn": { "foreground": "darker()" }
        }

	"colors": { ... /* same as before */ },
	"tokenColors": [ 
                // still there for syntax highlighting and for evaluating 
                // default styles for TokenTypes and Modifier
		{
			"scope": [ "storage.type.annotation" ],
			"settings": { "fontStyle": "bold", "foreground": "#AA3731" }
		},
		{
			"scope": [ "entity.name.type", "entity.name.namespace" ],
			"settings": { "fontStyle": "bold", "foreground": "#7A3E9D" }
		}
		// ...
	]
}

TokenTypes and TokenModifiers Contribution

{
   "contributes": {
       "tokenTypes": [ {
           "id": "annotationType",
           "default": {
               "textMateScopes": ["storage.type.annotation"],
               "value": "type" // inherits settings from 'type'
           }
       }],
       "tokenTypeModifier": [ {
           "id": "autoBoxed",
           "default": {
               "value": { "bold": "true" }  // default to be used for '*.autoboxed'
           }
       }]
}