feat: Custom file view provider for LSP4IJ TextMate files that uses semantic token information to create a simple PSI "tree" (fixes #98, #531, and #813) #853

SCWells72 · 2025-02-18T21:12:08Z

Okay, I'll be adding quite a bit of detail to the changes themselves, but at a high-level, this adds a FileViewProviderFactory for LSP4IJ TextMate files and plain text/abstract file type files in Community Edition IDEs to simulate a flat PSI tree based the language server's reported semantic tokens for the file including whether those elements represent declarations, references, comments, string and numeric literals, etc.

Non-user-defined language server integrations can also opt-in for these features without changing their files' PSI element hierarchies by registering a file view provider factory for their files' language or file type. This has been confirmed to work properly via collaboration with @ericdallo and is documented in LSPApi.md.

The end result is that the following issues are fixed and features are added:

For LSP files against servers that support sufficiently descriptive semantic tokens, only declarations and references to declarations are highlighted on Ctrl/Cmd+Mouseover, and nothing else is ever highlighted.
For LSP files against servers that do not support semantic tokens such as the CSS language server, the entire file is no longer highlighted on Ctrl/Cmd+Mouseover. This is also the behavior for LSP files against language servers that do support semantic tokens when the semantic tokens for the file are not yet populated.
The tooltip shown when hovering in this manner is now much more descriptive, showing the declaration type if known from the semantic tokens and their modifiers followed by the declaration name and, if hovering over a reference that resolves to a declaration in another file, the target file name is also shown. Also, the symbolic names are displayed as code in these tooltips.
Attempting to navigate on a reference takes you to the declaration. Attempting to navigate on a declaration shows its usages.

This feature can be disabled for user-defined language servers via the client configuration setting editor.enableSemanticTokensFileViewProvider. That setting defaults to true for all language server definitions, but as stated above, unless a non-user-defined language server integration opts in explicitly by registering a file view provider, it is effectively disabled.

Here's a little demo:

Regarding concerns about memory bloat or slowness from something like this, I confirmed the following:

IDE memory usage before opening large file was ~200MB (after multiple forced GCs).
Opened a ~30K line TypeScript file (lib.dom.d.ts).
IDE memory usage increased about 45MB (again, foricing GCs).
Verified that file editing performance was normal.
Verified that the new Ctrl/Cmd+Mouseover behavior worked properly and tracked accurately after making changes.
Verified that the standard Go To Declarations or Usages worked properly in the file from both references and declarations.

…Mate files that uses semantic token information -- if available -- to create a simple PSI "tree" (albeit flat) with the information from the semantic tokens.

SCWells72 · 2025-02-18T21:13:24Z

src/main/java/com/redhat/devtools/lsp4ij/LanguageServersRegistry.java

+     * @param language    the language
+     * @return true if the language of the file is supported by a language server and false otherwise.
+     */
+    public boolean isFileSupported(@Nullable VirtualFile virtualFile, @Nullable Language language) {


I needed a new signature of this to use in FileViewProviderFactory.createFileViewProvider() that doesn't need the PSI file at all as it's in the process of creating that file. We do, however, have the virtual file and language at that time, so this just perfoms the same general evaluation based on that information.

Please keep the same logic than

lsp4ij/src/main/java/com/redhat/devtools/lsp4ij/LanguageServersRegistry.java

Line 532 in 208b2f6

if (!file.isInLocalFileSystem()) {

I can't keep the exact same logic because I can't get the PSI file for the virtual file in this context of creating the PSI file. That will result in a StackOverflowError. This is the closest it can be to the original:

public boolean isFileSupported(@Nullable VirtualFile file, @Nullable Language language) { if (file == null) { return false; } FileType fileType = file.getFileType(); if (fileAssociations .stream() .anyMatch(mapping -> mapping.match(language, fileType, file.getName()))) { if (!file.isInLocalFileSystem()) { if (file instanceof LightVirtualFile) { return false; } } return true; } return false; }

which, using the IDE's inspection-based suggestions, simplifies down to:

public boolean isFileSupported(@Nullable VirtualFile file, @Nullable Language language) { if (file == null) { return false; } String fileName = file.getName(); FileType fileType = file.getFileType(); if (fileAssociations .stream() .anyMatch(mapping -> mapping.match(language, fileType, fileName))) { return file.isInLocalFileSystem() || !(file instanceof LightVirtualFile); } return false; }

Are you okay with that, or would you prefer the first option above?

Indeed it is better, please use that.

Mmm wait it is not the properversion, There is a check with phiycal also.

There's no PSI file to check for being physical here. That's the point...there needs to be a check for whether a virtual file and its language are supported by a configured language server definition while creating the PSI file in the view provider factory. So it's limited to what can be checked about a virtual file, and we're already asking if it's in the local file system and/or a light virtual file. Does that help explain the use case and associated limitations?

Oh sorry, you must not get the PsiFile if I have understood. It could look like this:

public boolean isFileSupported(@Nullable VirtualFile file, @NotNull Project project) { if (file == null) { return false; } Language language = LSPIJUtils.getFileLanguage(file, project); return isFileSupported(file, language, true); } } public boolean isFileSupported(@Nullable VirtualFile file, @NotNull Language language, boolean checkPsiFile) { if (file == null) { return false; } FileType fileType = file.getFileType(); if (fileAssociations .stream() .anyMatch(mapping -> mapping.match(language, fileType, file.getName()))) { if (!file.isInLocalFileSystem()) { if (file instanceof LightVirtualFile) { return false; } if (checkPsiFile) { PsiFile psiFile = LSPIJUtils.getPsiFile(file, project); if (psiFile != null && !psiFile.isPhysical()) { return false; } } } return true; } return false; }

This line:

PsiFile psiFile = LSPIJUtils.getPsiFile(file, project);

would lead to a StackOverflowException in this specific context because that PSI file is being created when this check is made. That's the whole reason for this VirtualFile/Language-based check.

I understand that, it is the reason why I add checkPsiFile parameter. In your case you will call the method with false.

We don't have a Project in this signature to retrieve a project-specific PsiFile for the VirtualFile, so that won't really work.

I think this should cover the concerns/confusion:

/** * Returns true if the virtual file and optional language are supported by a language server and false otherwise. * This signature should be used only when it is not yet possible yet to check the PSI file because it is in the * process of being created via a file view provider factory. If a PSI file is available, one of the other * signatures of {@link #isFileSupported} should be used instead. * * @param virtualFile the virtual file * @param language the language * @return true if the virtual file is supported by a configured language server and false otherwise. */ @ApiStatus.Internal public boolean isFileSupported(@Nullable VirtualFile virtualFile, @Nullable Language language) { if (virtualFile == null) { return false; } FileType fileType = virtualFile.getFileType(); String fileName = virtualFile.getName(); if (fileAssociations .stream() .anyMatch(mapping -> mapping.match(language, fileType, fileName))) { return virtualFile.isInLocalFileSystem() || !(virtualFile instanceof LightVirtualFile); } return false; }

The method is clearly documented for a specific use case and refers the caller to the other signatures if not for that use case. The parameter is clearly named virtualFile to clarify that it operates on a VirtualFile and not a PsiFile. The absence of a Project in the parameter list makes retrieval of the correct PsiFile an issue anyway. Finally, it is annotated as @ApiStatus.Internal to make it clear that this signature should not be used outside of the internal implementation of LSP4IJ.

I'll get this committed along with the change to lazy-create the semantic token PSI element, then I'll add a client config-based feature flag to disable this behavior if desired.

SCWells72 · 2025-02-18T21:13:43Z

src/main/java/com/redhat/devtools/lsp4ij/features/highlight/LSPTargetElementEvaluator.java

@@ -27,7 +30,12 @@
 */
 public class LSPTargetElementEvaluator extends TargetElementEvaluatorEx2 {
    @Override
-    public @Nullable PsiElement adjustReferenceOrReferencedElement(@NotNull PsiFile file, @NotNull Editor editor, int offset, int flags, @Nullable PsiElement refElement) {
+    @Nullable
+    public PsiElement adjustReferenceOrReferencedElement(@NotNull PsiFile file,


These are just formatting changes.

SCWells72 · 2025-02-18T21:14:22Z

src/main/java/com/redhat/devtools/lsp4ij/features/highlight/LSPTargetElementEvaluator.java

@@ -36,7 +44,13 @@ public class LSPTargetElementEvaluator extends TargetElementEvaluatorEx2 {
            return null;
        }

-        // Try to find the word at the caret and return a fake PSI element for it
+        // See if the view provider can provide an element
+        FileViewProvider viewProvider = file.getViewProvider();


If this file uses a file view provider based on semantic tokens, we can use it to find the semantic token-based element at the specified offset.

SCWells72 · 2025-02-18T21:14:36Z