Overview of approaches on a language server

[dummdidumm]

This thread is intended for an overview of approaches on how to implement a language server for svelte files. Discussion on a prefered solution can be done here aswell.

Current state of the language server

Currently syntax highlighting and some basic autocompletion works for svelte files.
Rich intellisense however is not provided, does not know about all the typescript files in the workspace, and also has no type information about the svelte components.
The same is true for the html part where special svelte syntax is highlighted but intellisense is not working. The vscode-html-languageservice is used which does not know about the special syntax by default.
The current language server does use the svelte compiler for parsing files and getting diagnostics like "no alt attribute on this img tag".
A more indepth analysis: Comment

Existing approaches/solutions

Disclaimer: I'm not the author of any of the existing solutions, I did look at the code however. If some of the information is wrong or not detailed enough, or you are missing a solution, feel free to comment, I will adjust it.

https://github.com/alexprey/sveltedoc-parser

Uses htmlparser2 to parse the html part of a svelte file. Uses hooks of the html parser to add custom parsing and extract relevant information.
Uses espree to parse the script parts of a svelte file. Then walks the AST to extract relevant information.
Provides the results in a JSON-Format.

https://github.com/ArdenIvanov/svelte-intellisense uses it to parse svelte files. It uses the result to attach some additional behavior.

Since this uses a javascript parser, it will not work with typescript.

https://github.com/halfnelson/svelte2tsx

Uses svelte's compiler to parse the HTMLx part of a svelte component. Then transforms the HTMLx plus the original script part to a tsx file with a self-written transformer.

https://github.com/simlrh/svelte-language-server/blob/feature/extend-ts-support/src/plugins/ts-svelte/service.ts (a fork from svelte-language-server) uses it to create shadow-tsx-files of svelte-files. It then uses typescript's language service, but proxies the requests: It replaces the original file path with the file path to the generated tsx-file. The typescript language server therefore checks the generated tsx files. Results are transformed back to get correct document positions.

https://github.com/halfnelson/svelte-type-checker-vscode is another language server using the svelte2tsx-library.

https://github.com/marcus-sa/svelte-ts

Description taken from implementer's comment:

Compiler:

1. The compiler reads all the source files
2. Everything except the content of the script tags gets thrown away
3. The script source code gets compiled to valid JS and declaration files.
4. Jumps back to step 2, where it replaces the TS source in script tags with the compiled JS
5. The Svelte compiler compiles the entire source file
6. The output compilation of above step gets cached, where the HTML AST is then used in the type checker.

It consumes the exported variables and functions from inside a Svelte script, and then generates it as a declaration files which contains a class that extends the runtime SvelteComponent.

Typechecker:

1. The type checker finds all components in the cached AST.
2. Checks all components in the template is a valid class which extends SvelteComponent
3. Checks for valid declarations, properties etc.

Other approaches

See comments of this thread.

• A language server similar to svelte2tsx but it produces plain ts files instead. It's a POC.

[tanhauhau]

So none of them are using Svelte parser?

[dummdidumm]

svelte2tsx strips out style/script tags and then uses the svelte compiler to parse the htmlx part.
svelte-ts uses the svelte compiler as a second step.
The current language server does not use the svelte compiler for parsing.
I'll update the initial post.

[ryanatkn]

I wasn't sure if I should share this, but some part of it might be useful. Last year (July/August 2019) I started playing around with typechecking Svelte, knowing that I was doing throwaway work just to learn. It supports some unknown amount of a full solution, maybe 30%, and it has some limitations (more on those later). There aren't any tests or external documentation because I've always considered it to be a dead end.

Svelte compilation goes as it does today with a non-typechecking TS preprocessor. If you wanted types in your HTMLx markup, not just your script tag, you'd need a preprocessor to handle that, or Svelte would need to handle TS internally.

The typechecker is a separate build step that uses the un-preprocessed TypeScript code from script blocks and the markup AST from svelte.compile to create a virtual Svelte TypeScript file solely for typechecking purposes. (here's where the Rollup plugin creates the virtual file and then typechecks it) It's similar to @halfnelson's strategy, but different in that it produces normal TS, not TSX, so in the markup it doesn't try to typecheck HTML tags and attributes, only the stuff inside mustache tags and Svelte components. It transforms Svelte components into constructors that get newed with their props. I got partway through supporting a lot of Svelte's constructs before realizing I got carried away more than I intended, and further progress should be Serious Engineering.

The TypeScript compiler API is used to construct a program in memory and typecheck the virtual Svelte TS. Unlike Svelte its view of the world is whole programs, not individual files. At the time it seemed like it would be straightforward to add incremental compilation and worker threads as future optimizations.

The virtual Svelte TS file is generated with a sourcemap which is then used to map TypeScript diagnostics back to the original Svelte source. It was pretty cool to see type errors point back to the Svelte source but I had no confidence in my approach overall.

One problem I remember that I couldn't figure out was with diagnostics that weren't specific enough in some cases. IIRC an example was with typechecking Svelte component constructor props - VSCode somehow would point to individual prop errors, but the TS diagnostics I was getting were pointing only to the constructor, not the problem property.

If you look through the code be aware that parts are a mess and some comments may be outdated. I originally used magic-string and at some point changed over to source-map. And remember it was a doomed endeavor from the start! Note that all of this is on the ts branch of that repo, not master.

Here are the 4 files referenced above:

• virtualSvelteTs.ts
• typecheck.ts
• tsDiagnostics.ts
• rollup-plugin-svelte-typescript-checker.ts

[dummdidumm]

I did some digging in the source code of the current svelte language server. Here are my thoughts on the typescript part of it:

Overview

The language server broadly works likes this:

• DocumentManager handles all documents that get opened.
• TypescriptPlugin registers at the DocumentManager to get invoked on language server events ("doHover" etc). It does not register directly: It gets wrapped with wrapFragmentPlugin which makes sure TypescriptPlugin only gets the content inside the script tag. Mapping of positions (of mouse/text cursor) is done in wrapFragmentPlugin to ensure the infos are shown at the correct position.
• TypescriptPlugin uses service, a typescript language service wrapper. It basically delegates all events to this language server, with some mapping to adhere to the method return types.
• service takes the document to do work on aswell as a createDocument method from TypescriptPlugin. service keeps his own map of documents which are at the moment basically only a delegate to the document of the DocumentsManager. Everytime service is invoked from TypescriptPlugin, service updates its documents map with the given document (after wrapping it). createDocument would be invoked if the language service opens a document that is not yet part of the documents map - but this will never happen since typescript language service does not find other svelte modules (see next section).

Current shortcomings and ideas for improvement

• Since the script tag is given as is to the typescript language service, it cannot deal with special svelte syntax ($).
• The typescript language service tries to lookup other files that are referenced in the currently open svelte file. For .ts/.js files this works, for .svelte files it does not. Reason: The typescript language service does not know about the .svelte ending, so it just assumes it's a normal typescript file and searches for files like ../Component.svelte.ts, which is wrong. In order to fix this, we need to implement the method resolveModuleNames on LanguageServiceHost and reroute all .svelte.ts file lookups to .svelte. Then the language service will walk all svelte files. NOTE: In order to implement this, we also need to fix the following bug: Since the typescript language service now would walk all files, it would invoke createDocument from TypescriptPlugin. But this currently returns the whole document, not only the content inside the script tag. This is a bug inside the wrapFragmentPlugin not wrapping openDocument.
• In order to implement more advanced features like "go to svelte file" by clicking on the svelte component import name, or make typescript grog the $ signs, we need to add a preprocessor. This preprocessor would somehow add the missing parts for typescript. Ideally, all additional stuff can be put on top of the existing code, so that mapping the positions is as easy as mapping the positions with offsets. This poses another challenge though: We now would have two position mappings: One from the whole svelte file to the script tag, and one from the script tag to the preprocessed typescript code. I'm not sure at this point if this is the way to go or if we should rework how TypescriptPlugin (and others) get its document - maybe both extraction of the script tag and preprocessing should be done in one step. This would mean changing or fully replacing the wrapFragmentPlugin. Another argument for reworking that is that, in order to correctly determine things like "unused function", we have to take the html part into account, so the script tag is not enough.

Thoughts?

[ryanatkn]

On resolveModuleNames, here's how a TypeScript team member did it for Vetur, and here's a similar version I did for Svelte, which includes a comment that mentions an alternate strategy that worked, but copying Vetur seemed better.

[swyxio]

@dummdidumm do you mind acting as tech lead and breaking these things out to specific issues? i would be happy to volunteer dev time, but i find alot of this overwhelming and would work better in small, defined issues. Unless there is some big architectural change that needs to be done (looks like no?)

Basically just open up a bunch of issues and let people claim them and manage them as our informal tech lead on this. I'd offer to do it but honestly just havent felt the pain yet haha

also linking to orta's original LSP post for others who are browsing sveltejs/svelte#4518

[dummdidumm]

Sure I would love to help on this, but right now I'm just a random dude who is very excited about the language server, digging into the code and making some suggestions 😄 I also need some more time to get familiar with the code, its architecture and lsp in general. But I'm definitely willing to push stuff forward. @orta since you are the maintainer of this repo, what do you think about this?

[orta]

👋 Yeah, we both are random people who just care - I don't even have any svelte codebases to trigger my "this should be better" itch. So, I'm very happy to see any movement at all - @dummdidumm - push away and I'll back you up and try make sure everything runs smoothly

[dummdidumm]

Ok great! I will go ahead and make some PRs restructuring some of the code and preparing it so we can then parallelize working on individual items.

[jasonlyu123]

Maybe we could combine @ryanatkn, svelete-ts's approach and eslint-plugin-svelte3's variable approach:

1. Use svelte preprocess to preprocess script into js.
   Here we could just tranpile to latest or next version of ES to do as little transformation as possible.
2. Let svelte compile the preprocessed source.
3. We can then inject result vars where injected = true into instance ts source or ts AST and then use it with source map to provide type check and auto-completion.

The script can have three fragments: module <script context="module">, instance and mustache (template). The mustache part could just be all dump into a template.js for now. But for the long term we could reflect template into tsx or plain ts like ryanatkn's approach.

Module vars, where module = true, would be injected into template.js and instance but not the other way around.

In this way we don't have to re-implement on how to find reactive variable $: a = 1 or $-prefixed store in typescript AST. These variable injected by svelte compiler derive from top-level statement and we're transpiling it into latest ES. So it should mostly not be rename by typescript.

About reactive variable $: a it could be written like this:

$: a = 1

let a: number
$: a = 1

or

$: a = someFunction(b) as Type

All it's valid ts that doesn't need transform before transpile

And $-prefixed store we can create a generic function like svelte/store get to extract store type

/** injected */
let $store = getType(store)

[dummdidumm]

1. Use svelte preprocess to preprocess script into js.
   Here we could just tranpile to latest or next version of ES to do as little transformation as possible.
2. Let svelte compile the preprocessed source.
3. We can then inject result vars where injected = true into instance ts source or ts AST and then use it with source map to provide type check and auto-completion.

I think svelte-ts does something like that. Seems like a good idea. On injected = true: What code exactly has this property?

The script can have three fragments: module <script context="module">, instance and mustache (template). The mustache part could just be all dump into a template.js for now. But for the long term we could reflect template into tsx or plain ts like ryanatkn's approach.

Module vars, where module = true, would be injected into template.js and instance but not the other way around.

In this way we don't have to re-implement on how to find reactive variable $: a = 1 or $-prefixed store in typescript AST. These variable injected by svelte compiler derive from top-level statement and we're transpiling it into latest ES. So it should mostly not be rename by typescript.

So you want to split up the file into a virtual .ts-file and a .template-file? Why not have the the mustache-parts on the bottom of the .ts-file? This way we also would get rid of "unused method"-warnings. On walking the AST: Yes, I think we can be smart here and skip looking into the AST too much because everything that is referenced in the template must be top level in script.

About reactive variable $: a it could be written like this:

$: a = 1

let a: number
$: a = 1

or

$: a = someFunction(b) as Type

All it's valid ts that doesn't need transform before transpile

Is it possible to infer the type? Maybe this is just something we leave as is and if the user wants to use typescript, he has to define let a: number himself. As an alternative, we could insert let a;, but then it would be any.

And $-prefixed store we can create a generic function like svelte/store get to extract store type

/** injected */
let $store = getType(store)

Yeah this seems nice. Another idea I had was to construct getters/setters.

[jasonlyu123]

I think svelte-ts does something like that. Seems like a good idea. On injected = true: What code exactly has this property?

The compile result has a property vars, which is an array of object that has the following property:
name, export_name, injected, module, mutated, reassigned, referenced_from_script and writable
you can see svelte compile api for more info

So you want to split up the file into a virtual .ts-file and a .template-file? Why not have the the mustache-parts on the bottom of the .ts-file? This way we also would get rid of "unused method"-warnings.

This is the approach of eslint-plugin-svelte3. Also I just thought if we want to support ts in template because there is no attribute to specify language.

Is it possible to infer the type?

Since this virtual source is only use for type check and autocomplete, I presume it can achieve by just copy the first assign statement to initialize it but it seems to be need more work.

[ryanatkn]

Is it possible to infer the type? Maybe this is just something we leave as is and if the user wants to use typescript, he has to define let a: number himself. As an alternative, we could insert let a;, but then it would be any.

Looks like normal inference doesn't get us all the way there. (did you mean compile-time inference? not sure if that could work)

let a; // type is "any" initially
$: a = 5;
a; // type is now "number", good
const cb = () => a; // type is implicitly "any", noo

The reason is that in the linear code flow, TypeScript can see a wasn't reassigned, but nested function scopes don't have that guarantee, and TypeScript errs on the side of type safety.

playground example