This proposal discusses support for a virtual file system (VFS) to TSServer. The contents of a virtual file system would be controlled by a client. Using virtual file systems, we believe we can deliver advanced features such as cross-file IntelliSense on vscode.dev and github.dev.

Context

The TypeScript server can currently work with two types of files: those on-disk and those in-memory (indicated by opening the file with a ^ prefix on the path). For the purposes of this discussion, on-disk files are files that the TSServer can independently read using nodejs file system apis, while the contents of in-memory files must always be synchronized with TSServer by a client.

Many IntelliSense features are only possible for on-disk files. This includes resolving imports across files, looking up typings, and constructing projects from a jsconfig or tsconfig. In all of these cases, TS implements these features by walking directories and reading files from the disk. None of this is currently possible for in-memory files.

However on VS Code, users are increasingly using virtual workspaces that TSServer cannot read directly. On GitHub.dev and vscode.dev for example, the workspace is provided by a file system provider that reads the workspace contents directly from GitHub or other code storage services. While we can synchronize the opened editors over the TS Server, IntelliSense support for them is still quite limited.

Brining proper virtual file system support to TSServer seems like best solution to enable a desktop like IntelliSense experience on GitHub.dev and vscode.dev

Motivating use cases

Cross-file IntelliSense on web

When a user opens a github.dev and vscode.dev workspace, we would like to provide cross-file IntelliSense by resolving imports. Eventually we would even like to provide project IntelliSense by parsing tsconfig/jsconfig files.

To implement this, we need to synchronize the workspace contents over to the TS Server so that the server can read files besides the ones that are currently opened.

Support for virtual workspaces on desktop

With desktop versions of VS Code, users can also open virtual workspaces. Working with JS/TS files in these virtual workspaces should be just like working with with JS/TS files on-disk.

The requirements to implement this are almost identical to the web case listed above.

Automatic type Acquisition (ATA) on web

When a user opens a JS/TS file from github.dev or vscode.dev, we would like to automatically download typings to provide better IntelliSense.

To implement this, we need a way to tell TS about typings files and where these d.ts files live within the project. Again, this is not possible today but we believe could be implemented using virtual file systems

Additional goals

• Do not introduce VS Code specific concepts even though VS Code will be the largest consumer.

• Do not requiring a significant rewrite of the entire compiler/server. For example, server is currently synchronous so our proposal must not require converting it to be asynchronous.

Out of scope

This proposal only discusses virtual file system support. We will discuss the specifics of the individual use cases above in separate issues.

Proposal

For the purposes of this proposal, a virtual file system (VFS) is a in-memory representation of a file system. The structure and contents of the VFS are provided to TSServer by the client. TSServer will use its in-memory VFS to implement file system operations, such as file reads and directory walks. By routing these operations through the VFS, we should be able to implement features such as cross-file IntelliSense without having to rewrite the entire server.

Implementing virtual file system support will require:

1. Establishing a protocol clients can use to work with a VFS.
2. Actually implementing VFS support inside TS Server.

This proposal focuses only on the protocol part of the proposal. I don't have enough knowledge of TSServer's internals to come up with a plan for actually implementing it.

Protocol

updateFileSystem

updateFileSystem is a new protocol request that clients use to update the contents of a VFS. It is inspired by updateOpen and would take a list of created, deleted, and updated files on the VFS.

Virtual file systems each have a unique identifier. This identifier is used in calls to updateFileSystem and also will be used to open a file against a specific VFS.

Here's an example request for a memfs VFS:

updateFileSystem {
    fileSystem: 'memfs',
    created: [
        { path: "/workspace/index.js", contents: "import * as abc from './sub/abc'" },
        { path: "/workspace/src/abc.js", contents: "export const abc = 123;" },
        { path: "/workspace/test/xyz.test.ts", contents: "..." },
    ],
    deleted: [],
    updated: [
        { path: "/workspace/test/xyz.test.ts", contents: "..." }
    ]
}

The above proposal takes a flat list of files similar to update opened. If we think it would be more convenient, we could instead take a tree-like structure.

When TSServer receives an updateFileSystem request, it must update its internal in-memory representation of this VFS. However it should not yet start processing any of these files.

Open file on a given VFS

After initializing a VFS, clients also need to then open a specific file on the VFS. For this, I propose we introduce a new style of path that can be used to talk about resources on a VFS:

memfs:/workspace/path/file.ts

This style of path is inspired by VS Code's uris. We would need to add support for them to all places in the protocol where we take or return a path.

Example

Let's walk through how VS Code could implement workspace-wide IntelliSense on vscode.dev using this proposal.

1. VS Code downloads and caches the entire contents of the workspace

   This is already implemented on the VS code side.

2. VS Code sends a static copy of the workspace over to TS Server using updateFileSystem

   updateFileSystem {
       fileSystem: 'memfs',
       opened: [
           { path: "/workspace/index.js", contents: "import * as abc from './sub/abc'" },
           { path: "/workspace/src/abc.js", contents: "export const abc = 123;" },
           { path: "/workspace/test/xyz.test.ts", contents: "..." },
       ]
   }
   

3. TS Server receives the file system contents and sets up its own representation of the virtual file system.

   With the above request, TS server would construct an in-memory representation of the file system that looks like:

   workspace/
       index.js
       src/
           index.js
       test/
           xyz.test.ts
   

   At this point, TS Server should not yet process any of these files or treat them part of a typescript project. The files are only held in-memory and can be read later

4. VS Code opens index.js on the virtual file system

   Let's assume this happens because the user clicked on index.js to view it.

   At this point, VS Code uses a normal updateOpen call to tell TS server that the user has opened a JS or TS file. This file is part of the virtual file system.

   updateOpen {
       openFiles: [
           {  file: "memfs:/workspace/index.js", contents: "import * as abc from './sub/abc';" }
       ]
   }
   

5. TS constructs project representation

   After index.ts is opened, TS processes it and starts building up a representation of the TS project. In this case, it sees the import ./sub/abc in index.ts and attempts to resolve the import. Using the virtual file system and opened files, the server first checks if the file memfs:/workspace/sub/abc.ts exists. Here all file system operations need to be routed through the virtual file system instead of trying to go to disk.

6. User requests go to definition on a reference to abc in index.js

   Here VS Code would send a definitionAndBoundSpan request:

   definitionAndBoundSpan {
       file: "memfs:/workspace/index.js",
       line: 1,
       offset: 10
   }
   

7. The server uses the VFS to respond

   definitionAndBoundSpanResponse {
       definitions: [
           { file: "memfs:/workspace/src/abc.ts", ....}
       ]
   }
   

Alternatives considered

Delegate file system operations to the client

Instead of eagerly syncing the VFS over to TSServer, we could instead delegate individual file system operations back to the client.

This is likely not possible without a significant rewrite of the server. The server expects file system operations to be synchronous, and there is no good way to synchronously communicate from the TSServer worker process back to main VS Code extension host process. Even if we could implement synchronous calls, doing so would not be ideal and would result in a large number of messages getting passed back and forth between the client and server.