🚀 Feature: API surface for developing an up-to-date VS Code Extension

[Danielku15]

Is your feature request related to a problem or a nice-to-have?? Please describe.
As of today there is no up-to-date integration of Mocha into Visual Studio Code. There is the outdated and not maintained Mocha Test Explorer using an own test explorer instead of the VSCode Integrated one.

I was thinking of making a new extension utilizing the VS Code Testing API based on the examples the provide.

Unfortuantely integrating with Mocha in the right way for scraping the tests and executing them is not easy.

Describe the solution you'd like

It would be great if Mocha would expose additional APIs to fill the missing gap between Mocha and VS Code. For this I would need:

1. Provide a programmatic way of initializing a Mocha instance from a workspace directory (loading any available configs).
2. Provide a programmatic way of accessing the test hierarchy including the related file location.
3. Provide a programmatic way of executing test given any "test hierarchy" objects (as exposed by 2.).
4. Provide a programmatic way of watching the workspace and listening for any modificiations in the tests to refresh the test hierarchy.

Describe alternatives you've considered

1. Adopting the code from vscode-mocha-test-adapter which does a lot of stuff manually.

Additional context

[JoshuaKGoldberg]

I'm curious how much this intersects with the other API requests, namely those filed by @nicojs for Stryker: #3882, #4572.

It makes sense why you'd want these APIs though. I think given #5027 we're not likely to get to this soon, but it's on our radar. @Danielku15 if you could post more details that'd be really helpful:

• What's the total list of important APIs you need? As in, for each, the root behavior they're implementing, and possible inputs & outputs?
• What APIs already exist that do what you want?
• What are the things vscode-mocha-test-adapter has done manually that you'd like to move to an API?

I think the OP mentions the "root behavior" points nicely - so getting more in-the-weeds on what proposed APIs would satisfy them would help us triage this. Thanks! 🤎

[Danielku15]

What's the total list of important APIs you need? As in, for each, the root behavior they're implementing, and possible inputs & outputs?

I think it is hard to tell directly how the API signatures could look like. Mainly because I am not yet aware how mocha internally already stores the whole test file > suite > case hierarchy. To teach VS Code about tests you create vscode.TestItem objects. Beside the attached custom infos (e.g. for later execution or syncing with Mocha) the properties are: https://vscode-api.js.org/interfaces/vscode.TestItem.html

Looking at the example extension from Microsoft you can get a bit the feeling how things would integrate:https://github.com/microsoft/vscode-extension-samples/tree/main/test-provider-sample/src

To adapt the example to Mocha roughly the points mentioned above would be needed. I tried to define an interface description that could serve the plugin:

class MochaTestExtensionFacade {
    // Goal: Provide a programmatic way of initializing a Mocha instance from a workspace directory (loading any available configs).
    // Usage: This would be the main "context" object within the VS code extension
    // Behavior: this would setup internally mocha with all the configs and details like during normal test execution
    // in a directory making everything ready for the other operations.
    // any transpilers, plugins and settings would have to be respected (e.g. to support typescript)
    constructor(workingDirectory: string);
    dispose(): void;
    
    // Goal: Provide a programmatic way of accessing the test hierarchy including the related file location.
    // Usage: This would be used to obtain all details to fill the test list/explorer.
    // Behavior: provides all test files with the contained details about the tests.
    getRootSuite(): Promise<mocha.Suite>;
           
    // Goal: Provide a programmatic way of executing test given any "test hierarchy" objects (as exposed by 2.).
    // Usage: This would be used when a user requests test execution on. we 
    // Behavior: would create a runner which can be used execute the test (suite). 
    createRunner(suite:mocha.Suite): Promise<mocha.Runner>; 
    createRunner(suite:mocha.Test): Promise<mocha.Runner>; 
    
    // Watching Variant A: using Mocha watcher
    
        // Goal: Provide a programmatic way of watching the workspace and listening for any modificiations in the tests to refresh the test hierarchy.
        // Behavior: Similar to the existing `mocha --watch` this would internally setup
        //           all file system watchers and components to detect any dynamic changes done by the user. 
        //           on the input code. 
        constructor(workingDirectory: string, watch: boolean);
        // Goal: get notified about the changes in the root suite
        // Usage: We would listen to those changes and then dynamically update the test list/explorer accordingly. 
        onRootSuiteChanged( callbackWithChanges: (added:(mocha.Suite|mocha.Test)[], removed:(mocha.Suite|mocha.Test)[], modified:(mocha.Suite|mocha.Test)[])=>void );

    // Watching Variant B: using VS code watchers
        // Goal: Provide a programmatic way of watching the workspace and listening for any modificiations in the tests to refresh the test hierarchy.
        
        // Usage: We use those patterns to setup vscode.workspace.createFileSystemWatcher(<pattern>)
        // Behavior: It returns the file path patterns to start listeners. If the mocha config is changed this is signaled. 
        getWatchPatterns(): Promise<string[]>
        onWatchPatternsChanged( callbackWithChanges: (added: string[], removed:string[]) => void ): void;
        
        // Usage: We would call this when VS code detects changes on files.
        // Behavior: It would add or reload the tests and suites from the specified file 
        addOrUpdateFile( uri: string ): Promise< { added:(mocha.Suite|mocha.Test)[], removed:(mocha.Suite|mocha.Test)[] } >;
        
        // Usage: We would call this when VS code detects deleted files.
        // Behavior: It would add or reload the tests and suites from the specified file 
        removeFile( uri: string ): Promise< { removed:(mocha.Suite|mocha.Test)[] } >;
}

There might be the need for extensions/adaptions on existing interfaces and types to hold all information.

What APIs already exist that do what you want?
There are some APIs on the main Mocha object but from a workflow they don't really fit:

• Initializing a Mocha object like the CLI does is not publicly available. You have to do the loading manually.
• There APIs for loading and unloading files. But the loaded test structure is not available.
• For exeuction most things seem available. But unfortunately no "incremental" status updates are there. A user would expect to see how tests are pending, executing and done (success/fail) including the related logs attached, execution times shown, code coveragte trackable etc.

What are the things vscode-mocha-test-adapter has done manually that you'd like to move to an API?

Basically the things described above. The old adapted patched out various things in mocha to obtain the list of tests with their locations. I think also for execution a similar pattern is used to hook into the execution chain. For options loading they require duplicated configuration or attempt loading some settings manually from configs.

https://github.com/hbenl/vscode-mocha-test-adapter/blob/master/src/worker/patchMocha.ts
https://github.com/hbenl/vscode-mocha-test-adapter/blob/master/src/configReader.ts
https://github.com/hbenl/vscode-mocha-test-adapter/blob/master/src/worker/reporter.ts

[voxpelli]

Just wanted to chime in and say that I'm very 👍 to this and I personally would also be very open on having an official VSCode extension be maintained within the mochajs org, based on https://github.com/hbenl/vscode-mocha-test-adapter or written from scratch.

Is this something you would be interested in helping out with @Danielku15?

[Danielku15]

It would be certainly in my interest to have an adapter but I'm also new into the topic of writing such a test extension and also to the mocha codebase. Hence I'm not sure how well I can progress in making the required changes. I lately had to shift the focus in my project a bit more to the features than to the dev-experience but somehow I miss every time such a plugin when working on features.

[voxpelli]

@Danielku15 I wasn't meaning to suggest that you should be building it all, just thinking if you want to be included if we are taking on doing an official VSCode Extension ;)

[Danielku15]

@Danielku15 I wasn't meaning to suggest that you should be building it all, just thinking if you want to be included if we are taking on doing an official VSCode Extension ;)

@voxpelli It would be definitely interesting. I am already thinking about this a lot and I think I have most bits together to define whats needed to develop this 😁 Warning: this post is again a bit alonger but should give some insights on how we can tackle this topic.

VS Code Extension Development

I was fiddling a bit these days with the extension APIs of VS Code and realized something funny: The offical extension for Testing VS Code Extensions is using Mocha under the hood 😆

https://marketplace.visualstudio.com/items?itemName=ms-vscode.extension-test-runner
https://github.com/microsoft/vscode-extension-test-runner

I think this is the best starting point for a standalone mocha extension. They have their own config files and additional features for testing against VS Code but from what I've seen we should be able to adopt a big portion of it. Some interesting bits of how the plugin is organized:

1. They have their own config files where they define the wilcard pattern for the tests. https://github.com/microsoft/vscode-extension-test-runner/blob/main/src/constants.ts#L8
2. They work on compiled files. e.g. if you use TypeScript you have to have a watch compilation active writing the test outputs. But they consider source maps.
3. They extract the tests by running the JavaScript files in a separate JavaScript VM having own symbols defined for the Mocha functions: https://github.com/microsoft/vscode-extension-test-runner/blob/main/src/extract/evaluate.ts#L101-L122
4. As runner they have an own dedicated CLI package https://github.com/Microsoft/vscode-test-cli There they launch in a CLI command Mocha with a special JSON reporter to enable communication with VS Code.

High level concept for the Mocha VS Code Extension

With all these learnings I am thinking of following approach for an official Mocha VS Code plugin:

1. The plugin watches for mocharc configuration files within workspaces and creates test controllers for them.
2. We spin up a long-running mocha CLI exposing a JSON interface (via stdout/stdin).
   1. The CLI starts up like mocha --watch keeping track of all suites and tests in the project.
   2. The CLI exposes JSON events about the "Test Model" (suites/tests) modifications.
   3. The UI synchronized the test tree with the mocha model.
   4. If the CLI exits unexpectedly we try to restart it.
3. For test execution, coverage and debugging we send the test execution request to the already running CLI.
4. For debugging we let VS Code attach to the already running process.

The benefits of this approach:

1. The VS Code extension and normal test execution via CLI will behave the same (with all features like transpiling, file watching,...).
2. Development and testing of this functionality can be done almost fully without VS Code Extension development skills.
3. The VS Code extension only needs changes in case the interaction with the CLI has breaking changes or we want new features. With new features added to Mocha the extension might not need to be extended every time.

High level work items

1. Define the JSON RCP Protocol which will be used to establish the communication between the mocha CLI and the VS Code plugin.

   • For the execution parts we can build on https://github.com/microsoft/vscode-test-cli/blob/main/src/fullJsonStreamReporter.cts or the existing JSON Stream reporter https://mochajs.org/api/reporters_json-stream.js
   • For the Test Model synchronization I will have to dive first into the details how Mocha updates and maintains the test tree when using --watch. The events will just be a 1:1 mapping to what already exists there.

2. The test model and internals in mocha have to be extended with more details.

   • We need unique IDs for all 'nodes' (Suites, tests,..) in the test model.
   • We need to store for 'nodes' their source location (including source map support).
   • The test results have to be stored and propagated on all nodes of the test model.
   • No information about skipped tests is exposed.

3. We need to implement the Mocha CLI with the JSON RCP interface.

   • Simply a new CLI parameter like --json-rpc
   • A new API controller should take care of connecting the STDOUT/STDIN with the internal Mocha instance. The Mocha instance should expose the data and events in a normally typed fashion.
   • For supporting cancellation we might need to execute some parts asynchronously so that we have a chance to read the cancel message on the API controller.

4. We need to implement the VS Code Extension

   • Adopting the extension from Microsoft should give a great foundation as their architecture is similar.
   • The extension will spawn something like npx mocha --json-rpc in the respective directories holding package.json and mocharc files (+ maybe setting an environment variable like MOCHA_CLI_JSON_RPC=true).
   • Users might customize the command via settings (e.g to a simple npm run test). This ensures best compatibility and flexibiltiy.

[Danielku15]

I tried to sketch out an initial API design here: https://github.com/Danielku15/mocha/blob/feature/json-api/docs/api-tutorials/json-api.md

I'm not sure if it will be technically possible to provide it like this but I think this is roughly what the Mocha backend would need to provide.

My main worries are around handling stdout/stderr and the cancellation of tests as we're single threaded in Node.js. Might be better in step 1 to only support cancelling of full test runs. then we can just spawn another nested mocha process when executing tests and capture all outputs from it or cancel the process.

Would be good to get some feedback about the feasability and design of the API. I'd like to make a small PoC based on what mocha already exposes today.

[Danielku15]

@voxpelli Success 🥳🎉 I forked the VS Code Extension Test runner from Microsoft and rewrote it to work with plain Mocha. Here a demo from within my project. It activates with the mocharc files and then provides all features like running and debugging tests. It also works with TypeScript. Only code coverage I left out for now as this depends again on the framework used.

Here a sneak peek video:

Code_lQARMvDNpt.mp4

I cleaned my code and uploaded my current state to CoderLine/mocha-vscode#1
Still some things to figure out and discuss but I'm already happily running and debugging tests in my project using this extension 😁

Let me know what you folks think.