[tsserver] custom command (ex: to collect Angular2 @Component/selector)

[angelozerr]

At first I explain you what I would like to do with tsserver. @dbaeumer I think you could be interested with this issue for VSCode and Angular2 support.

Take the sample https://angular.io/docs/ts/latest/quickstart.html with Angular2 & TypeScript. There is a ts file:

import {Component} from 'angular2/angular2';
@Component({
    selector: 'my-app'
})
class AppComponent { }

And the HTML:

<my-app></my-app>

The HTML contains <my-app> which is binded with decorator @Component/selector property. My goal is to provide completion, navigation inside HTML (Eclipse) editor for <my-app>. In other words, I need to collect the whole @Component/selector node of the project to use it in a HTML editor. I tell me how I could do that:

• with TypeScript language service? Develop a walk kind which loop for each @component decorators? If yes, could you give me some info please to do that.
• after that I will need to provide this new command to tsserver. So it should be very cool if tsserevr could provide the capability to contribute with custom tsserver command (without compiling a custom tsserver).

With ternjs it's possible to write a tern plugin which is able to do that (collect @Component/selector) because:

• ternjs gives the capability to write a tern plugin where you can add your custom function when AST is walked (to help inference engine to create the well type, or collect modules like I have done with Angular Explorer .
• ternjs load tern plugin which follow a name pattern (tern-xxxx) from the node_modules to load it. See https://github.com/ternjs/tern/blob/master/bin/tern#L120

tsserver could do the same thing by searching inside node_modules with a pattern name like tsserver-xxxx to provide a custom command. After that any editor will be able to consume it just by installing with `npm installl tsserver-angular2'

Hope you will understand my issue.

[mhegazy]

//cc:@billti
@billti is looking into the extensiblity portion, so i will let him comment on that. one thing to note, the connection should go both ways. i.e. the template LS worker needs to know about the TS files, but the TS LS worker needs to know about it also to be able to handle things like rename, and find all references.

for your other question about extracting decorator information, here is a sample:

function visit(sourceFile: ts.SourceFile) {
    visitNode(sourceFile);

    function visitNode(node: ts.Node) {
        if (node.kind === ts.SyntaxKind.ClassDeclaration) {
           // only look in classes
            if (node.decorators) {
                node.decorators.forEach(decorator => {
                    if (decorator.expression.kind === ts.SyntaxKind.CallExpression) { 
                        // decorators of the form doc({...})
                        let callExpression = <ts.CallExpression>decorator.expression;
                        if (callExpression.expression.getFullText() === "Component") {
                            // found a component
                            if (callExpression.arguments && callExpression.arguments.length >= 1) {
                                let firstArg = callExpression.arguments[0];
                                if (firstArg.kind === ts.SyntaxKind.ObjectLiteralExpression) { 
                                    let object = <ts.ObjectLiteralExpression>firstArg;
                                    object.properties.forEach(p => {
                                        if (p.kind === ts.SyntaxKind.PropertyAssignment) {
                                            // key is selector
                                            if ((<ts.PropertyAssignment>p).name.getText() === "selector") {
                                                // found it!
                                                console.log((<ts.PropertyAssignment>p).initializer.getText());
                                            }
                                        }
                                    });
                                }
                             }
                        }
                    }
                });
            }
        }
        ts.forEachChild(node, visitNode);
    }
}


// Parse 
let sourceFile = ts.createSourceFile("test.ts", `
import {Component} from 'angular2/angular2';
@Component({
    selector: 'my-app'
})
class AppComponent { }
`, ts.ScriptTarget.ES6, /*setParentNodes */ true);


visit(sourceFile);

[angelozerr]

Thank a lot @mhegazy for your help! I will study your suggestion.

I'm glad that @billti works about this topic, very cool!

[angelozerr]

@mihailik @billti have you a road map to implement custom command? Otherwise I could start to develop something and try to do a PR with my idea.

Thanks for your help.

[billti]

I actually spent a lot of time on this over the past couple weeks. This turns out to be quite a non-trivial problem, and I've hit a few dead ends and back-tracked. Part of the problem is to do anything meaningful with the template code, you need to synthesize a file with some generated content from the template and ask questions about that, yet the program used by the language service is immutable once created, so you end up needing another language service working with the synthesized program, which the actual language service the editor uses then delegates to. (You still need the original one also to ask some of the initial questions before creating/delegating the synthesized one).

I've been playing around with a few different ideas (my scratch pad for various prototypes is at https://github.com/billti/TypeScript/commits/ngml . I use a similar syntactic model to Mohamed above for now to detect components, but note this could be fooled by a similar shape class/decorator that wasn't actually an Angular component), but I don't think we're close to committing to any plugin model and timeline yet. Once you expose something like this for folks to start using, you've committed yourself to supporting it and maintaining compatibility, and this is too complex an area to rush out a design.

[angelozerr]

@billti wow, I'm very impressed with your work. If I have understood you provide the capability to add plugin to support completion for template object property completion for ng attributes, for model, etc.

but I don't think we're close to committing to any plugin model and timeline yet

You mean that you have not planned to integrate your work in master TypeScript? Is there any chance that you do that in the future?

If it's possible, it should be cool if tsserver could load this plugin dynamicly (without compiling a custom tsserver)

My initial idea was to add a new command in tsserver at runtime to provide for instance a command which returns the angular model of a project (modules, directives) to display it in an outline like I have done for Angular1 https://github.com/angelozerr/angularjs-eclipse/wiki/Angular-Explorer-View

When angular model changes, teh command fires an event with the new change in order to the outline refresh as soon as model changes.

This command is added at runtime (no need to compile a custom version of tsserver). Your command is a npm module which starts with tscmd-

So you could have :

• node_modules
  • typescript
    • bin
      • tsserver
  • tscmd-ngml

After you call your tsserver with parameters -cmd ngml and with node require.resolve you load your custom command by searching require.resove("tscmd-ngml").

Thoses command could be hosted too inside tsconfig.json. This idea follows the same idea than ternjs to load her plugins.

[mihailik]

May I suggest an alternative?

Your proposal is about introducing plugin framework to tsserver. Instead you can achieve extensibility with composition.

The routing/execution of the requests inside tsserver goes through Session.executeCommand() and Session.handlers map. If you were able to intercept executeCommand or inject properties into handlers, you have your extensibility.

The idea is to have a wrapper script that fires the standard tsserver in a customised way. The necessary change to tsserver is minimal (see the actual patch):

process.nextTick(() => { //  <-- look over here

  const ioSession = new IOSession(ts.sys, logger);
  process.on("uncaughtException", function(err: Error) {
       ioSession.logError(err, "unknown");
  });
   // Start listening
   ioSession.listen();

});

Your wrapper script will go:

1. var ts = require('tsscript.js')
2. replace/modify ts.server.Session.prototype.executeCommand
3. let it continue

Now your project is in charge of complexity, not TypeScript.

This pay-as-you-go model is important to flexibility and support cost. If tsserver becomes a plugin host, the actual hosting model may suit some use cases better than others. When tsserver is used as a component, your app can take make all those decisions as suits your use case.

Your custom request handler may recycle the instance of tsserver, collect some data, spawn some crazy async logic — all that be hard with tsserver as an entry point and you as a plugin.

[billti]

@angelozerr The goal is to get this into master at some point, we just can't commit as to when exactly that point is right now, as we've just started exploring this space and trying to understand the scope of the work. There are a number of use-cases we need to think through, and ideally prototype, to ensure we make them practical/possible once we release and support this.

The goal is that any plug-ins or extensions would be written and compiled separately, referencing a supported API typing file. I'm just compiling my work as part of TypeScript for now as some of my experimentation also requires changes on the TypeScript side.

If I understand your requirements correctly, want you want might be something as simply as an event to your plugin whenever some structure in the code changes that you observe (in this case the Angular model), and you don't want to interfere with any of the existing compiler/language service functionality (such as completions, errors, compilation, etc.). Is this right?

One challenge with some of the suggestions above is that currently we can't assume that Node is the runtime and host for the language service or compiler. For example in Visual Studio or tsc.exe, there isn't a 'require' global method that has a standard resolution for locating and loading files. This is why we have abstractions for dealing with the host such as exist in https://github.com/Microsoft/TypeScript/blob/master/src/compiler/sys.ts or the LanguageServiceHost interface.

@mihailik While being able to monkey-patch the Session object would certainly be powerful, the approach doesn't really lend itself to a supported and strongly typed extensibilty model.

[mihailik]

@billti agree in long term!

My point is to leave the hosting logic to the outer application. Host application is best suited to coordinate lifetime, discovery, asynchrony. And tsserver should focus on AST-related things.

Monkey-patching of executeCommand is just a trivial option. Even better would be to expose handlers publicly (it is strongly typed already). Then you can module.exports=ioSession and give the caller an explicit typed instance of the session to deal with.

(I've changed the PR accordingly)

[billti]

After another brief discussion in the team, a further concern is about the other end of the "pipe". We can expose additional commands easily enough on the tsserver end, but then currently most of the editor hosts (e.g. VS, VS Code, Sublime, etc.), have no way of sharing their end of that stdin/stdout pipe to send the new commands. The part of the plugin living in the editor host could spin up it's own instance of tsserver to work with, but then that is duplicating a lot of cost (memory, file watchers, etc.) and gets expensive very quickly.

Maybe by definition any plugin that exposes additional commands REQUIRES editor host specific work anyway to call them (i.e. Python code for Sublime, JS for VSCode, Java for Eclipse, etc). But it is an additional wrinkle/challenge that'd we'd need to do work for each editor also to have a plugin model if we wanted a general extensibility story there, rather than baking support for certain plugins into the existing editor support.

[mihailik]

You can't own the editor-side story whatever your best intentions. Brackets, vim, emacs, Cloud9 etc. have their own plans, so prescriptive model will create friction and cost, even if there were one good for Python, Sublime and Java.

Perhaps the least-prescriptive model today is to let the editor-side drive, and get out of their way?

[angelozerr]

@angelozerr The goal is to get this into master at some point, we just can't commit as to when exactly that point is right now, as we've just started exploring this space and trying to understand the scope of the work. There are a number of use-cases we need to think through, and ideally prototype, to ensure we make them practical/possible once we release and support this.

I understand. I'm very exciting with this plugien feature. Hope it will be available soon.

The goal is that any plug-ins or extensions would be written and compiled separately, referencing a supported API typing file.

Great!

I'm just compiling my work as part of TypeScript for now as some of my experimentation also requires changes on the TypeScript side.

Yes sure.

If I understand your requirements correctly, want you want might be something as simply as an event to your plugin whenever some structure in the code changes that you observe (in this case the Angular model),

Yes to provide an Angular2 Outline.

and you don't want to interfere with any of the existing compiler/language service functionality (such as completions, errors, compilation, etc.). Is this right?

No, I need that too!

To be honnest with you, I have already done those 2 features (Angular Outline and custom completion for ng template) for Angular1 by developping a tern plugin for angular1. So I have started to develop a tern plugin for angular2. But as Angular2 is based on TypeScript, I need to customize acorn parser to parse TypeScript to support decorator and additional syntax of ts (it starts working but it's a very big work). So if TypeScript with tsserver is able to provide the same feature than ternjs plugin, I will use tsserver to support Angular2 inside Eclipse.

More, if you provide plugin with tsserver, you will able to support completion inside string for a lot of JavaSCript framework. For instance you could support CSS selectors completions, validation etc like have done https://github.com/angelozerr/tern-browser-extension#css-selector

This CSS selectors features works too for jQuery $(""). To support that, I mark the AST node $ and document.querySelector as "CSS selectors". I have a tern plugin which use this information to provide completion, navigation, validation, hover. IMHO, I think you should do the same thing. Mark your node as template.

One challenge with some of the suggestions above is that currently we can't assume that Node is the runtime and host for the language service or compiler. For example in Visual Studio or tsc.exe, there isn't a 'require' global method that has a standard resolution for locating and loading files. This is why we have abstractions for dealing with the host such as exist in https://github.com/Microsoft/TypeScript/blob/master/src/compiler/sys.ts or the LanguageServiceHost interface.

Ok, require.resolve was just a suggestion (tern use that) to load plugin at runtime. But if you provide an another mean to host plugin, that's fine.

@mihailik thanks for your PR. I will see it.

[angelozerr]

@billti @mhegazy I know it's diffiicult for you to give me an answer because plugins features is a POC, but this feature doesn't appear in the Roadmap and I tell me when you could (have the intention to) provide this plugin features (in several weeks? months? years?).

I would like just know if I continue to integrate tsserver inside Eclipse to support Angular2 or if I develop an Angular2 tern plugin by waiting for your plugin features (I would prefer the first option with tsserver).

Many thanks for your answer!