[javascript] Support javascript libraries?

[pingpongboss]

Open question.

[appsforartists]

Here are some AST parsers for JS:

• https://github.com/babel/babylon
• https://github.com/jquery/esprima

Here are some things that came up when I searched for JS AST diffing:

• https://github.com/GumTreeDiff/gumtree
• https://github.com/prettydiff/prettydiff
• https://github.com/Vunovati/astii

What's needed to build something like this? Writing a tool to diff ASTs and glean meaningful data out of the diffs sounds non-trivial.

[pingpongboss]

Extract Structured Form of Public API

This is the hard part.

You need an enumeration of the public API: public class signatures, their public method signatures, and their public field signatures. Maybe javascript's definition of a public API is different, you'll have to figure that out.

The enumeration can be in any structured form. For example:

• Java libraries outputs a file like https://gist.github.com/pingpongboss/95392faf84648c55f988333e108d1774 for each class when built. You can see how each line can be parsed quite easily, no AST needed.
• iOS libraries can be processed through sourcekitten to get a JSON object for the entire project's AST. I assume that sourcekitten doc only processes the public classes, methods, and fields since that is all that's needed for docs. Maybe it would help if @jverkoey provides a Gist of what the AST looks like in JSON form.

It seems like you've found several tools that can parse javascript and generate an AST. Babylon looks promising with ClassDeclaration, ClassMethod, ObjectMethod, FunctionDeclaration, VariableDeclaration, etc. My gut says that most of the "AST diffing" tools may lead you in the wrong direction. They're mainly concerned with replacing line-by-line diff, but you're only interested in the public APIs not their implementations.

The next few steps are quite easy and can borrow heavily from the iOS and Android scripts.

Parse Public API into Intermediary Format

You'll parse that AST and extract only the information you need into an intermediary format for easy processing. This is done for both the old AST and new AST.

Example:
iOS
Android

Calculate Changes between Public APIs

Your intermediary format should make this easy.

Example:
iOS
Android

Print Changes in Markdown

Example:
iOS
Android

[appsforartists]

Thanks. That's helpful info.

It could be especially hard in JS because JS is a dynamic, untyped language. Therefore, you can write the same code in a handful of different ways, generating different ASTs for semantically equivalent code.

To attract internal clients, we're going to need to use a type system. Which one is still an open question, but internal teams are going to expect Closure annotations. Thus, even if we chose TypeScript or Flow, we'll still be generating Closure annotations from those formats to support most people in Piper.

Since we're going to need Closure annotations for all our public-facing APIs anyway, that's probably a reasonable boundary for diffing.

The Closure tool is written in Java. We could presumably create a CompilerPass that writes the AST as a JSON file that apidiff understands.

There's also a JavaScript tool that generates documentation from Closure annotations called JSDoc. I haven't dug into it deeply enough to see how it builds/stores its AST, but since Closure compatibility is essential, it might be better to use the Closure tool to generate our AST. If there are any deviations between Closure and JSDoc, we should stay on the Closure side.

I wonder if this is an area @MatrixFrog is interested in. Presumably, we're not the only team that would find an API diff tool for Closure-annotated code helpful.

[MatrixFrog]

I think @joeltine was looking at doing something similar.

[appsforartists]

+@jleyba

[appsforartists]

@tofuness You may find this interesting. Eventually, we'll want a tool that can spit out API changelogs comparing two releases. Click that link for example output from Objective-C, but effectively, we want to be able to compare two APIs:

write({ stream: MotionObservable, to: Property })

write({ from: MotionObservable, to: Property })

and see something like:

runtime.write

removed named parameter: stream
added named parameter: from

TypeScript's API will probably be helpful in this.

Is this something you'd want to take astab at?