"deno ast script.ts" #2355

ry · 2019-05-14T03:09:50Z

Should print to stdout a JSON blob representing the typescript AST.

kitsonk · 2019-05-14T03:52:46Z

The TypeScript AST is quite a different beast than something that is aligned to an ES AST (because the TypeScript parser is closer aligned to Roslyn than it is to what ES thinks of as the AST).

What is the use case, as that might determine things.

https://astexplorer.net/ is a good reference that would demonstrate what a dump would look like and includes the TypeScript AST as well as other parsers, and you can see that there is a big difference between a TypeScript AST and ES based ASTs like acorn.

ry · 2019-05-14T04:18:33Z

TS also can output AST for JS. (As far as I can remember.)

I want this because

It's useful for generating docs.
It invites cool projects.
We have the ability to do this with relative ease, so we might as well surface it.

kitsonk · 2019-05-14T04:31:41Z

It can, but it is the Roslyn-like AST, not compatible with the traditional ES ASTs.

For example, given this code:

/**
 * Example of JSDOC
 *
 * @param foo Something foo
 */
function bar(foo) {
  console.log(foo);
}

bar("hello world");

You would get the following from TypeScript:

{
  "pos": 0,
  "end": 122,
  "flags": 0,
  "kind": 279,
  "text": "/**\n * Example of JSDOC\n *\n * @param foo Something foo\n */\nfunction bar(foo) {\n  console.log(foo);\n}\n\nbar(\"hello world\");\n",
  "bindDiagnostics": [],
  "languageVersion": 6,
  "fileName": "astExplorer.tsx",
  "languageVariant": 1,
  "isDeclarationFile": false,
  "scriptKind": 4,
  "pragmas": {},
  "referencedFiles": [],
  "typeReferenceDirectives": [],
  "libReferenceDirectives": [],
  "amdDependencies": [],
  "hasNoDefaultLib": false,
  "statements": [
    {
      "pos": 0,
      "end": 100,
      "flags": 0,
      "parent": "[Circular ~]",
      "kind": 239,
      "jsDoc": [
        {
          "pos": 0,
          "end": 58,
          "flags": 0,
          "parent": "[Circular ~.statements.0]",
          "kind": 291,
          "tags": [
            {
              "pos": 30,
              "end": 56,
              "flags": 0,
              "parent": "[Circular ~.statements.0.jsDoc.0]",
              "kind": 299,
              "tagName": {
                "pos": 31,
                "end": 36,
                "flags": 0,
                "parent": "[Circular ~.statements.0.jsDoc.0.tags.0]",
                "escapedText": "param"
              },
              "name": {
                "pos": 37,
                "end": 40,
                "flags": 0,
                "parent": "[Circular ~.statements.0.jsDoc.0.tags.0]",
                "escapedText": "foo"
              },
              "isNameFirst": true,
              "isBracketed": false,
              "comment": "Something foo"
            }
          ],
          "comment": "Example of JSDOC"
        }
      ],
      "modifierFlagsCache": 536870912,
      "name": {
        "pos": 67,
        "end": 71,
        "flags": 0,
        "parent": "[Circular ~.statements.0]",
        "escapedText": "bar"
      },
      "parameters": [
        {
          "pos": 72,
          "end": 75,
          "flags": 0,
          "parent": "[Circular ~.statements.0]",
          "kind": 151,
          "name": {
            "pos": 72,
            "end": 75,
            "flags": 0,
            "parent": "[Circular ~.statements.0.parameters.0]",
            "escapedText": "foo"
          }
        }
      ],
      "body": {
        "pos": 76,
        "end": 100,
        "flags": 0,
        "parent": "[Circular ~.statements.0]",
        "kind": 218,
        "multiLine": true,
        "statements": [
          {
            "pos": 78,
            "end": 98,
            "flags": 0,
            "parent": "[Circular ~.statements.0.body]",
            "kind": 221,
            "expression": {
              "pos": 78,
              "end": 97,
              "flags": 0,
              "parent": "[Circular ~.statements.0.body.statements.0]",
              "kind": 191,
              "expression": {
                "pos": 78,
                "end": 92,
                "flags": 0,
                "parent": "[Circular ~.statements.0.body.statements.0.expression]",
                "kind": 189,
                "expression": {
                  "pos": 78,
                  "end": 88,
                  "flags": 0,
                  "parent": "[Circular ~.statements.0.body.statements.0.expression.expression]",
                  "escapedText": "console"
                },
                "name": {
                  "pos": 89,
                  "end": 92,
                  "flags": 0,
                  "parent": "[Circular ~.statements.0.body.statements.0.expression.expression]",
                  "escapedText": "log"
                }
              },
              "arguments": [
                {
                  "pos": 93,
                  "end": 96,
                  "flags": 0,
                  "parent": "[Circular ~.statements.0.body.statements.0.expression]",
                  "escapedText": "foo"
                }
              ]
            }
          }
        ]
      }
    },
    {
      "pos": 100,
      "end": 121,
      "flags": 0,
      "parent": "[Circular ~]",
      "kind": 221,
      "expression": {
        "pos": 100,
        "end": 120,
        "flags": 0,
        "parent": "[Circular ~.statements.1]",
        "kind": 191,
        "expression": {
          "pos": 100,
          "end": 105,
          "flags": 0,
          "parent": "[Circular ~.statements.1.expression]",
          "escapedText": "bar"
        },
        "arguments": [
          {
            "pos": 106,
            "end": 119,
            "flags": 0,
            "parent": "[Circular ~.statements.1.expression]",
            "kind": 10,
            "text": "hello world"
          }
        ]
      },
      "modifierFlagsCache": 536870912
    }
  ],
  "endOfFileToken": {
    "pos": 121,
    "end": 122,
    "flags": 0,
    "parent": "[Circular ~]",
    "kind": 1
  },
  "nodeCount": 22,
  "identifierCount": 6,
  "identifiers": {},
  "parseDiagnostics": [],
  "path": "astExplorer.tsx",
  "resolvedPath": "astExplorer.tsx",
  "originalFileName": "astExplorer.tsx",
  "imports": [],
  "moduleAugmentations": [],
  "ambientModuleNames": []
}

Where as you would get the following from acorn:

{
  "type": "Program",
  "start": 0,
  "end": 122,
  "body": [
    {
      "type": "FunctionDeclaration",
      "start": 59,
      "end": 100,
      "id": {
        "type": "Identifier",
        "start": 68,
        "end": 71,
        "name": "bar"
      },
      "expression": false,
      "generator": false,
      "params": [
        {
          "type": "Identifier",
          "start": 72,
          "end": 75,
          "name": "foo"
        }
      ],
      "body": {
        "type": "BlockStatement",
        "start": 77,
        "end": 100,
        "body": [
          {
            "type": "ExpressionStatement",
            "start": 81,
            "end": 98,
            "expression": {
              "type": "CallExpression",
              "start": 81,
              "end": 97,
              "callee": {
                "type": "MemberExpression",
                "start": 81,
                "end": 92,
                "object": {
                  "type": "Identifier",
                  "start": 81,
                  "end": 88,
                  "name": "console"
                },
                "property": {
                  "type": "Identifier",
                  "start": 89,
                  "end": 92,
                  "name": "log"
                },
                "computed": false
              },
              "arguments": [
                {
                  "type": "Identifier",
                  "start": 93,
                  "end": 96,
                  "name": "foo"
                }
              ]
            }
          }
        ]
      }
    },
    {
      "type": "ExpressionStatement",
      "start": 102,
      "end": 121,
      "expression": {
        "type": "CallExpression",
        "start": 102,
        "end": 120,
        "callee": {
          "type": "Identifier",
          "start": 102,
          "end": 105,
          "name": "bar"
        },
        "arguments": [
          {
            "type": "Literal",
            "start": 106,
            "end": 119,
            "value": "hello world",
            "raw": "\"hello world\""
          }
        ]
      }
    }
  ],
  "sourceType": "module"
}

kitsonk · 2019-05-14T04:37:09Z

For the record, I think it is a good idea.

The other bit, is that the compiler has a decent API for digesting the JSDoc, but it is a side loading thing where you have to query the trivia on each node to see if there is some parsed JSDoc. So I think there are some enhancements for the use cases above to improve on the structure of the AST for documentation generation purposes, or instead of really focusing on a "pure" AST, we actually walk the AST, outputting a JSON "documentation" structure, worrying about the "surface" of the modules integrated with the inline documentation, instead just dumping loads of internals that won't really be useful.

ry · 2019-05-14T04:47:06Z

The TS AST looks great to me.

axetroy · 2019-05-14T16:26:15Z

TS AST no need to compatible with ES AST.

and consider adding AST API for deno? or maybe put it into deno_std is better.

jamiebuilds · 2019-10-01T04:24:47Z

ESTree is a little bit more of a shared standard and the TypeScript ESLint team has a node package that can generate a TypeScript+JSX extension of it which also exports type definitions for all of the AST nodes.

kitsonk · 2019-10-01T04:45:02Z

Yeah, I am still torn a bit. Ry and I have gone back and forth a few times offline on this as well, related to how we handle the transpilation. Our chats caused me to raise microsoft/TypeScript#33502 as the biggest problem is that while the circular references in the TypeScript AST are possible, there is no easy way to "rehydrate" that AST to be leveraged with the compiler, which is really what something like this would need to be useful.

While ESTree-like ASTs are certainly more common, some potential for reversibility would be critical to this feature, that at some point we could ingest an AST and feed it to the compiler to emit. I can't see that path easy with going down the ESTree-like route.

jamiebuilds · 2019-10-02T14:26:37Z

If you would like to explore maintaining a AST-to-AST converter, I can describe the challenges and strategies for making it easier to do. I’ve maintained/contributed to several similar tools.

kitsonk · 2020-04-18T11:01:33Z

No... there is a big difference between the doc output and an AST output.

bartlomieju · 2020-04-18T11:04:36Z

If we really want this subcommand I can add it in the evening, should be 15-minute work...

@ry thoughts?

kitsonk · 2020-04-18T11:07:28Z

Personally I think it is valuable, especially if we are referring to swc's AST, which I assume we are. It is estree like and there is a fair amount of tooling to be able to do things with it. It would be more useful exposed as a runtime API, IMO, but still useful as a subcommand.

bartlomieju · 2020-04-18T11:08:52Z

@kitsonk, yeah, that'd be SWC AST. As for runtime API - sure, but we need to fix a few issue before that (#4781)

ry · 2020-04-18T16:37:50Z

I'm a bit worried about having to maintain a stable output...

I think it's definitely useful as an internal utility - and I thought it might be useful for external people... but not sure.

kitsonk · 2020-04-18T21:26:54Z

My advice, let's keep it open for now, and if there is a compelling use case down the road, we do it. There are a couple things on my head but won't have time in the near future.

axetroy · 2020-04-19T02:05:02Z

I think it would be better to expose the APIs of Typescript

const { typescript } = Deno

const sourceFile = typescript.createSourceFile("script.ts", "// typescript content")

// then do whatever you what

kitsonk · 2020-04-19T03:08:38Z

@axetroy the compiler sits in another worker... Our structure cloning isn't compliant at the moment, but even then, it would get very expensive very quickly to sync up heavily between the two workers like that. We need to always keep the APIs narrowed and focused between the two.

littledivy · 2020-07-24T10:07:06Z

This would be a great feature/utility imo. Perhaps, using swc ast parser would be sufficient.

For Example: https://github.com/nestdotland/deno_swc/blob/master/examples/parse.ts

I'd be happy to open a PR for this.

bartlomieju · 2020-07-24T11:07:08Z

There's really nothing stopping us from exposing deno ast --unstable <file> subcommand. There have been no major changes in the swc_ecma_ast implementation in the past couple months (sans support for new TypeScript features). @ry what do you think?

bartlomieju · 2020-10-30T12:24:06Z

After more discussion we think that this functionality should be provided by the user land libraries - namely there's deno_swc that provides access to SWC APIs using WASM.

dsherret · 2020-11-08T17:19:19Z

@bartlomieju that's the right decision. It's better for consumers to be able to run any version of deno while picking or sticking to a version of swc to use (avoids: "I can't upgrade deno yet because my code is not compatible with the latest AST changes"/"this code doesn't work in deno [old version goes here]")

bartlomieju · 2020-11-08T17:21:18Z

@bartlomieju that's the right decision. It's better for consumers to be able to run any version of deno while picking or sticking to a version of swc to use (avoids: "I can't upgrade deno yet because my code is not compatible with the latest AST changes"/"this code doesn't work in deno [old version goes here]")

As well as it's a lot less code to maintain in Deno repo - the typings alone for SWC are several thousand lines.

kitsonk mentioned this issue Sep 25, 2019

"deno doc lib.ts" #3021

Closed

kitsonk mentioned this issue Mar 24, 2020

Typescript Custom Transformers Support #3354

Closed

This comment has been minimized.

Sign in to view

littledivy mentioned this issue Jul 24, 2020

feat: Add "deno ast" subcommand #6867

Closed

bartlomieju closed this as completed Oct 30, 2020

bartlomieju mentioned this issue Nov 8, 2020

Use swc to expose an AST to the runtime #3983

Closed

shadowtime2000 mentioned this issue Jan 3, 2021

Plugin env alephjs/aleph.js#77

Closed

kitsonk mentioned this issue Dec 30, 2021

Native Deno typescript parser api #13237

Closed

ghost mentioned this issue Sep 16, 2022

[Feature Request] Replace phantomjs with something else? yt-dlp/yt-dlp#1354

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"deno ast script.ts" #2355

"deno ast script.ts" #2355

ry commented May 14, 2019

kitsonk commented May 14, 2019

ry commented May 14, 2019

kitsonk commented May 14, 2019

kitsonk commented May 14, 2019

ry commented May 14, 2019

axetroy commented May 14, 2019

jamiebuilds commented Oct 1, 2019 •

edited

Loading

kitsonk commented Oct 1, 2019

jamiebuilds commented Oct 2, 2019

This comment has been minimized.

kitsonk commented Apr 18, 2020

bartlomieju commented Apr 18, 2020

kitsonk commented Apr 18, 2020

bartlomieju commented Apr 18, 2020

ry commented Apr 18, 2020

kitsonk commented Apr 18, 2020

axetroy commented Apr 19, 2020 •

edited

Loading

kitsonk commented Apr 19, 2020

littledivy commented Jul 24, 2020 •

edited

Loading

bartlomieju commented Jul 24, 2020

bartlomieju commented Oct 30, 2020

dsherret commented Nov 8, 2020 •

edited

Loading

bartlomieju commented Nov 8, 2020

"deno ast script.ts" #2355

"deno ast script.ts" #2355

Comments

ry commented May 14, 2019

kitsonk commented May 14, 2019

ry commented May 14, 2019

kitsonk commented May 14, 2019

kitsonk commented May 14, 2019

ry commented May 14, 2019

axetroy commented May 14, 2019

jamiebuilds commented Oct 1, 2019 • edited Loading

kitsonk commented Oct 1, 2019

jamiebuilds commented Oct 2, 2019

This comment has been minimized.

kitsonk commented Apr 18, 2020

bartlomieju commented Apr 18, 2020

kitsonk commented Apr 18, 2020

bartlomieju commented Apr 18, 2020

ry commented Apr 18, 2020

kitsonk commented Apr 18, 2020

axetroy commented Apr 19, 2020 • edited Loading

kitsonk commented Apr 19, 2020

littledivy commented Jul 24, 2020 • edited Loading

bartlomieju commented Jul 24, 2020

bartlomieju commented Oct 30, 2020

dsherret commented Nov 8, 2020 • edited Loading

bartlomieju commented Nov 8, 2020

jamiebuilds commented Oct 1, 2019 •

edited

Loading

axetroy commented Apr 19, 2020 •

edited

Loading

littledivy commented Jul 24, 2020 •

edited

Loading

dsherret commented Nov 8, 2020 •

edited

Loading