Speed up TypeScript compilation for iteration

[samreid]

@jessegreenberg reported in slack:

So far I have been using tsc -b --watch in chipper/tsconfig/all, but compilation time there has gone up to about 30 seconds. I will start a watch process just in the sim I am testing in from now on.

We discussed that changes to Property.ts compile quickly, but changes to Sim.js constructor take 30-60 seconds for @jessegreenberg. We would like to find a faster way.

To investigate:

• Running babel as a WebStorm file watcher
• Running a babel watch process
• Running swc
• Writing our own script that layers into a watch process

TypeScript describes babel integration here: https://www.typescriptlang.org/docs/handbook/babel-with-typescript.html. However, we currently have constraints about how and where the files are output (chipper dist with correct relative paths and *.js suffixes, and *.js files should be output too), so not everything is expected to work right out of the box.

I'm already seeing that babel doesn't copy *.js files by default, and does not change suffix if using --out-file in a file watcher.

Also, to support paralllel file outputs, we will likely need to export type definitions separately, which can be checked and enforced via "isolatedModules": true. We can also discuss whether we will allow tsc to output files or just type check.

Given the constraints above, I'm considering that we will need our own script that takes a file and transpiles it into the right spot using babel. We can then combine that with WebStorm file watcher or chokidar. I also haven't looked into swc yet--it may work right out of the box.

[samreid]

@jonathanolson and I reviewed the proposal on my working copy and agreed it seems a good direction to go in, so I committed the current version.

Next steps:

• Follow https://www.typescriptlang.org/docs/handbook/babel-with-typescript.html and add:

"compilerOptions": {
  // Ensure that .d.ts files are created by tsc, but not .js files
  "declaration": true,
  "emitDeclarationOnly": true,
  // Ensure that Babel can safely transpile files in the TypeScript project
  "isolatedModules": true
}

This will require updating the imports on every sim and I'll wait until @jonathanolson has pushed changes for the large scenery refactor to do so, since it will modify imports for many repos.

• Add the new process to the build steps
• Add the new process to CT
• Add the new process to phet-server
• When/where to run tsc for type checking?
• Code review for the new process
• Test the new process on windows
• Inform the team how to use the new process
• Update READMEs
• Update the Development Overview
• Do sourcemaps still work?

[samreid]

I tried watching active-repos to update the watch paths, but if I add in js, images and sounds, then watching active-repos fails. I'm probably hitting some limit, but it doesn't give me confidence in chokidar.

Index: transpile/transpile-all.js
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/transpile/transpile-all.js b/transpile/transpile-all.js
--- a/transpile/transpile-all.js	(revision 08fe7a06aae894b59e824856b6a1a999943b82ac)
+++ b/transpile/transpile-all.js	(date 1637975934584)
@@ -9,8 +9,6 @@
  *
  * OPTIONS:
  * --watch                continue watching all directories and transpile on detected changes
- * --ignoreInitial false  (default) transpile all files on startup
- * --ignoreInitial true   skip initial transpile on startup
  * --clean                dispose of the cache that tracks file status on startup, can be combined with other commands
  *                        you would need to run --clean if the transpiled file status ever gets out of sync with the
  *                        cache file status, for example if you deleted/modified a transpiled file
@@ -48,40 +46,56 @@
   fs.writeFileSync( statusPath, JSON.stringify( status, null, 2 ) );
 }
 
-let ignoreInitial = false;
-const index = args.indexOf( '--ignoreInitial' );
-if ( index >= 0 ) {
-  if ( args[ index + 1 ] === 'true' ) { ignoreInitial = true;}
-  else if ( args[ index + 1 ] === 'false' ) { ignoreInitial = false;}
-  else {throw new Error( 'illegal value for ignoreInitial' );}
-}
-
-const activeRepos = fs.readFileSync( '../perennial/data/active-repos', 'utf-8' ).trim().split( '\n' );
+const ACTIVE_REPOS_PATH = '../perennial/data/active-repos';
+const activeRepos = fs.readFileSync( ACTIVE_REPOS_PATH, 'utf-8' ).trim().split( '\n' );
+const watchedRepos = [ ...activeRepos ];
 const paths = [];
+paths.push( '../perennial/data/active-repos' );
 activeRepos.forEach( repo => {
   paths.push( `../${repo}/js` );
-  paths.push( `../${repo}/images` );
   paths.push( `../${repo}/sounds` );
+  // paths.push( `../${repo}/images` );
 } );
 
+let count = 0;
+let ready = false;
 chokidar.watch( paths, {
-  ignoreInitial: ignoreInitial,
-  ignored: [ '**/chipper/dist/**/' ]
+  ignoreInitial: true,
+  // ignored: [ '**/node_modules/**', '**/chipper/dist/**', '**/.git/**' ]
+  // ignored: path => path.includes( 'node_modules' )
 } ).on( 'all', ( event, path ) => {
-  if ( path.endsWith( '.js' ) || path.endsWith( '.ts' ) ) {
-    const changeDetectedTime = Date.now();
-    const text = fs.readFileSync( path, 'utf-8' );
-    const hash = crypto.createHash( 'md5' ).update( text ).digest( 'hex' );
-
-    if ( !status[ path ] || status[ path ].md5 !== hash ) {
-      status[ path ] = { md5: hash };
-      transpileFunction( path, text );
-      fs.writeFileSync( statusPath, JSON.stringify( status, null, 2 ) );
-      const elapsed = Date.now() - changeDetectedTime;
-      console.log( elapsed + 'ms: ' + path );
-    }
+  count++;
+  // console.log( path );
+  if ( count % 1000 === 0 ) {
+    console.log( count );
+  }
+  // console.log( path );
+  // if ( path.endsWith( '.js' ) || path.endsWith( '.ts' ) ) {
+  //   const changeDetectedTime = Date.now();
+  //   const text = fs.readFileSync( path, 'utf-8' );
+  //   const hash = crypto.createHash( 'md5' ).update( text ).digest( 'hex' );
+  //
+  //   if ( !status[ path ] || status[ path ].md5 !== hash ) {
+  //     status[ path ] = { md5: hash };
+  //     transpileFunction( path, text );
+  //     fs.writeFileSync( statusPath, JSON.stringify( status, null, 2 ) );
+  //     const elapsed = Date.now() - changeDetectedTime;
+  //     console.log( elapsed + 'ms: ' + path );
+  //   }
+  // }
+  // else if ( path === ACTIVE_REPOS_PATH ) {
+  //   console.log( path, ACTIVE_REPOS_PATH );
+  //   const newRepos = fs.readFileSync( ACTIVE_REPOS_PATH, 'utf-8' ).trim().split( '\n' );
+  //   const toRemove = watchedRepos.filter( repo => !newRepos.includes( repo ) );
+  //   const toAdd = newRepos.filter( repo => !watchedRepos.includes( repo ) );
+  //   console.log( 'removing: ' + toRemove.join( ', ' ) + ', adding: ' + toAdd.join( ', ' ) );
+  // }
+
+  if ( ready ) {
+    console.log( path );
   }
 } ).on( 'ready', () => {
+  ready = true;
   if ( !args.includes( '--watch' ) ) {
     console.log( 'Finished transpilation in ' + ( Date.now() - start ) + 'ms' );
     process.exit( 0 );

[samreid]

I updated the other documentation in the commit, @pixelzoom can you please review?

[pixelzoom]

typescript-quick-start.md errors and questions:

6. Turn on TypeScript support in WebStorm: Preferences > Languages & Frameworks > TypeScript. Sublime also has an officially-supported plugin.

My understanding is that this is no longer an option, much less a required "Getting Started" step.

1. Compile the source and its dependencies via grunt output-js-project. This compiles the sim and its dependencies to chipper/dist/. It uses "Project References" (tsc --build) to trace the dependencies.
   ...
2. Compile via grunt output-js-project and run it in the browser. Did it print 7?

Does this really use tsc --build. It looks like it's using transpiler.js.

Why not just replace occurrences of grunt output-js-project with cd chipper; node node js/scripts/transpile.js? The node task is what's used in the proposed README.md steps in #1157.

1. TypeScript sims need to be compiled using grunt output-js-project before generating their PhET-iO API using grunt generate-phet-io-api.

This is error prone. Should grunt generate-phet-io-api be responsible for running grunt output-js-project?

2. Please do not convert common code to TypeScript until that phase is approved by the team. Some common code repos have a "typescript" branch for investigation in the meantime.

My understanding is TS in common code is OK; we're past this "gate".

8. Gravity and Orbits, Bending Light, and Circuit Construction Kit Common have been written in TypeScript,

The set of TS sims is larger now.

Back to @samreid.

[samreid]

Thanks, I updated the quick start guide more thoroughly.

My understanding is that this is no longer an option, much less a required "Getting Started" step.

I revised that part to show how to turn on the language service, but turn off the compilation.

This is error prone. Should grunt generate-phet-io-api be responsible for running grunt output-js-project?

It has been changed to autorun the transpiler.

The set of TS sims is larger now.

I removed the list altogether. Users can search for *.ts filenames if desired.

Ready for re-review.

[pixelzoom]

Turn on TypeScript support in WebStorm: Preferences > Languages & Frameworks > TypeScript. Make sure you are using your system's absolute path for chipper/node_modules/typescript, turn on "TypeScript language service" and turn on "Recompile on Changes". Turn off "Recompile on changes".

Read this again.

[samreid]

Oops, was a typo, should read:

Turn on "TypeScript language service", turn on "Show project errors", and turn off "Recompile on changes".