Skip to content

Expose type dependencies (call graph) for accurate incremental transpiling #99

Open

Description

Is your feature request related to a problem? Please describe.
For accurate incremental transpiling, external tooling (Maven, Gradle, etc.) need to determine which files to re-transpile when a given file changes.
The mapping from source file to generated files is needed too (naming conventions are not enough, because it's not unusual for a Java file to contain multiple types, and can even contain several top-level types; and source files aren't even required to follow the common naming convention based on the package and type names) to accurately cleanup the output directory when a source file is deleted.
J2CL also copies source files (.java and .js) to the output, possibly relocating them, so the mapping from source path to output path is needed as well.

Even better if it could also map to JARs or directories in the -classpath for external dependencies, such that if the JAR (or class in the directory) changes the build tool could re-process all files that depend on it (and possibly on all following path entries in the classpath, in case class is now shadowed, or a class that was shadowed no longer is; if it could map to the actual class file, this could be made even more precise, by detecting those shadowing cases). Otherwise, each change to the classpath would mean reprocessing all files. Gradle, for instance, does such processing for its incremental javac.

For example, given those two source files:

package p;

class A {
  B a() {
    return new B(1);
  }
}

and

package p;

class B {
  B(Number n) {}
}

If src/p/B.java is modified, then we'll want to reprocess both src/p/B.java and src/p/A.java because p.B would have had its API modified in a way that would change p.A's output (e.g. here adding a B(int) constructor overload).

Processing those 2 files will result in 8 files being generated or copied:

  • out/p/A.java (copied from the sources)
  • out/p/A.java.js
  • out/p/A.impl.java.js
  • out/p/A.js.map
  • out/p/B.java (copied from the sources)
  • out/p/B.java.js
  • out/p/B.impl.java.js
  • out/p/B.js.map

If src/p/A.java is deleted, we'll want to delete all the out/p/A.* files.

Now if a src/p/A.native.js is created, then src/p/A.java would have to be reprocessed. And then again if src/p/A.native.js is deleted. But because J2CL uses a naming convention here, this does not need to appear in the mapping (correct me if I'm wrong).

If src/p/B.java is modified to add an inner class, it would generate 3 new files (out/p/B$Inner.java.js, out/p/B$Inner.impl.java.js, and out/p/B$Inner.js.map). If src/p/B.java is then deleted, then all 7 files need to be deleted, and src/p/A.java be reprocessed (which would fail, unless it was also modified to remove its dependency on p.B).

Describe the solution you'd like
Define a stable file format that J2clCommandLineRunner could output for use by external (non-Bazel) tooling.
It needs to contain 2 things:

  • the list of dependencies between source files (src/p/A.java depends on src/p/B.java; this could use types as an indirection, e.g. type p.A comes from src/p/A.java and references type p.B that comes from src/p/B.java, and foo.Bar that comes from jar:foo.jar!/foo/Bar.class)
  • the mapping back to the source file(s) (there could be a .java and a .native.js) for each output file

In an incremental use of the transpiler, the build tool would have to merge the generated mapping file with the previously known mapping (and removing entries from that global mapping when a file is deleted).

Describe alternatives you've considered
J2CL outputs a source map file (.js.map) for each generated file (pair of .java.js and .impl.java.js), containing the path of the source; but that references the source that J2CL copied to the output, not the actual path to the source (as can be seen in bazel-bin/third_party/jbox2d.js.zip vs bazel-bin/third_party/libjbox2d-src.jar after a bazel build //third_party:jbox2d.js.zip: the .js.zip contains a java/lang/StrictMath.js.map referencing, using a relative path, java/lang/StrictMath.java, but the actual source in the -src.jar was external/org_jbox2d/src/main/java/org/jbox2d/gwtemul/java/lang/StrictMath.java), because it's targeted at tools that need to access those .java files.

Additional context
J2CL internally already has all the information; it populates a LibraryInfo protobuf that it can serialize to a file (for later use by RTA) but the flag to do so is not exposed to J2clCommandLineRunner (probably to keep the LibraryInfo as an implementation detail of J2CL). Just like the source map though, the library_info does not contain the actual source path either (modifying third_party/BUILD to pass readable_library_info = True to //third_party:jbox2d generates a library_info_debug.json in the .js.zip but it does not contain the full src/main/java/org/jbox2d/gwtemul/java/lang/StrictMath.java path)
The Kythe indexing metadata includes the information though (I modified build_defs/internal_do_not_use/j2cl_common.bzl to unconditionally pass -generatekytheindexingmetadata), so J2CL has all the needed information to generate such a mapping file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions