Tool support for passing in pre-built macro binaries

[jakemac53]

For running internally at least, we will want the ability to pass in pre-built binaries for macros to all tools, instead of having them do their own compilation of macros. In particular, the source files for macros would ideally not even be made available, only their summary/outline files, which means that compilation is not possible.

Note that this only applies to tools which are directly invoked by the build rules internally, we don't need to pre-emptively add support beyond that for now. But we will likely need API support from the analyzer/cfe to support this.

We should align on the command line arguments for this across tools so that we can integrate with them more easily and have consistency. I propose the following arguments, based largely what the frontend_server already supports:

--precompiled-macro

Informs the tool of the binary to be used for all macros defined in a given library.

Usage: --precompiled-macro=<absolute-path-to-binary>;<macro-library-uri-1>;<macro-library-uri-2>

Example: -precompiled-macro="/path/to/compiled/macro;package:some_macro/some_macro.dart

This would be a multi-option so you can pass as many of these as you need.

See existing option configuration here. This is the same but supporting multiple macro libraries, and swapping the order to support that better.

[Optional] --precompiled-macro-format

For now, only AOT mode needs to be supported as this is what initial prototypes will use.

Informs the tool of the type of binary that is provided (kernel file and aot binary are supported for now).

Usage: --precompiled-macro-format=<aot|kernel>

Example: --precompiled-macro-format=aot

Default: aot

This is a single option, all precompiled macros must use the same format.

See existing option configuration here.

[Optional] --macro-serialization-mode

The frontend_server does support a mode for customizing whether JSON or a custom byte data format is used, but the JSON format is really only for debugging purposes. I don't think we need to pre-emptively add support for this.

[Optional] --macro-communication-channel

Informs the tool of which communication channel to use when talking to a macros.

Usage: --macro-communication-channel=<socket|stdio>

Example: --macro-communication-channel=stdio

Default: socket

This is a single option, all precompiled macros must use the same format.

Tracking:

• analyzer worker
• analyzer based codegen rules
• kernel worker

[jakemac53]

cc @emmanuel-p for FYI regarding internal build rules (no work required for now but I may ping you)
cc @scheglov as I will likely need some work from the analyzer to expose the appropriate API
cc @johnniwinther for FYI on the CFE side of things. I may not need work here, the frontend_server integration may have already exposed sufficient API.

[scheglov]

Currently, in the analyzer we can have more than one library with macros inside the same library cycle, so we build them all into a single binary. This is supported by bootstrapMacroIsolate. So, maybe we should change --precompiled-macro to --precompiled-macro=<absolute-path-to-binary>;<macro-library-uri-1>;<macro-library-uri-2>;etc? So, we would be able to start them all together.

For integration into google3, I think we will do the same as for summaries, which we pass as externalSummaries argument to AnalysisDriver. Maybe Map<Uint8List, List<Uri>> externalMacros. And internal summaryAnalysisSession will pass it, as it passes summaries.

[jakemac53]

So, maybe we should change --precompiled-macro to --precompiled-macro=<absolute-path-to-binary>;<macro-library-uri-1>;<macro-library-uri-2>;etc? So, we would be able to start them all together.

Sure, that seems reasonable to me, I updated the proposal above to reflect that. For now at least in google3 I doubt we will utilize it (because of the nature of blaze it will be a lot more intuitive to create a macro binary for each possible macro). But we could do something later on where we had more global control over macros and created shared binaries.

[jakemac53]

Another question that comes up here is what the URIs should look like for non-package imports, especially in the bazel/build_runner case. Ideally we would use a custom scheme to specify the build-root relative URI (ie: org-dartlang-app or bazel-root scheme URIs etc), but I am not sure if that will work or not.

When the CFE/Analyzer ask to instantiate a macro, can they use these abstract URIs instead of the actual root file system URI they correspond to?

[scheglov]

I don't remember all the details of google3 integration, but we use something named ExplicitSourceResolver(Map<Uri, File> uriToFileMap), maybe to support non-package URIs. We already need to analyze somehow code outside of lib/. I guess that google3 people will have better answer on this.

[jakemac53]

cc @emmanuel-p who might have some insight

[jakemac53]

Everything from the CFE/Analyzer perspective looks to be in place now. I still need to hook things up for analyzer based codegen actions internally, but that shouldn't require any upstream work.

[jakemac53]

Note that gen_kernel.dart should also be updated for this