Resolve config specific imports for Dart 2

[leafpetersen]

Config specific imports are only partially supported by Dart tools (notably the build tools). There are some limited existing uses of config specific imports, mostly for testing purposes. There are possibly planned uses in the future for Dart SASS.

This issue is for discussion of possible short term resolutions for Dart 2 to avoid shipping something that we will later regret. Options on the table:

• Fully unship the feature
  • pro: the feature as it exists is problematic
  • con: it fulfills a serious need
• Ship the feature as it is
  • pro: the feature is useful
  • con: many build tools do not and may not ever support it
• Move the feature back behind a flag a-la super-mixins with the intent of revisiting the feature in the next year (possibly with breaking changes)
  • pro: gives us license to make breaking changes
  • con: leaves the feature in an unfortunate heisen-state, users can't rely on it

@matanlurey @sigmundch Can you summarize on the issue why build tools can't support this?

@nex3 Can you comment on the Dart SASS requirements?

cc @zanderso @lrhn @anders-sandholm @vsmenon @munificent @a-siva

[leafpetersen]

Summarizing some discussion so far from elsewhere:

• Dart currently assumes that platform defines may only be available at runtime
• In order to do modular compilation with config specific imports, we need to know the platform defines
  • Unless we define an interface conformance check so that we can compile against default interfaces, which isn't happening any time soon

So questions I have are:

• Why is it infeasible to allow modular/AOT compilers to require a specific set of platform defines at compile time (with an implied link time check for consistency)?
  • We could still allow platform defines to be provided at runtime for tools that support it (e.g. the vm).
• Are there existing uses of platform defines which are incompatible with the above?
• If so, could we not make config specific imports use something other than general platform defines?

[nex3]

I have an in-progress document that defines semantics for interface conformance that's more thorough than the sketch that's in the original proposal, and that I hope will be easier to implement as well. If there's interest, I could flesh that out and send it around.

Can you comment on the Dart SASS requirements?

Dart Sass needs to be able to target three platforms with the same codebase: the Dart VM, Node, and browsers. Right now, we look at the dart.library.io and dart.library.js automatic variables to do so.

[sigmundch]

Can you summarize on the issue why build tools can't support this?

It comes down to the modularity points you made earlier: without conformance, build systems don't have a clear design:

• they could create build artifacts for different configurations, but that doesn't scale when combining configurations from dependencies (configuration explosion); or
• they could otherwise store configuration-specific information on each build artifact and validate consistency at link time, but that makes maintenance of large codebases really hard for developers. That is because issues may arise down the line when someone happens to import two packages that have inconsistent platform defines in their configurations.

I want to add another option on the table:

• ship the feature that is implemented in the CFE
  • pros: it is useful, it is implemented, and build tools can support this.
  • cons: it means making a last minute spec change (although small).

The CFE implementation has one extra limitation: only platform defines that are invariant in a platform are visible for config specific imports. That is, you can see only defines of the from dart.library.foo which indicate whether dart:foo is supported by the platform.

Like Dart-SASS, we believe most uses of config-specific imports are choosing between platforms, and hence only need this limited form. Note: dart2js defines a different platform when it compiles for the browser or for compiling for node (e.g. dart:html is only supported in the former)

Build systems can support this because they now only need to create a fixed number of build artifacts: one per platform.

[leafpetersen]

ship the feature that is implemented in the CFE

@zanderso @nex3 Presumably this covers both of your existing use cases?

@vsmenon Had one similar/alternative suggestion, which is that we could provide a way to redirect dart:foo libraries. So dart:html points to the real dart:html on some platforms and to an either platform defined or possibly even user defined shim library on others. Would this cover existing use cases?

[leafpetersen]

cc @grouma for comment on how much work it would be for the build system to support:

• locked down per platform configurations
• threading arbitrary build configurations through

[leafpetersen]

I have an in-progress document that defines semantics for interface conformance that's more thorough than the sketch that's in the original proposal, and that I hope will be easier to implement as well. If there's interest, I could flesh that out and send it around.

I'm personally quite interested in doing something around this at some point, but there's no way we can get it for Dart 2, and seems unlikely for the next release or two. But I'd definitely be happy to see work done on this to work towards some future release.

[matanlurey]

Thanks for filing and reviewing this @leafpetersen.

BTW, I do agree with @sigmundch that if we need to keep this in the current form then the feature already shipped and supported in the CFE is the right subset to support today.

To reiterate my reservations publicly:

• This feature, as it stands, feel very incomplete:

  • Static analysis is limited to the "default" import (analyzer), unlikely to change in short-term.
  • No guarantee that different switchable libraries are compliant or have the same API surface

• Modular build systems are unproven to support this feature (we have no examples of it working)

  • Bazel (might?) need different flavored summaries or kernel files.
  • I don't see any staffed effort to get modular flavored summaries/dills working anytime soon
  • As a result, this feature is unusable internally and might be unusable externally for web, as well

@nex3:

I have an in-progress document that defines semantics for interface conformance that's more thorough than the sketch that's in the original proposal, and that I hope will be easier to implement as well. If there's interest, I could flesh that out and send it around.

• And this leads to the BIGGEST ISSUE IMO: If we officially ship this feature, as-is, in Dart 2.0 stable, and its picked up by a large EAP client, are we going to want to refine the feature further in Dart 3.x, and do we risk being unable to ever improve configurable imports because we shipped an implementation that was meant for Dart 1.x with much different constraints (no build system, source VM, etc).

I think we should, in order of most preferred to least preferred:

• Put the feature, as implemented in the CFE, behind an experiment to unblock current users, but not block us from refining or changing the feature in the future (whether @nex3 or someone else wants to work on a concrete proposal).
• Remove the feature, and use an alternative (@vsmenon) method for the same functionality (i.e. make sure the two uses cases we know about today continue working by another method, but remove the feature otherwise).
• Ship the feature, as implemented in the CFE, removing the -d custom semantics entirely. This should mean that we've validated, tested, and documented that it works with (a) Flutter AOT, (b) Dart2JS on CFE with BuildRunner, (c) DDC on CFE with BuildRunner. If we aren't able to validate and test these use cases, we shouldn't ship it at all IMO.

[leafpetersen]

Put the feature, as implemented in the CFE, behind an experiment to unblock current users, but not block us from refining or changing the feature in the future

This reminds me of the other question: @nex3 for Dart SASS do you need to ship a package that relies on this feature, or do you just need this for building Dart SASS itself? That is, if we move this behind a flag, will we need to worry about end users having to set the flag in order to use important packages, or can we firewall this into code used to build tools?

[nex3]

@vsmenon Had one similar/alternative suggestion, which is that we could provide a way to redirect dart:foo libraries. So dart:html points to the real dart:html on some platforms and to an either platform defined or possibly even user defined shim library on others. Would this cover existing use cases?

This is pretty similar to my original proposal years ago, which was to provide stub libraries that just throw (like we currently do for dart:io). I like the idea in general, but I'm worried that getting a good design in place and implemented by Dart 2 is infeasible.

This reminds me of the other question: @nex3 for Dart SASS do you need to ship a package that relies on this feature, or do you just need this for building Dart SASS itself? That is, if we move this behind a flag, will we need to worry about end users having to set the flag in order to use important packages, or can we firewall this into code used to build tools?

The sass package on pub.dartlang.org does use configuration-specific imports; they aren't just build-time, because there's no build before publishing a VM package. So yes, it would be a pretty serious usability problem if we had to make downstream users of Sass in Dart toggle this flag. However, I don't think people using Sass from Dart that's then compiled to JS is a use-case we care a lot about

This suggests a possible solution: we could support the CFE subset, and put it behind a flag only for dart2js and DDC. The VM would be able to consume config-specific imports by default, but anyone using it in practice would have to opt into the experiment somewhere.

[grouma]

Discussed offline with Leaf to provide some context of the current platform support which is related to config specific imports.

Some notes:

• In our build rules we have the concept of platforms: https://github.com/dart-lang/rules_dart/blob/master/dart/build_rules/common/context.bzl#L55
• Currently this is used only internally to ensure compatible platforms between dart libraries
• In the near future we intend to make the internal analyzer platform aware, meaning we can adjust the SDK given the platform of the dart library
• Besides analysis, we do not change the behavior of builds based on the platform and have no current plans to do so
• As far as I know package:build has no concept of platforms and there is no work planned to provide platform support
• The pub site has basic platform support by showing the inferred support for a package. It uses https://pub.dartlang.org/packages/pana to do analysis.

[natebosch]

I'm in favor of keeping support hidden behind a flag so that we can delay the decision on what the final feature will look like. I think it's plausible we can support the "platform" specific import idea, but I'd really like to have at least a prototype working in build_runner, bazel, and flutter, before we decide to ship it as a fully supported feature. I think it's unlikely we'll have the bandwidth to build these prototypes in a reasonable timeframe.

[zanderso]

There are some limited existing uses of config specific imports, mostly for testing purposes.

To clarify a bit, the uses in Fuchsia are limited, but the alternatives to using them involve writing one or more new embeddings of the VM or doing a large refactoring of FIDL bindings generation.

@zanderso @nex3 Presumably this covers both of your existing use cases?

The current implementation in the CFE is working great for the Fuchsia use-case.

I would like to hear more details about how this feature would work when under a flag. What would accept the flag, and what would it mean?