[flutter_plugin_tools] Add Android native UI test support

[stuartmorgan-g]

Adds integration test support for Android to native-test. Also fixes
an issue where the existing unit test support was not honoring
--no-unit.

Updates all plugins to annotate the integration_test hook test so that it
can be skipped when running in this mode.

Fixes flutter/flutter#86490

Pre-launch Checklist

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• I read the Tree Hygiene wiki page, which explains my responsibilities.
• I read and followed the relevant style guides and ran the auto-formatter. (Note that unlike the flutter/flutter repo, the flutter/plugins repo does use dart format.)
• I signed the CLA.
• The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
• I listed at least one issue that this PR fixes in the description above.
• I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
• I updated CHANGELOG.md to add a description of the change.
• I updated/added relevant documentation (doc comments with ///).
• I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test exempt.
• All existing and new tests are passing.

[bparrishMines]

Would the integration tests actually finish if one of the tests is ran with @RunWith(FlutterTestRunner.class)? I thinkFlutterTestRunner would prevent the test from finishing since it waits for a callback from Dart when integration tests are done. And the default target would be main.dart.

I think this command would also need a target option as well.

[stuartmorgan-g]

Would the integration tests actually finish if one of the tests is ran with @RunWith(FlutterTestRunner.class)?

I just did a sanity test that it seemed to run tests in a plugin I tried, so that may well be the case. I guess this is one of the issues you were referring to when you said we needed a way to match .java files and the intended Dart test?

Is there a way we could segregate tests that are using integration_test and those that are simply Android UI tests int separate targets, so we could only drive the latter?

I think this command would also need a target option as well.

The intention of this command is specifically to run the Java integration tests (not the Dart integration tests), for which I would expect main.dart to be the right target (that's how iOS works certainly).

But I'm starting to wonder if thinking about this as two completely distinct groups of tests is not the right mental model. Do we have tests that blend Java and Dart test driving?

[bparrishMines]

Is there a way we could segregate tests that are using integration_test and those that are simply Android UI tests int separate targets, so we could only drive the latter?

It looks like there is a way to specify which Java tests to run: https://developer.android.com/studio/test/command-line#RunTestsDevice

We could go in the direction of having a standard java test file that uses FlutterTestRunner. (e.g. DartIntegrationTest.java). And then create separate commands for that test and the other Java UI tests. The link above shows you can specify the package when running instrumented tests, but there are alot more steps than running ./gradlew app:connectedAndroidTest.

Do we have tests that blend Java and Dart test driving?

What do you mean by this? As in you run the Dart integration tests & Java integration tests in the same run?

[stuartmorgan-g]

Do we have tests that blend Java and Dart test driving?

What do you mean by this? As in you run the Dart integration tests & Java integration tests in the same run?

I mean something like: a Dart widget test that does steps A, B, and C of a test via Dart things, then control passes to the native side somehow in order to do steps D, E, and F on native elements using Espresso. Something that would mean we can't consider Dart integration tests and Java integration tests to be completely separate.

[blasten]

./gradlew app:connectedAndroidTest definitely needs the -Ptarget flag. https://github.com/flutter/flutter/tree/master/packages/integration_test#android-device-testing

I mean something like: a Dart widget test that does steps A, B, and C of a test via Dart things, then control passes to the native side somehow in order to do steps D, E, and F on native elements using Espresso.

I see. I think this translates into filtering the tests by test runner. e.g. Only tests meant to run with AndroidJUnitRunner, then another filter for FlutterTestRunner. This needs to go a level higher, and look for JUnit flags. I don't know the answer, but I found this: https://blog.jdriven.com/2017/10/run-one-or-exclude-one-test-with-gradle/

I could try to look into it in a bit.

[stuartmorgan-g]

Actually, the simplest solution would be to modify FlutterTestRunner so it skips all the Dart tests when the -Ptarget flag is missing. Would this handle the use case?

If it internally no-oped that would probably work, but my feeling is that we should explore the filtering option first; deliberately building a mode where tests silently no-op worries me a lot given how bad a failure mode "silently doesn't run" is (as we keep learning in this repo).

[bparrishMines]

Would it be possible to move the pure Java integration tests into the android module instead? As in, they go in webview_flutter/android/src/androidTest/..... And then Dart integration tests will be done in the example folder.

I was close to converting the webview_flutter as an example, but I ran into a couple problems. Will probably play work on it a bit more if this is a viable route.

[stuartmorgan-g]

Don't the native UI tests need to be run in the context of an application though?

[stuartmorgan-g]

@blasten @bparrishMines This is ready for another review; I've addressed the issue with running FlutterTestRunner tests.

I tried doing it with build variants per our discussion offline, but ran into issues:

• Build modes don't work for this AFAICT (I tried make an offshoot of debug) because tests are only built for a single mode, rather than all modes or a specific mode chosen at build time. Seems like an odd limitation, but it turns out it is documented.
• Flavors probably work, but the ergonomics of it is very poor for an example app. Flutter has no concept of a default flavor (which maybe should be addressed?) so once Android flavors are added it's impossible to run the app without --flavor <some flavor>. We would have to document that for every plugin, and hope people noticed (many wouldn't), and running from IDEs would be much harder. That's just not viable IMO.

So after that I explored the filtering option, and got that working. I considered:

• Name-based: Seems error-prone, since we'd have to always name them exactly the right thing. It should work though.
• Package-based: Doesn't accept wildcards, which makes it very awkward; we'd have to construct the filter string dynamically based on path inspection.
• Annotation-based: Ideally we'd filter based on the RunWith(FlutterTestRunner annotation itself, but I couldn't find a way to use annotation arguments in the filter. It works with a new custom annotation though.

The way I've gone here is the last option, since it gives us a clear, explicit way to mark tests to skip in this mode. The annoying part is duplicating the same tiny file to declare the annotation in every plugin. If we agree this is the approach to go with, I think we should strongly consider adding this annotation to integration_test, so that eventually (once that reaches stable) we could switch our plugins over to it and remove these copies. That would also make it easier for third parties to adopt the pattern.

[bparrishMines]

I agree that this is probably the simplest and easiest approach out of the ones that you listed. Could you make a quick issue about adding DartIntegrationTests to integration_test after this lands?

[bparrishMines]

LGTM

[stuartmorgan-g]

submit-queue is incorrectly showing red here; the tree is green.