Description
Currently the Pigeon testing is using unit tests on the Dart and native sides; this is useful, but leaves open the potential for unit tests on each side to pass, but for the protocol being unit tested to mis-align slightly such that the end-to-end call doesn't actually work. #111083 is an example of a case that should have been trivially caught by basic integration tests, but has to be more explicitly tested correctly with unit tests based on details of the serialization format on each end. (We also don't have that unit test on the Swift side, which is a problem, but a simple test would be even better.)
I would suggest that we rework the platform_tests
directory so that instead of having a combination (depending on language) of bespoke tests and per-language dummy plugins, we just have one single test_plugin
plugin that supports every platform/language (or I guess, two plugins: one for ObjC and Java, and one for everything else, since they can't really co-exist with their alternate languages in the same plugin). That plugin would have:
- Unit tests for each language, using exactly the same structure that native unit tests in flutter/plugins use. That's already how Windows works, and the others could be converted.
integration_test
tests that would drive full end-to-end calls for a variety of core use cases.
(Eventually I would like to revisit using the plugin repo tooling to drive all of this; I temporarily abandoned that when setting up Windows because it would involve more tool changes than I had time to make, but it would make things more sustainable long term that maintaining lots of run_tests.{dart,sh}
logic. Using this structure would have the added benefit of setting ourselves up for that being easier to add later.)
Checklist:
- Add Swift macOS integration tests.
- Add Swift iOS integration tests.
- These are mostly redundant with macOS, but there are features like background platform channels that we don't currently support on desktop. And the added cost of having both should be low. (I don't want to do iOS-only, because for local iteration a macOS integration test is way faster.)
- Add Windows integration tests.
- Add Java Android integration tests, local-only (not in CI)
- Add Kotlin Android integration tests, local-only (not in CI)
- Update [pigeon] Consolidate platform test harnesses part 3.2 - iOS ObjC packages#2816 to include adding Obj-C integration tests
- Migrate the handful of (Obj-C-only) tests that are currently in the
e2e_tests
directory to the new structure, and eliminatee2e_tests
. - Enable Java and Kotlin integration tests in CI
- This is currently hard, which is why it's a separate step:
- We could use FTL, like we do in flutter/plugins, but that's a lot of complexity/setup to bring into the tooling (or we need to adapt the repo tooling to work with Pigeon's unusual structure), and:
- I'm hopeful that we can use the new (WIP?) LUCI Android emulator support instead, but that will require us to get that part of the testing moved over to LUCI, and emulator support wired in.
- This is currently hard, which is why it's a separate step: