[Fuchsia] Create dedicated testers to run tests and deprecate femu_test

[zijiehe-google-com]

This change creates a fuchsia_profile_arm64_tester builder inside of the existing linux_fuchsia group. Comparing with fuchsia_profile_arm64, it uses --no-lto to limit the cost of compilation of test suites.
Similar to the fuchsia_profile_x64, it runs release build test cases + aot test.

E.g. https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Engine%20Drone/1998852/overview or more accurately https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8755990392554423809/+/u/test:_arm64_emulator_based_profile___aot_tests/stdout.

The same approach also applied to other builders. Now each fuchsia builder has its own tester which compiles test cases without lto. The change reduces the time cost of linux_fuchsia to around 20 min, e.g. https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_fuchsia/6662/overview

After this change, linux_fuchsia group has a higher coverage than the combination of linux fuchsia emu + linux fuchsia emu arm64. So I removed both builders from the .ci.yaml as well.

Bug: flutter/flutter#140179

Pre-launch Checklist

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• I read the Tree Hygiene wiki page, which explains my responsibilities.
• I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
• I listed at least one issue that this PR fixes in the description above.
• I added new tests to check the change I am making or feature I am adding, or the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
• I updated/added relevant documentation (doc comments with ///).
• I signed the CLA.
• All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

[CaseyHillers]

Nice, thanks!

\cc @keyonghan as we can remove https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine/femu_test.py after this lands.

[zanderso]

The linux_fuchsia builder is currently the slowest builder at >40 minutes on average. Before we continue work in this direction, I would like to understand the plan for getting this builder back down below 30 minutes.

[zijiehe-google-com]

Yes, Zachary, as we discussed before, I do believe the current setup is not ideal.
The time consumption can be categorized into three,

1. build & test, ~10min. Multiple drones are running in parallel, so changes like this one won't meaningfully impact the overall time.
2. archive, upload and download via cas. 6 out of 7 (without the one added in this change) are uploading the binaries to cas in parallel, and they will then be downloaded in sequential. Each of them costs around 40 seconds. Previously it was ~10 seconds each, but newly added test targets increased the consumption. (from ~1 min to ~4 min).
3. upload archives to cipd. The binaries from the step 2 are uploaded to cipd. It costs 10-15 min. Newly added targets do not noticeably increase the time, time consumption of cipd itself is not very stable anyway.

So it's pretty obvious, both the try & prod builds do not need to run every step above. I.e.
Try: run build & test only, the cost will be dropped to ~10min.

• Archiving the try built binaries seems not reasonable.

Prod: run build, archive & upload only. The cost reduction may not be noticeable, since the lto build usually is the slowest, and they are running in parallel.

• The tests are fairly reliable - I haven't noticed any flakiness so far. Any broken changes should be caught by the try builders anyway.

But the above improvements need to make the try and prod builds different, I am not very sure if it's easily doable in the current setup.

Or another solution is to fully detach the testers with the builders. Same as the new fuchsia_profile_arm64_tester added by this CL, we can have testers for other variants, they won't archive the binaries, but use the time to run the tests. So the overall time consumption won't accumulate, but spread across multiple drones. It won't cause divergence between try and prod builds. But the downside is the extra drones needed (6 -> 12).

For the immediate action, I can go through several latest prod runs and see if any noticeable slow tests can be excluded as low hanging fruit, and update this CL.

Let me know your opinion.

[flutter-dashboard]

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact "@test-exemption-reviewer" in the #hackers channel in Chat (don't just cc them here, they won't see it! Use Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

[zanderso]

Reducing are sharding the tests that are running sounds like the right approach to me. The appropriate differences between prod and try would need to be coordinated with @CaseyHillers.

[zijiehe-google-com]

Reducing are sharding the tests that are running sounds like the right approach to me. The appropriate differences between prod and try would need to be coordinated with @CaseyHillers.

Done

[zanderso]

I see https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20linux_fuchsia/6672/overview completed in 18 minutes. Awesome!

[zijiehe-google-com]

I will wait for next Monday to submit this change.

[zijiehe-google-com]

FYI, the latest run dropped from 46m to 24m:

1520 2024-02-20 10:39 2024-02-20 11:03 24m 22a709cc7
No Summary.

1519 2024-02-20 09:40 2024-02-20 10:25 46m d2b5dfdb4
No Summary.