[Android] Update integration test AVD dependency to use Android 35 emulators #54186

camsim99 · 2024-07-29T18:00:36Z

Updates AVD dependency to use latest available AVD dependency (older at the time of publishing but the same I set in the framework -- flutter/flutter#152498) so that engine integration tests run on emulators running Android 35, the current latest version of Android.

Does not change dependency of one test that is intended to run on Android 28.

Also adds tasks to run the same emulator tests on those running Android 34 since Android 35 emulators seem to crash off and on.

Pre-launch Checklist

I read the Contributor Guide and followed the process outlined there for subm)tting PRs.
I read the Tree Hygiene wiki page, which explains my responsibilities.
I read and followed the Flutter Style Guide and the C++, Objective-C, Java style guides.
I listed at least one issue that this PR fixes in the description above.
I added new tests to check the change I am making or feature I am adding, or the PR is test-exempt. See testing the engine for instructions on writing and running engine tests.
I updated/added relevant documentation (doc comments with ///).
I signed the CLA.
All existing and new tests are passing.

zanderso · 2024-08-08T21:39:32Z

Presubs look happier with the updated version. Should maybe kick it a few times by uploading empty commits or with led to make sure.

camsim99 · 2024-08-08T22:03:17Z

I'm happy to see it! I was trying to run the previous image locally and something seems off in my environment. I'll pause on that for now and next I will use led to check for flakes.

camsim99 · 2024-08-09T16:50:00Z

Kicked off one led run https://ci.chromium.org/ui/p/flutter/builders/try.shadow/Linux%20linux_android_emulator_tests/17/overview and will do a couple of times at least to double check!

Edit other led runs:
https://ci.chromium.org/ui/p/flutter/builders/try.shadow/Linux%20linux_android_emulator_tests/18/overview

zanderso · 2024-08-09T17:22:02Z

I thought I'd also give this a shot in the framework repo. Unfortunately, I'm seeing the same issue with both a more recent version and an older version:
Newer (same as this PR): https://ci.chromium.org/ui/p/flutter/builders/try/Linux_android_emu%20external_textures_integration_test/2296/overview
Older (3-ish weeks ago before an avd update in the cipd package change log): https://ci.chromium.org/ui/p/flutter/builders/try/Linux_android_emu%20external_textures_integration_test/2297/overview

If these tests were passing on API level 34, and there's no other symptom aside from the emulator disconnecting and not dropping logs, my suggestion would be that we test on both API level 34 and API level 35 emulators, and tolerate the flakes when using the 35 emulators. To do that, we can increase the number of retries if needed, and as a follow-on teach the CI recipes that it is an infra failure and not a test failure when an emulator disconnects unexpectedly.

camsim99 · 2024-08-09T18:03:39Z

That sounds like a good plan to me. I can take on getting them to run on both in the engine in this PR.

My only concern is how we would accomplish the follow-up task of classifying the Android 35 emulator failures as an infra flake. Seems like we only know because the adb logs stop unexpectedly but is there a reliable way for us to classify that?

zanderso · 2024-08-09T18:31:57Z

My possibly naive idea would be that if the output from a test contains adb: device offline, which I think only happens when a device disappears unexpectedly, then that attempt wouldn't count against the failure count at:

https://github.com/flutter/flutter/blob/6d3e3204223f414c6d57fc58ec11f5937f25898e/dev/devicelab/lib/framework/runner.dart#L69

The internal retries would continue until the emulator or device behaves correctly, or the whole job hits the LUCI timeout.

cc @christopherfujino for thoughts on that.

christopherfujino · 2024-08-09T19:09:28Z

My possibly naive idea would be that if the output from a test contains adb: device offline, which I think only happens when a device disappears unexpectedly, then that attempt wouldn't count against the failure count at:

https://github.com/flutter/flutter/blob/6d3e3204223f414c6d57fc58ec11f5937f25898e/dev/devicelab/lib/framework/runner.dart#L69

The internal retries would continue until the emulator or device behaves correctly, or the whole job hits the LUCI timeout.

cc @christopherfujino for thoughts on that.

This SGTM

flutter-dashboard · 2024-08-12T17:27:56Z

Golden file changes have been found for this pull request. Click here to view and triage (e.g. because this is an intentional change).

If you are still iterating on this change and are not ready to resolve the images on the Flutter Gold dashboard, consider marking this PR as a draft pull request above. You will still be able to view image results on the dashboard, commenting will be silenced, and the check will not try to resolve itself until marked ready for review.

Changes reported for pull request #54186 at sha f292c02

zanderso · 2024-08-12T18:18:40Z

.ci.yaml

@@ -76,6 +73,27 @@ targets:
      - testing/scenario_app/**
      - testing/skia_gold_client/**

+  - name: Linux linux_android_emulator_tests_34
+    bringup: true


To be on the safe side, I'd suggest also shifting the the tests that will now be running on the 35 image to bringup as well.

Otherwise, lgtm.

…roid 35 emulators (flutter/engine#54186)

…153326) flutter/engine@5d8ee52...07ecd90 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from c9fd814b106a to 7bed8fb6fae2 (2 revisions) (flutter/engine#54516) 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from e232752d04c5 to c9fd814b106a (3 revisions) (flutter/engine#54515) 2024-08-12 43054281+camsim99@users.noreply.github.com [Android] Update integration test AVD dependency to use Android 35 emulators (flutter/engine#54186) 2024-08-12 jonahwilliams@google.com [iOS] keep threads merged when using Skia renderer on iOS (flutter/engine#54514) 2024-08-12 chris@bracken.jp iOS,macOS: Clean up create_ios_framework.py (flutter/engine#54500) 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from ec7558d41b34 to e232752d04c5 (3 revisions) (flutter/engine#54511) 2024-08-12 skia-flutter-autoroll@skia.org Roll Dart SDK from cfe2b63d3aca to a6cd8099ff85 (1 revision) (flutter/engine#54512) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC jsimmons@google.com,rmistry@google.com,zra@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Updates the steps based on what I found by working on flutter/engine#54186 and flutter/engine#53574 + general cleanup.

…lutter#153326) flutter/engine@5d8ee52...07ecd90 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from c9fd814b106a to 7bed8fb6fae2 (2 revisions) (flutter/engine#54516) 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from e232752d04c5 to c9fd814b106a (3 revisions) (flutter/engine#54515) 2024-08-12 43054281+camsim99@users.noreply.github.com [Android] Update integration test AVD dependency to use Android 35 emulators (flutter/engine#54186) 2024-08-12 jonahwilliams@google.com [iOS] keep threads merged when using Skia renderer on iOS (flutter/engine#54514) 2024-08-12 chris@bracken.jp iOS,macOS: Clean up create_ios_framework.py (flutter/engine#54500) 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from ec7558d41b34 to e232752d04c5 (3 revisions) (flutter/engine#54511) 2024-08-12 skia-flutter-autoroll@skia.org Roll Dart SDK from cfe2b63d3aca to a6cd8099ff85 (1 revision) (flutter/engine#54512) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC jsimmons@google.com,rmistry@google.com,zra@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Updates the steps based on what I found by working on flutter/engine#54186 and flutter/engine#53574 + general cleanup.

…lutter#153326) flutter/engine@5d8ee52...07ecd90 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from c9fd814b106a to 7bed8fb6fae2 (2 revisions) (flutter/engine#54516) 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from e232752d04c5 to c9fd814b106a (3 revisions) (flutter/engine#54515) 2024-08-12 43054281+camsim99@users.noreply.github.com [Android] Update integration test AVD dependency to use Android 35 emulators (flutter/engine#54186) 2024-08-12 jonahwilliams@google.com [iOS] keep threads merged when using Skia renderer on iOS (flutter/engine#54514) 2024-08-12 chris@bracken.jp iOS,macOS: Clean up create_ios_framework.py (flutter/engine#54500) 2024-08-12 skia-flutter-autoroll@skia.org Roll Skia from ec7558d41b34 to e232752d04c5 (3 revisions) (flutter/engine#54511) 2024-08-12 skia-flutter-autoroll@skia.org Roll Dart SDK from cfe2b63d3aca to a6cd8099ff85 (1 revision) (flutter/engine#54512) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC jsimmons@google.com,rmistry@google.com,zra@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Updates the steps based on what I found by working on flutter/engine#54186 and flutter/engine#53574 + general cleanup.

camsim99 and others added 30 commits June 25, 2024 15:44

Make base engine changes

03266f1

Merge remote-tracking branch 'upstream/main' into eng_35

10e169b

Correct 35 version

7ca603d

compileSdk

8c0e538

Undo packages + change for test

291cb41

format

65d90a9

Change back to 35

e317ce4

format

5ce0a8d

Test with buildroot changes

9b01ee7

Update 35 version to 35v1

3e09995

Update impeller test

7ff2d27

Correct Imepller version

b530500

Test correct root

35265f8

Another impeller update

380766e

Correct build tools version

a9b171c

Bump target version

d5963e2

Bump test runner AGP version

ea34633

bump manifest target

380d49d

deps

5961681

Suppress deprecations

32339a9

Undo added code

1f07cd8

Merge remote-tracking branch 'upstream/main' into eng_35

1aec7b3

Merge remote-tracking branch 'upstream/main' into eng_35

a8e44ac

Update AGP, update packages.txt, remove unneeded test case

d3b8f43

Merge remote-tracking branch 'upstream/main' into eng_35

645d8d2

what is src dir

77bf104

Correct directory

53ee46d

Merge remote-tracking branch 'upstream/main' into eng_35

ca1611b

Merge branch 'main' into eng_35

8edba5f

Update buildroot commit

7fecb2a

zanderso mentioned this pull request Aug 9, 2024

Speculatively try newer avd_cipd_version flutter/flutter#153161

Closed

Add 34 targets

f292c02

camsim99 marked this pull request as ready for review August 12, 2024 17:21

camsim99 requested a review from zanderso August 12, 2024 17:21

flutter-dashboard bot added the will affect goldens label Aug 12, 2024

zanderso approved these changes Aug 12, 2024

View reviewed changes

Mark 35 tests flaky

695d915

camsim99 added the autosubmit Merge PR when tree becomes green via auto submit App label Aug 12, 2024

auto-submit bot merged commit b877e89 into flutter:main Aug 12, 2024
31 checks passed

engine-flutter-autoroll mentioned this pull request Aug 12, 2024

Roll Flutter Engine from 5d8ee52e985b to a64883bcec8f (6 revisions) flutter/flutter#153323

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Aug 12, 2024

b877e8944 [Android] Update integration test AVD dependency to use And…

189a5a7

…roid 35 emulators (flutter/engine#54186)

engine-flutter-autoroll mentioned this pull request Aug 13, 2024

Roll Flutter Engine from 5d8ee52e985b to 07ecd90b7755 (7 revisions) flutter/flutter#153326

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Aug 13, 2024

b877e8944 [Android] Update integration test AVD dependency to use And…

eb7ca0b

…roid 35 emulators (flutter/engine#54186)

This was referenced Aug 13, 2024

[Android] Build the engine against Android 35 flutter/flutter#150215

Closed

Update docs to use new Android version in the engine flutter/flutter#151601

Merged

auto-submit bot pushed a commit to flutter/flutter that referenced this pull request Aug 14, 2024

Update docs to use new Android version in the engine (#151601)

f697d61

Updates the steps based on what I found by working on flutter/engine#54186 and flutter/engine#53574 + general cleanup.

DBowen33 pushed a commit to DBowen33/flutter that referenced this pull request Aug 16, 2024

Update docs to use new Android version in the engine (flutter#151601)

148a243

Updates the steps based on what I found by working on flutter/engine#54186 and flutter/engine#53574 + general cleanup.

Buchimi pushed a commit to Buchimi/flutter that referenced this pull request Sep 2, 2024

Update docs to use new Android version in the engine (flutter#151601)

52b6ef2

Updates the steps based on what I found by working on flutter/engine#54186 and flutter/engine#53574 + general cleanup.

jmagman mentioned this pull request Oct 22, 2024

Proposal: Remove engine top-level orchestrator builders that only kick off and collect sub-builds (no generators), and hoist sub-build to "standalone" flutter/flutter#155041

Open

16 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Android] Update integration test AVD dependency to use Android 35 emulators #54186

[Android] Update integration test AVD dependency to use Android 35 emulators #54186

Uh oh!

camsim99 commented Jul 29, 2024 •

edited

Loading

Uh oh!

zanderso commented Aug 8, 2024

Uh oh!

camsim99 commented Aug 8, 2024

Uh oh!

camsim99 commented Aug 9, 2024 •

edited

Loading

Uh oh!

zanderso commented Aug 9, 2024

Uh oh!

camsim99 commented Aug 9, 2024 •

edited

Loading

Uh oh!

zanderso commented Aug 9, 2024

Uh oh!

christopherfujino commented Aug 9, 2024

Uh oh!

flutter-dashboard bot commented Aug 12, 2024

Uh oh!

zanderso Aug 12, 2024

Uh oh!

Uh oh!

Uh oh!

[Android] Update integration test AVD dependency to use Android 35 emulators #54186

[Android] Update integration test AVD dependency to use Android 35 emulators #54186

Uh oh!

Conversation

camsim99 commented Jul 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

zanderso commented Aug 8, 2024

Uh oh!

camsim99 commented Aug 8, 2024

Uh oh!

camsim99 commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zanderso commented Aug 9, 2024

Uh oh!

camsim99 commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zanderso commented Aug 9, 2024

Uh oh!

christopherfujino commented Aug 9, 2024

Uh oh!

flutter-dashboard bot commented Aug 12, 2024

Uh oh!

zanderso Aug 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

camsim99 commented Jul 29, 2024 •

edited

Loading

camsim99 commented Aug 9, 2024 •

edited

Loading

camsim99 commented Aug 9, 2024 •

edited

Loading