Android engine test flaking on failure to fetch dependencies

[zanderso]

Feature: Migrate gradle to use --offline mode to not download dependencies dynamically during CI.

In https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20Android%20Debug%20Engine/20211/overview

ERROR: Failed to fetch maven artifact org.robolectric:android-all-instrumented:4.2.2_r1.2-robolectric-r1-i3

This flake is causing Android build/test times on CI to increase by several minutes due to automatic re-tries.

[reidbaker]

We should not be fetching dependencies as part of our CI. I am not familiar with this code but it is something my team can make sure gets fixed.

io.flutter.plugin.localization.LocalizationPluginTest > computePlatformResolvedLocale_selectOnlyLanguageLocale_beforeAndroidN[17] STANDARD_ERROR
    ERROR: Failed to fetch maven artifact org.robolectric:android-all-instrumented:4.2.2_r1.2-robolectric-r1-i3
    java.util.concurrent.ExecutionException: javax.net.ssl.SSLProtocolException: Connection reset
    	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:552)
    	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
    	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
    	at org.robolectric.internal.dependency.MavenArtifactFetcher.fetchArtifact(MavenArtifactFetcher.java:97)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.lambda$getLocalArtifactUrls$0(MavenDependencyResolver.java:80)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.whileLocked(MavenDependencyResolver.java:100)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrls(MavenDependencyResolver.java:75)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrls(MavenDependencyResolver.java:65)
    	at org.robolectric.internal.dependency.MavenDependencyResolver.getLocalArtifactUrl(MavenDependencyResolver.java:116)

[reidbaker]

Links and resources that might be helpful
I think this is the test that failed. https://github.com/flutter/engine/blob/main/shell/platform/android/test/io/flutter/plugin/localization/LocalizationPluginTest.java
Luci python file https://flutter.googlesource.com/recipes/+/refs/heads/main/recipes/engine/engine.py

Command that failed to find any mention of localization plugin testing in recipe repo ❯ find . -type f -exec grep 'LocalizationPluginTest' {} \;

[reidbaker]

Main entry point for engine tests:
https://github.com/flutter/engine/blob/main/testing/run_tests.py

What I think is the most recent version of the dependencies that are available locally
https://chrome-infra-packages.appspot.com/p/flutter/android/robolectric_bundle/+/

Pull request that migrated android to the current dependency support.
flutter/engine#17116

[reidbaker]

Current theory android-all-instrumented-4.2.2_r1.2-robolectric-r1-i4.jar is not included in the CIPD cash so gradle is downloading the dependency dynamically. I am working with @ricardoamador to try to prove that if CIPD has an update jar this issue is resolved.

Also note that gradle has an offline mode that will make our tests fail if we do not add the dependency to cipd. We might consider enabling it to see what breaks.
https://docs.gradle.org/current/userguide/dynamic_versions.html#sec:controlling_dependency_caching_command_line

[reidbaker]

PR to use offline mode
flutter/engine#39465
Failed build
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20Android%20Debug%20Engine/48013/overview

[reidbaker]

cipd create -pkg-def=somefile.yaml
Where somefile.yaml has the contents below from a directory that is the current contents of https://chrome-infra-packages.appspot.com/p/flutter/android/robolectric_bundle/+/ with any additional jars that need to be included in a folder named lib

package: flutter/android/robolectric_bundle
description: Dependencies for robolectric android testing
data:
    - dir: lib

[reidbaker]

CIPD has been update with a robolectric package but I am not sure if that will solve the initial flake on its own.

[reidbaker]

Another relevant link https://cs.opensource.google/flutter/engine/+/main:DEPS;l=662

[zanderso]

We can check in instructions for updating CIPD bundles here https://github.com/flutter/engine/tree/main/tools/cipd

[reidbaker]

Met with @christopherfujino here are the notes from our meeting.

• DEPS files are for built dependencies and CIPD is for prebuilt dependencies, ci.yaml is for dependencies that are part of your source (not sure I follow this)
• I am not missing lots of information, finding dependencies is actually this hard and it is not surprising that I have spent hours looking for where dependencies come from.
• Chris says "we agree that we don't want to download things during builds but we do because dependency management is so difficult"
• We looked through the builds and didn't see anyplace where roboletric is downloaded. 3 Ideas for what could be happening. 1. we missed it, and one of these steps explicitly download robolectric 2. robolectric was already on the bot before the build started

3. one of these steps was implicitly fetching robolectric

• go/luci-users-chat might be a good place to ask if there is a way to see if a build is using a package.
• Our best guess is that that the CIPD update we did last week is not affecting the current builds.

[reidbaker]

Another place dependencies are defined is in the docker image used in ci.
Defined here https://github.com/flutter/flutter/blame/master/dev/ci/docker_linux/Dockerfile

[reidbaker]

#121233 (comment) is another instance of reaching out to the network.

[reidbaker]

#121711 another instance

[reidbaker]

A solution for gradle can probably be broken apart from from the rest of the way dependencies are cached.

We can follow the instructions here to host the gradle files in the correct directory. Then we can modify the dockerfile to have the versions of gradle we depend on (it is a lot).

[reidbaker]

related #125232

[reidbaker]

https://maven.apache.org/guides/mini/guide-mirror-settings.html

[reidbaker]

Packages now has the ability to mark steps as infra steps and a new command to download dependencies is called and marked is infra.

I need to audit framework for more instances where we do not download gradle but packages should be handled. Engine repo however (one of the cause of this bug) is not done.

[flutter-triage-bot]

This issue is assigned to @reidbaker but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!

[loic-sharma]

This issue also affects the Flutter framework's tests. The tree closed 2+ times this week due to Gradle network errors:

1. https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_3_4/16981/overview - Exception in thread "main" java.net.SocketException: Connection reset, caused by:

   flutter/packages/flutter_tools/test/integration.shard/android_gradle_outputs_app_link_settings_test.dart

   Lines 318 to 326 in 6193615

   	result = await processManager.run(<String>[
   	'.${platform.pathSeparator}${getGradlewFileName(platform)}',
   	...getLocalEngineArguments(),
   	'-q', // quiet output.
   	'-PoutputPath=${fileDump.path}',
   	'outputDebugAppLinkSettings',
   	], workingDirectory: androidApp.path);
   	expect(result, const ProcessResultMatcher());

2. https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_3_4/16954/overview - Exception in thread "main" java.net.SocketException: Connection reset, caused by

   flutter/packages/flutter_tools/test/integration.shard/android_plugin_skip_unsupported_test.dart

   Lines 105 to 109 in 6193615

   	final ProcessResult buildApkResult = await testUnsupportedPlugin(
   	project: project, createAndroidPluginFolder: false);
   	expect(buildApkResult.stderr.toString(),
   	isNot(contains('Please fix your settings.gradle')));
   	expect(buildApkResult, const ProcessResultMatcher());

3. https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20tool_integration_tests_3_4/16822/overview - Could not GET 'https://repo.maven.apache.org/<omitted>.pom'. Received status code 503 from server: Backend unavailable, connection timeout, caused by

   flutter/packages/flutter_tools/test/integration.shard/android_gradle_outputs_app_link_settings_test.dart

   Lines 318 to 326 in 6193615

   	result = await processManager.run(<String>[
   	'.${platform.pathSeparator}${getGradlewFileName(platform)}',
   	...getLocalEngineArguments(),
   	'-q', // quiet output.
   	'-PoutputPath=${fileDump.path}',
   	'outputDebugAppLinkSettings',
   	], workingDirectory: androidApp.path);
   	expect(result, const ProcessResultMatcher());

[flutter-triage-bot]

This issue is assigned to @reidbaker but has had no recent status updates. Please consider unassigning this issue if it is not going to be addressed in the near future. This allows people to have a clearer picture of what work is actually planned. Thanks!