[Tool] New tool to download android dependencies #4408

reidbaker · 2023-07-07T20:38:40Z

This pr is pushed for high level feedback/conversation. I will add tests before serious review.
should be read in conjuction with https://flutter-review.googlesource.com/c/recipes/+/46980

Create new top level command to run flutter dependencies on changed packages
when running android tests download dependencies before running tests

flutter/flutter#120119

Pre-launch Checklist

…ackages

flutter-dashboard · 2023-07-07T20:38:44Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat (don't just cc him here, he won't see it! He's on Discord!).

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

stuartmorgan

This looks great! If there's a way to share the cache for the whole test run rather than per-plugin that would be even better (or is Gradle actually doing that under the hood?), but either way this is a clear improvement, and should really help us quickly diagnose network and server flake.

.ci/targets/android_platform_tests.yaml

script/tool/lib/src/fetch_gradle_deps_command.dart

reidbaker · 2023-07-10T15:30:12Z

https://flutter-review.googlesource.com/c/recipes/+/46980

So our testing system is complex but gradle lays out the files it relies on here by default all tests on a machine will share the gradle cache unless we configure it differently. Searching I was not able to find a server lifecycle doc to see how often those machines are wiped [1]. Searching for where we might configure gradle home I found a bunch of places where we set GRADLE_USER_HOME and all the the values we set are the same [cache]/gradle value. So those places should share a gradle directory which holds both dependencies and gradle wrapper version and cleans itself of unused versions every 7-30 days depending on the type of thing in the cache.

[1] Moma search, exploring https://flutter.googlesource.com/infra/

.ci/targets/android_platform_tests.yaml

keyonghan · 2023-07-10T21:14:07Z

Searching for where we might configure gradle home I found a bunch of places where we set GRADLE_USER_HOME and all the the values we set are the same [cache]/gradle value. So those places should share a gradle directory which holds both dependencies and gradle wrapper version and cleans itself of unused versions every 7-30 days depending on the type of thing in the cache.

Do you have a build example which downloads gradle dependencies that I can take a look? [cache]/gradle relies on swarming named cache, and we should have a corresponding named cache entry defined so that different tests can share the cache on the same testbed. Based on existing infra config, I do not see any named cache pointing to [cache]/gradle.

johnmccutchan · 2023-07-10T22:15:03Z

script/tool/lib/src/fetch_deps_command.dart

+import 'common/plugin_utils.dart';
+import 'common/repository_package.dart';
+
+/// Run 'gradlew dependencies' on android.


Update this comment now that this command is more generic.

I think I did. If you notice the line below you can see that I document the other things this command could do but this comment is correct with what it does now.

My expectation is that if/when we add dart support the tool comment will be updated to say

/// Run 'gradlew dependencies' on android. /// Run 'pub get' on dart packages. /// Run 'pod install' and 'swift package download --all' depending on the configuration.

For now I updated the top line to have a summary then the specifics. Hopefully that addresses your concern.

…tep improve documentaion in fetch-deps command

reidbaker · 2023-07-11T19:26:31Z

Searching for where we might configure gradle home I found a bunch of places where we set GRADLE_USER_HOME and all the the values we set are the same [cache]/gradle value. So those places should share a gradle directory which holds both dependencies and gradle wrapper version and cleans itself of unused versions every 7-30 days depending on the type of thing in the cache.

Do you have a build example which downloads gradle dependencies that I can take a look? [cache]/gradle relies on swarming named cache, and we should have a corresponding named cache entry defined so that different tests can share the cache on the same testbed. Based on existing infra config, I do not see any named cache pointing to [cache]/gradle.

This is the only pr that should have the new step to download dependencies. That said all existing android test would have gradle dependencies because I think we include kotlin by default.

Here is a recent build https://ci.chromium.org/ui/p/flutter/builders/try/Linux_android%20android_build_all_packages%20master/505/overview

I am not sure if I understand your comment about the existing infra config all I have found is in https://cs.opensource.google/search?q=GRADLE_USER_HOME%20-f:gradle-wrapper.properties&start=1.

reidbaker · 2023-07-11T19:32:41Z

Tests failing https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8775849073271137377/+/u/Run_package_tests/download_android_deps/stdout

Unhandled exception:
Could not find an option named "packages-for-branch".

keyonghan

android_platform_tests.yaml LGTM

…kages#4408)

flutter/packages@aa1eace...369ee7e 2023-07-14 reidbaker@google.com [Tool] New tool to download android dependencies (flutter/packages#4408) 2023-07-14 zaldypagaduanjr@gmail.com [video_player] added iOS exception on incorrect asset path (flutter/packages#4318) 2023-07-14 joonas.kerttula@codemate.com [google_maps_flutter_platform_interface] Add support for cloud-based map styling (flutter/packages#4141) 2023-07-14 10687576+bparrishMines@users.noreply.github.com [webview_flutter] Adds support for receiving a url with WebResourceError (flutter/packages#3884) 2023-07-14 stuartmorgan@google.com [xdg_directories] Remove `process` dependency (flutter/packages#4460) 2023-07-13 stuartmorgan@google.com [various] Update Pigeon in Swift plugins (flutter/packages#4461) 2023-07-13 engine-flutter-autoroll@skia.org Roll Flutter from 544d30d to c40173f (10 revisions) (flutter/packages#4457) 2023-07-13 engine-flutter-autoroll@skia.org Roll Flutter (stable) from 796c8ef to f468f33 (2 revisions) (flutter/packages#4456) 2023-07-13 bisor0627@gmail.com Fix Router Config Issues #4300 (go_router_builder/example) (flutter/packages#4369) 2023-07-13 10687576+bparrishMines@users.noreply.github.com [webview_flutter_platform_interface] Adds url to `WebResourceError` (flutter/packages#4439) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

flutter/packages@aa1eace...369ee7e 2023-07-14 reidbaker@google.com [Tool] New tool to download android dependencies (flutter/packages#4408) 2023-07-14 zaldypagaduanjr@gmail.com [video_player] added iOS exception on incorrect asset path (flutter/packages#4318) 2023-07-14 joonas.kerttula@codemate.com [google_maps_flutter_platform_interface] Add support for cloud-based map styling (flutter/packages#4141) 2023-07-14 10687576+bparrishMines@users.noreply.github.com [webview_flutter] Adds support for receiving a url with WebResourceError (flutter/packages#3884) 2023-07-14 stuartmorgan@google.com [xdg_directories] Remove `process` dependency (flutter/packages#4460) 2023-07-13 stuartmorgan@google.com [various] Update Pigeon in Swift plugins (flutter/packages#4461) 2023-07-13 engine-flutter-autoroll@skia.org Roll Flutter from 544d30d to c40173f (10 revisions) (flutter/packages#4457) 2023-07-13 engine-flutter-autoroll@skia.org Roll Flutter (stable) from 796c8ef to f468f33 (2 revisions) (flutter/packages#4456) 2023-07-13 bisor0627@gmail.com Fix Router Config Issues flutter#4300 (go_router_builder/example) (flutter/packages#4369) 2023-07-13 10687576+bparrishMines@users.noreply.github.com [webview_flutter_platform_interface] Adds url to `WebResourceError` (flutter/packages#4439) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC flutter-ecosystem@google.com,rmistry@google.com on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

reidbaker added 2 commits July 7, 2023 15:39

Create new top level command to run flutter dependencies on changed p…

8ccc636

…ackages

when running android tests download dependencies before running tests

96d0254

flutter-dashboard bot added the needs tests label Jul 7, 2023

reidbaker requested a review from stuartmorgan July 7, 2023 20:43

stuartmorgan reviewed Jul 7, 2023

View reviewed changes

.ci/targets/android_platform_tests.yaml Outdated Show resolved Hide resolved

script/tool/lib/src/fetch_gradle_deps_command.dart Outdated Show resolved Hide resolved

script/tool/lib/src/fetch_gradle_deps_command.dart Outdated Show resolved Hide resolved

reidbaker added 4 commits July 10, 2023 11:34

remove unrelated comment

3a552d2

Merge branch 'main' into i120119-plugin-command-to-fetch-dependencies

7023c57

Merge branch 'main' into i120119-plugin-command-to-fetch-dependencies

aad004a

Rename command to fetch deps

6877907

keyonghan reviewed Jul 10, 2023

View reviewed changes

.ci/targets/android_platform_tests.yaml Outdated Show resolved Hide resolved

Rename command to fetch deps, add tests

30829a0

reidbaker requested a review from johnmccutchan July 10, 2023 20:45

Bump dependency downloading to before build

681fe6e

johnmccutchan approved these changes Jul 10, 2023

View reviewed changes

Document that "always" does not mean always when dealing with infra_s…

f5871c9

…tep improve documentaion in fetch-deps command

fix name of arg with command name change

2ffae59

keyonghan approved these changes Jul 11, 2023

View reviewed changes

Merge branch 'main' into i120119-plugin-command-to-fetch-dependencies

ab45c2a

johnmccutchan approved these changes Jul 12, 2023

View reviewed changes

reidbaker added 3 commits July 13, 2023 14:11

Merge branch 'main' into i120119-plugin-command-to-fetch-dependencies

1bbe430

Add import for moved utility

6689161

Merge branch 'main' into i120119-plugin-command-to-fetch-dependencies

d6a9168

reidbaker added autosubmit Merge PR when tree becomes green via auto submit App and removed needs tests labels Jul 14, 2023

auto-submit bot merged commit 369ee7e into flutter:main Jul 14, 2023
73 checks passed

engine-flutter-autoroll mentioned this pull request Jul 14, 2023

Roll Packages from aa1eace00188 to 369ee7e1a1cb (10 revisions) flutter/flutter#130592

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jul 14, 2023

369ee7e [Tool] New tool to download android dependencies (flutter/pac…

3b1c1e5

…kages#4408)

reidbaker deleted the i120119-plugin-command-to-fetch-dependencies branch July 14, 2023 17:38

reidbaker mentioned this pull request Jul 14, 2023

[tooling] mark infra steps as infra steps in yaml files #4473

Merged

11 tasks

engine-flutter-autoroll mentioned this pull request Jul 14, 2023

Roll Packages from aa1eace00188 to 369ee7e1a1cb (10 revisions) flutter/flutter#130605

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Jul 14, 2023

369ee7e [Tool] New tool to download android dependencies (flutter/pac…

638a57a

…kages#4408)

reidbaker mentioned this pull request Jul 26, 2023

[Tooling] Add google owned cache for dependencies as an option in ci #4567

Merged

11 tasks

reidbaker mentioned this pull request Aug 4, 2023

[All] Expand artifact hub to all plugins #4645

Merged

11 tasks

camsim99 mentioned this pull request May 30, 2024

Gradle wrapper download failures should be classified as an infra flake. flutter/flutter#122031

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tool] New tool to download android dependencies #4408

[Tool] New tool to download android dependencies #4408

reidbaker commented Jul 7, 2023 •

edited

Loading

flutter-dashboard bot commented Jul 7, 2023

stuartmorgan left a comment

reidbaker commented Jul 10, 2023

keyonghan commented Jul 10, 2023 •

edited

Loading

johnmccutchan Jul 10, 2023

reidbaker Jul 11, 2023

reidbaker commented Jul 11, 2023

reidbaker commented Jul 11, 2023

keyonghan left a comment

[Tool] New tool to download android dependencies #4408

[Tool] New tool to download android dependencies #4408

Conversation

reidbaker commented Jul 7, 2023 • edited Loading

Pre-launch Checklist

flutter-dashboard bot commented Jul 7, 2023

stuartmorgan left a comment

Choose a reason for hiding this comment

reidbaker commented Jul 10, 2023

keyonghan commented Jul 10, 2023 • edited Loading

johnmccutchan Jul 10, 2023

Choose a reason for hiding this comment

reidbaker Jul 11, 2023

Choose a reason for hiding this comment

reidbaker commented Jul 11, 2023

reidbaker commented Jul 11, 2023

keyonghan left a comment

Choose a reason for hiding this comment

reidbaker commented Jul 7, 2023 •

edited

Loading

keyonghan commented Jul 10, 2023 •

edited

Loading