Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nix: make fetching Gradle dependencies more generic #17161

Merged
merged 2 commits into from
Sep 4, 2023

Conversation

jakubgs
Copy link
Member

@jakubgs jakubgs commented Aug 31, 2023

Before what we did was essentially guess what files might exist for any given package. This approach mostly works, but not entirely. This is especially problematic when dealing with weird edge case packages like react-native, which you can read about here:

In order to avoid as much the guessing aspect of fetching Gradle dependencies we are using both HTML listsings of files and artifact-metadata.json files that exist for more recent packages.

This way we can avoid having to add special edge cases that have been found out when working on React Native 72 upgrade in:

@jakubgs jakubgs added android nix dependencies Pull requests that update a dependency file labels Aug 31, 2023
@jakubgs jakubgs self-assigned this Aug 31, 2023
@status-im-auto
Copy link
Member

status-im-auto commented Aug 31, 2023

Jenkins Builds

Click to see older builds (31)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 46633b8 #1 2023-08-31 14:27:41 ~6 min android 🤖apk 📲
✔️ 46633b8 #1 2023-08-31 14:27:44 ~6 min android-e2e 🤖apk 📲
✔️ 46633b8 #1 2023-08-31 14:27:46 ~6 min ios 📱ipa 📲
✔️ 46633b8 #1 2023-08-31 14:29:31 ~8 min tests 📄log
✔️ 822520b #2 2023-08-31 15:55:40 ~6 min android 🤖apk 📲
✔️ 822520b #2 2023-08-31 15:55:59 ~6 min ios 📱ipa 📲
✔️ 822520b #2 2023-08-31 15:56:52 ~7 min android-e2e 🤖apk 📲
✔️ 822520b #2 2023-08-31 15:59:13 ~9 min tests 📄log
✔️ 491fc15 #3 2023-08-31 16:13:28 ~6 min android-e2e 🤖apk 📲
✔️ 491fc15 #3 2023-08-31 16:13:33 ~6 min android 🤖apk 📲
✔️ 491fc15 #3 2023-08-31 16:13:51 ~6 min ios 📱ipa 📲
✔️ 491fc15 #3 2023-08-31 16:15:28 ~8 min tests 📄log
✔️ 2465ffb #4 2023-08-31 17:21:43 ~6 min ios 📱ipa 📲
✔️ 2465ffb #4 2023-08-31 17:21:48 ~6 min android-e2e 🤖apk 📲
✔️ 2465ffb #4 2023-08-31 17:21:49 ~6 min android 🤖apk 📲
✔️ 2465ffb #4 2023-08-31 17:23:57 ~8 min tests 📄log
✔️ ad20022 #5 2023-08-31 18:12:05 ~6 min android-e2e 🤖apk 📲
✔️ ad20022 #5 2023-08-31 18:12:19 ~6 min android 🤖apk 📲
✔️ ad20022 #5 2023-08-31 18:12:31 ~6 min ios 📱ipa 📲
✔️ 1ba7714 #6 2023-08-31 18:18:46 ~5 min android-e2e 🤖apk 📲
✔️ 1ba7714 #6 2023-08-31 18:19:47 ~6 min ios 📱ipa 📲
✔️ 1ba7714 #6 2023-08-31 18:21:26 ~8 min android 🤖apk 📲
✔️ 1ba7714 #6 2023-08-31 18:22:41 ~9 min tests 📄log
✔️ 8feac89 #7 2023-08-31 23:08:09 ~6 min android-e2e 🤖apk 📲
✔️ 8feac89 #7 2023-08-31 23:08:28 ~6 min ios 📱ipa 📲
✔️ 8feac89 #7 2023-08-31 23:10:42 ~8 min android 🤖apk 📲
✔️ 8feac89 #7 2023-08-31 23:11:55 ~9 min tests 📄log
✔️ 4c32c43 #8 2023-09-04 07:46:21 ~6 min ios 📱ipa 📲
✔️ 4c32c43 #8 2023-09-04 07:49:09 ~9 min android-e2e 🤖apk 📲
✔️ 4c32c43 #8 2023-09-04 07:49:14 ~9 min android 🤖apk 📲
✔️ 4c32c43 #8 2023-09-04 07:49:50 ~10 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 83f622d #10 2023-09-04 08:04:29 ~5 min android 🤖apk 📲
✔️ 83f622d #10 2023-09-04 08:04:35 ~6 min android-e2e 🤖apk 📲
✔️ 83f622d #10 2023-09-04 08:05:06 ~6 min ios 📱ipa 📲
✔️ 83f622d #10 2023-09-04 08:07:27 ~8 min tests 📄log
✔️ f433956 #11 2023-09-04 08:52:08 ~6 min ios 📱ipa 📲
✔️ f433956 #11 2023-09-04 08:55:07 ~9 min android 🤖apk 📲
✔️ f433956 #11 2023-09-04 08:55:18 ~10 min android-e2e 🤖apk 📲
✔️ f433956 #11 2023-09-04 08:55:44 ~10 min tests 📄log

@jakubgs jakubgs force-pushed the nix/generic-maven-url2json branch 5 times, most recently from ad20022 to 1ba7714 Compare August 31, 2023 18:13
@jakubgs
Copy link
Member Author

jakubgs commented Aug 31, 2023

Here's the difference in sizes of the Gradle dependencies derivation:

Before

 > du -hsL $(nix-build --no-out-link -A pkgs.deps.gradle)
957M	/nix/store/6dahmcwyygam7whvwi3zsrgxa49i5xwp-status-mobile-maven-deps

 > find -L $(nix-build --no-out-link -A pkgs.deps.gradle) -type f | wc -l
4651

After

 > du -hsL $(nix-build --no-out-link -A pkgs.deps.gradle)
1010M	/nix/store/0v8m8a321pp3vf01vn7inf5m3xl0rbpz-status-mobile-maven-deps

 > find -L $(nix-build --no-out-link -A pkgs.deps.gradle) -type f | wc -l
7585

Which is not horrible. Out of the new files most are the .asc files with GPG signatures. To be exact 1445.

This is annoying because as far as I can tell only one package actually requires the .asc file is conceal-1.1.3:

Could not determine the dependencies of task ':app:lintVitalReportPr'.
> Could not resolve all task dependencies for configuration ':app:prRuntimeClasspath'.
   > Could not resolve com.facebook.conceal:conceal:1.1.3.
     Required by:
         project :app > project :react-native-keychain
      > No cached version of com.facebook.conceal:conceal:1.1.3 available for offline mode.

@jakubgs jakubgs marked this pull request as ready for review September 1, 2023 08:01
Copy link
Contributor

@siddarthkay siddarthkay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and it works.
My benchmarks for single runs :

  • make nix-update-gradle ~ 18 mins
  • make run-android ~ 13 mins

@jakubgs
Copy link
Member Author

jakubgs commented Sep 4, 2023

I think I can make the blacklist more strict. I believe we only need ASC files for AARs - of which there is only 32 - for builds.

@jakubgs
Copy link
Member Author

jakubgs commented Sep 4, 2023

That's better:

 > du -hsL $(nix-build --no-out-link -A pkgs.deps.gradle)
1006M	/nix/store/8g5vzlrfhihy7pv7ya6f29hr4941n9kq-status-mobile-maven-deps

 > find -L $(nix-build --no-out-link -A pkgs.deps.gradle) -type f | wc -l
4763

A make nix-update-gradle run for me takes 7 minutes.

Before what we did was essentially guess what files might exist for any
given package. This approach mostly works, but not entirely.
This is especially problematic when dealing with weird edge case
packages like `react-native`, which you can read about here:

react-native-community/discussions-and-proposals#508
https://github.com/react-native-community/discussions-and-proposals/blob/4a06fc64/proposals/0508-out-of-npm-artifacts.md#the-react-native-android-archive

In order to avoid as much the guessing aspect of fetching Gradle
dependencies we are using both HTML listsings of files and
`artifact-metadata.json` files that exist for more recent packages.

This way we can avoid having to add special edge cases that have been
found out when working on React Native 72 upgrade in:
#17062

Signed-off-by: Jakub Sokołowski <jakub@status.im>
Signed-off-by: Jakub Sokołowski <jakub@status.im>
@jakubgs jakubgs merged commit f433956 into develop Sep 4, 2023
2 checks passed
@jakubgs jakubgs deleted the nix/generic-maven-url2json branch September 4, 2023 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
android dependencies Pull requests that update a dependency file nix
Projects
Archived in project
Archived in project
Development

Successfully merging this pull request may close these issues.

4 participants