-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JVM languages can't share remote cache artifacts cross-platform #11551
Comments
You've identified a problem we are well aware of. We already have a long term project to improve this, with the deliverables named in different ways. So this is essentially a duplicate, but I can't, without searching, find the issues to dup against. |
@aiuto are there any public docs about the workstreams here? We might be interested in helping out if possible |
Any indication of what this long-term project is? I found #8125 but that seems to be a subset of the work needed for this cross-platform reuse (if I read it right) because there are likely toolchain differences between macos and linux that would cause cross-platform re-use of JVM artifacts to fail anyways. |
This is a bit of work we can't easily invite others to. The key element is to trim configuration fragments from targets when they can not matter to the end result. Your example of pure Java code with no deps on native code is a perfect example of that. The changes are an intertwined mix of (to start)
There is no single issue or quick fix. This is more of a known goal we are working towards and we use that goal as a way to evaluate changes and set priorities thoughout Bazel |
Remote Execution should also allow different local host vs remote host for JVM and android_library/android_binary targets, ie running local build on macOS that connects to Remote Execution service backed by Linux machines. |
We're expecting progress when the host/exec architecture is the same (like everyone's building on common Linux machines, targeting varying architectures). That's manageable with path stripping ( If you also need caching across host/exec machines, that's an independent problem: there are legitimately different JDKs across architectures. We need to define equivalences for those. That's being looked at by a different group. Feel free to ping here for more details. To be fair for everyone, it's probably best to close all bugs in this space and open one canonical one describing the whole problem, and who's doing what on it. |
Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale. |
Description of the problem / feature request:
Even though JVM languages that don't include native code can share artifacts cross-platform (theoretically) due to targeting the JVM and not the host/target/execution architecture, the command input/output paths used in the cache digest routinely the host/target/execution platform (e.g., darwin-fastbuild) into the path which means that we cannot share those JVM artifacts cross-platform.
Feature requests: what underlying problem are you trying to solve with this feature?
It would be great to be able to share those JVM artifacts cross-platform since they should be roughly interchangeable, especially Android artifacts.
Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Not a bug, but a feature request. However, you can quite easily reproduce this by building on a linux machine, upload to remote cache, then build on a darwin machine and note that you don't get any remote cache hits.
What operating system are you running Bazel on?
Darwin and linux (ubuntu)
What's the output of
bazel info release
?release 3.2.0
If
bazel info release
returns "development version" or "(@non-git)", tell us how you built Bazel.N/A
What's the output of
git remote get-url origin ; git rev-parse master ; git rev-parse HEAD
?git@github.com:justhecuke/bazel_rules_detekt.git
bffa68b4948aa26153185f4feda278557dc01951
bffa68b4948aa26153185f4feda278557dc01951
Have you found anything relevant by searching the web?
No, I found this data by building a custom bazel that output the command protos being used to generate action cache digests and noting the paths used and how they would prevent cross-platform sharing of JVM artifacts.
Any other information, logs, or outputs that you want to share?
No.
The text was updated successfully, but these errors were encountered: