Bundler platform locking causes problems out-of-box, on CI, on Heroku etc.

[pond]

Describe the problem as clearly as you can

I'm hopeful we can come up with a compromise to a significant pain point that's worsening over time, not getting better.

In Bundler 2.2, PLATFORMS in Gemfile.lock only includes the local machine's architecture when that machine causes a Gemfile.lock to be built (but confusingly, not if it is merely updated). Bundler lockfiles cease to become architecture independent, even though Ruby is supposed to be. Any attempt to install using that lockfile on any other architecture fails unless new architectures are manually added. This even includes developers upgrading e.g. macOS versions from 11 to 12 and similar, since their platform includes a Darwin kernel version. On Heroku, this broke the world for a while, as the 2.2 approach appeared to not be backwards compatible with Bundler 2.1 that Heroku provided at the time.

Developers are forced to constantly remember to check, double check and where necessary hand-maintain their Gemfile.lock's PLATFORMS, to nanny things through CI (e.g. macOS development, AWS Linux CI) or just allow for the fact that they might not be on the same precise e.g. macOS version as their peers, or might be using Arch Linux, or a different CPU (e.g. M1), or whatever.

There is presently no clean way around this. Unfrozen is too permissive and has unrelated side effect of mutable Ruby gem versions, rather than only dealing with native extension platforms. Bundler should not be locking itself to the arbitrary architecture of the local developer machine IMHO; its job is to lock gem versions. I am aware of this comment but it doesn't really seem to resonate; the "safety" argument talks about a gem which doesn't have a particular architecture variant and installs then runs on CI presumably building "generic" native extensions, CI passes, then subsequently an architecture-specific variant is pushed but it's buggy and this gets deployed after the CI run to Production. If the timing of things were thus, it'd still happen regardless of what PLATFORMS might say - it'd still build generic on CI because no arch-specific version existed. Meanwhile, if malicious code is being pushed to RubyGems, then there are far bigger issues at play and far greater concern than specific platform variant locking.

A reasonable solution IMHO might be to provide a configuration option that allows alternative platforms, as if the running platform was included in the PLATFORMS array, without needing to modify Gemfile.lock. That could be a bundle config item or env var. That's not a question of just forcing to ruby, the resolution of which allegedly is buggy or unsafe and is the whole reason why the new 2.2 behaviour was introduced anyway - though if Bundler guaranteed and formally documented that forcing to ruby was still fully supported, perhaps that would be acceptable (now that it works again, thanks to this fix!). Otherwise we're building on sand.

Did you try upgrading rubygems & bundler?

Yes. Bundler 2.2.x is working as designed. I do not believe this design is helpful, with widespread community posts across forums and blogs indicating the developer pain it has inadvertently introduced.

Post steps to reproduce the problem

Cause a Gemfile.lock to be generated under Bundler 2.2.x on one architecture. Try to bundle install on another. This will fail, even though Ruby and gems all work just fine on the target architecture. In Bundler 2.1 and all earlier versions, the gems would install.

What were you expecting to happen?

• If the gem is not compatible with the chosen platform, the native extension build will fail, just as it always has.
• If the gem can build on the chosen platform, then it should do so, and not have Bundler fail the whole thing.

What actually happened?

Bundler refuses to install the gems:

Your bundle only supports platforms ["x86_64-darwin-21"] but your local platform
is x86_64-linux. Add the current platform to the lockfile with `bundle lock
--add-platform x86_64-linux` and try again.

[indirect]

There is no such thing as “gem compatible with the platform”. As soon as gems have a platform (beyond “ruby”) they are entirely different code bases. Historically, there are extremely popular gems like Nokogiri that have completely different dependencies for the exact same version on a different platform. On a new platform, Bundler literally has to start the resolution over again from scratch, find out what gems and dependencies are needed, and create a new lockfile. Without an explicit lock for a given platform, you are asking for Bundler to silently allow new gems to be resolved and installed only in production, which is a complete nonstarter.

Can you please describe what about your workflow makes it impossible for developers to run bundle lock once per platform you use?

[pond]

There is no such thing as “gem compatible with the platform”. As soon as gems have a platform (beyond “ruby”) they are entirely different code bases

But they are not different gems, don't have different APIs and don't have different major, minor or patch versions. The gem is honour-bound by contract of its API to behave the same on whatever platforms it can build on.

The workflow I'd like is the one where Bundler <= 2.1 did all this just fine and nobody had to run "bundle lock" ever.

...impossible for developers to run bundle lock once per platform you use?

It's impossible for CI to do that. The developers have to remember to 'bundle lock' not just for whatever arch their local machine is, but also include the arch that CI uses.

[whithajess]

@indirect In terms of workflow won't this always be the case? ->

1. Developer runs bundle install - Generates lock file with Local Architecture
2. CI / CD picks up the lock file to try install the gems specified (Often specifically specifying frozen to intentionally fail and catch as mismatch between Gemfile and Gemfile.lock)
3. Fails every time unless the CI/CD Architecture has been specifically added to the lockfile generated by the developer (and is maintained there)

As far as I can see the only workaround for the issue is as follows:

1. Developer runs bundle install - Generates lock file with Local Architecture
2. Developer runs bundle lock --add-platform X (knowing the remote platforms architecture)
   • Must happen before it gets to CI/CD otherwise prevented from update due to being frozen
   • Must remember to do this always when recreating or initially creating any Gemfile.lock where the remote architecture differs

force_ruby_platform won't work either with frozen

The only reason this isn't more widely reported is because people have started their projects on:

PLATFORMS
  ruby

And haven't recreated their Gemfile.lock

Everyone who starts a new project with a different remote platform architecture will run into this if they use frozen which is best practice.

[deivid-rodriguez]

I do understand the pain caused by this, although I disagree with "things worked just fine on Bundler 2.1". Things worked just fine... sometimes. Sometimes they didn't. But I understand that the workflow was easier before in some simple cases that did work.

I don't really understand the proposed solution though. You proposed defining a configuration option to allow resolving for selected platforms without modifying the lockfile. That would mean completely ignoring the lockfile? That seems wrong, and you can get the same effect by... removing the lockfile, no? I guess you instead mean assuming the resolve for the new platform is the same as the one already recorded in the lockfile, except for platform specific variants, where Bundler should try some other variants on the fly. I don't really like that solution because it essentially means ignoring the "frozen" flag and allowing potentially different code from the one recored in the lockfile to be installed as @indirect points out. Plus, other platform specific gems may have different dependencies, potentially conflicting with the ones already recorded, so there's no way to make such a solution correct.

In my opinion, any change to alleviate this problem must be done at Gemfile.lock generation time, not at installation time.

I was thinking that we could try something like this at Gemfile.lock generation time:

• Resolve for the current platform.
• If no resolved gems have platform specific variants, then add the RUBY platform to the lockfile too, since it's guaranteed to be correct.
• If some resolved gems have platform specific variants, then
  • If those variants are only for the running platform, then do what we do now (lock only the current platform).
  • It those variants are for the running platform and for other platforms too, then double check whether the resolution is valid for those platforms.
    • If yes, add those platforms and platform specific variants to the lockfile as well.
    • If no, do nothing.

This means that running the classic rails new foo command on Darwin will generate a Gemfile.lock file valid for both Darwin and Linux. And I think it wouldn't have significant overhead because we would only have to resolve once, only overhead would be checking whether that resolution is also valid for other platforms.

[indirect]

@deivid-rodriguez I like that idea! I worry a little bit that this might lead to locking a lot of platforms, but I’d be interested to try it out in a prerelease.

[aprilmintacpineda]

I'm having a similar error with GIthub Actions. This used to work last time I tried it (months ago), I don't know why I have to do this now.

bundle install
  /Users/runner/hostedtoolcache/Ruby/3.0.2/x64/bin/bundle config --local path /Users/runner/work/beny-react-native/beny-react-native/vendor/bundle
  /Users/runner/hostedtoolcache/Ruby/3.0.2/x64/bin/bundle config --local deployment true
  Cache key: setup-ruby-bundler-cache-v3-macos-11.0-ruby-3.0.2-Gemfile.lock-3ee146a75c2f746ba7e1b2c2206de7[36](https://github.com/disruptivsolutions/beny-react-native/runs/5176473300?check_suite_focus=true#step:4:36)66e06f2b00fdcfdaaabcde931febe78d
  /Users/runner/hostedtoolcache/Ruby/3.0.2/x64/bin/bundle install --jobs 4
  Your bundle only supports platforms ["x86_64-darwin-20"] but your local platform
  is x86_64-darwin-19. Add the current platform to the lockfile with `bundle lock
  --add-platform x86_64-darwin-19` and try again.
  Took   1.12 seconds
Error: The process '/Users/runner/hostedtoolcache/Ruby/3.0.2/x64/bin/bundle' failed with exit code 16

• What benefit does this add?
• What if you can't control the architecture that your CI will run on? Should you then add all possible architectures that the CI is using to Gemfile.lock? Is that a good idea because at that point the whole thing no longer makes sense, might as well just remove it completely.

[deivid-rodriguez]

@indirect Yes, I have the same concern only it doesn't seem like a big deal in principle I think. We could, however, start by doing this only for Linux and see how it goes.

[pond]

Any further progress or more thoughts on this?

[deivid-rodriguez]

No, just that I plan to work on this soon.

[lucaspiller]

@deivid-rodriguez We are working on a platform that is developed on macOS running either Intel or Apple Silicon CPUs and Linux laptops, and run on Linux servers with Intel CPUs.

What's not clear is what problem this is trying to solve. The example you gave of nokogiri being different on different hosts/architectures makes sense, but I don't see how adding an extra flag to the Gemfile.lock would solve it. Maybe if there was a different Gemfile.lock for each archiecture, but that's not what this is. It seems like this is more an indication that it should work, but why force developers to change their workflow? Trust people are smart and know what they are doing if they try to mix hosts/architectures.

[deivid-rodriguez]

Not sure which example you mean, but locking the specific platform in the Gemfile.lock file did fix platform issues as already explained. Anyways, I already admitted that this is inconvenient and that I will work on solving it, not sure what it is that you expect to get from your message.

[deivid-rodriguez]

Finally got around to working on this, feel free to give #5700 a try!

[pond]

@deivid-rodriguez

Uuuh - but - I just updated Ruby Gems and this forced me up to Bundler 2.3.22, where I found the behaviour not fixed at all - it has become even worse! When I do bundle update, it's now forcibly overwriting anything I do in PLATFORMS. I cannot put ruby in there, even in addition to x86_64-darwin-21 (or the Apple Silicon equivalent) as the entire thing is thrown away and replaced.

This makes it impossible for us to move our code even between different development machines, e.g. Apple Silicon vs x86, and impossible to deploy to CI!

How can this be resolved please? This is now an extremely critical, entirely workflow-breaking bug. I can't even uninstall 2.3.22 to go back to the previous working version, because I'm told:

Gem bundler-2.3.22 cannot be uninstalled because it is a default gem

...so I'm basically trapped and locked into a broken version via RubyGems itself, and now unable to deploy working Ruby code anywhere because - very obviously! - our deployed Staging and Production systems are not x86 Apple Macs.

I note in v2.3.21:

https://github.com/rubygems/rubygems/blob/master/bundler/CHANGELOG.md#bug-fixes-1

Fix Ruby platform incorrectly removed on bundle update #5832

...so this was either reverted in 2.3.22, or was never properly fixed in the first place.

Is PR #5700 dead or something? It's still in draft and there are still very major bugs with Bundler - I certainly cannot imagine a more serious bug than one which as meant bundle update force-rewrites Gemfile.lock into a form which is unusable on any other computer, including our CI system. We must lock our CI, Staging and Production bundles, obviously, so we can be sure that the gems which were used to pass tests in CI are the same gems used in deployment.