Bundler 2.2.3 generated Gemfile.lock on a mac cannot be used on linux out of the box

[schneems]

Describe the problem as clearly as you can

Here's an example repo: https://github.com/schneems/bundler_223_fails_deploy. It has one dependency rake and the Gemfile.lock was generated by running bundle install on my mac. When I try to use this Gemfile/Gemfile.lock combo it fails:

docker run -it --rm ruby:2.7 bash
git clone https://github.com/schneems/bundler_223_fails_deploy
cd bundler_223_fails_deploy
export BUNDLE_WITHOUT='development:test'
export BUNDLE_DEPLOYMENT=1
export BUNDLE_PATH=vendor/bundle
bundle install -j4

Then

 bundle exec rake -P
bundler: failed to load command: rake (/bundler_223_fails_deploy/vendor/bundle/ruby/2.7.0/bin/rake)
Gem::Exception: can't find executable rake for gem rake. rake is not currently included in the bundle, perhaps you meant to add it to your Gemfile?
  /usr/local/lib/ruby/2.7.0/bundler/rubygems_integration.rb:374:in `block in replace_bin_path'
  /usr/local/lib/ruby/2.7.0/bundler/rubygems_integration.rb:402:in `block in replace_bin_path'
  /bundler_223_fails_deploy/vendor/bundle/ruby/2.7.0/bin/rake:23:in `<top (required)>'

What were you expecting to happen?

I expect that a Gemfile.lock produced on mac can also work on linux

What actually happened?

The gem appears to install, but then it cannot be loaded.

Extra context

Either adding ruby to the PLATFORMS section or x86_64-linux fixes the issue, but this is not immediately clear.

If not included with the output of your command, run bundle env and paste the output below

bundle env (on my mac)

## Environment

Bundler       2.2.3
  Platforms   ruby, x86_64-darwin-19
Ruby          3.0.0p0 (2020-12-25 revision 95aff214687a5e12c3eb57d056665741e734c188) [x86_64-darwin19]
  Full Path   /Users/rschneeman/.rubies/ruby-3.0.0/bin/ruby
  Config Dir  /Users/rschneeman/.rubies/ruby-3.0.0/etc
RubyGems      3.2.3
  Gem Home    /Users/rschneeman/.gem/ruby/3.0.0
  Gem Path    /Users/rschneeman/.gem/ruby/3.0.0:/Users/rschneeman/.rubies/ruby-3.0.0/lib/ruby/gems/3.0.0
  User Home   /Users/rschneeman
  User Path   /Users/rschneeman/.gem/ruby/3.0.0
  Bin Dir     /Users/rschneeman/.gem/ruby/3.0.0/bin
Tools         
  Git         2.28.0
  RVM         not installed
  rbenv       not installed
  chruby      0.3.9

Bundler Build Metadata

Built At          2020-12-22
Git SHA           29dc3c8398
Released Version  true

Bundler settings

jobs
  Set for the current user (/Users/rschneeman/.bundle/config): 8
build.nokogiri
  Set for the current user (/Users/rschneeman/.bundle/config): "--use-system-libraries"
gem.test
  Set for the current user (/Users/rschneeman/.bundle/config): "rspec"
gem.mit
  Set for the current user (/Users/rschneeman/.bundle/config): true
gem.coc
  Set for the current user (/Users/rschneeman/.bundle/config): true
--path
  Set for the current user (/Users/rschneeman/.bundle/config): "vendor/"

Gemfile

Gemfile

source 'https://rubygems.org'
gem 'rake'

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    rake (13.0.3)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  rake

BUNDLED WITH
   2.2.3

bundle env (on docker)

## Environment

Bundler       2.1.4
  Platforms   ruby, x86_64-linux
Ruby          2.7.2p137 (2020-10-01 revision 5445e0435260b449decf2ac16f9d09bae3cafe72) [x86_64-linux]
  Full Path   /usr/local/bin/ruby
  Config Dir  /usr/local/etc
RubyGems      3.1.4
  Gem Home    /usr/local/bundle
  Gem Path    /root/.gem/ruby/2.7.0:/usr/local/lib/ruby/gems/2.7.0:/usr/local/bundle
  User Home   /root
  User Path   /root/.gem/ruby/2.7.0
  Bin Dir     /usr/local/bundle/bin
Tools
  Git         2.20.1
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2021-01-11
Git SHA           unknown
Released Version  false

Bundler settings

jobs
  Set for your local app (/usr/local/bundle/config): 4
without
  Set for your local app (/usr/local/bundle/config): [:development, :test]
  Set via BUNDLE_WITHOUT: [:development, :test]
app_config
  Set via BUNDLE_APP_CONFIG: "/usr/local/bundle"
deployment
  Set via BUNDLE_DEPLOYMENT: true
silence_root_warning
  Set via BUNDLE_SILENCE_ROOT_WARNING: true
path
  Set via BUNDLE_PATH: "vendor/bundle"

Gemfile

Gemfile

source 'https://rubygems.org'
gem 'rake'

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    rake (13.0.3)

PLATFORMS
  x86_64-darwin-19

DEPENDENCIES
  rake

BUNDLED WITH
   2.2.3

[deivid-rodriguez]

Yes, this is correct.

Sorry, I've been going back and forth here trying to get a correct (and as fast as previously) resolution, while keeping this thing usable, but yeah right now if you want to develop under a platform and deploy to a different one (linux in this example), you'd need to run bundle lock --add-platform <platform_to_deploy_to> and commit the resulting lockfile.

I could try restoring the RUBY platform as a fallback but you wouldn't get any platform specific gems unless you explicitly add the platform you want to deploy to.

[schneems]

Thanks for the ultra fast response!

Workaround

I think in the short term I'm going to try to upgrade Heroku customers to Bundler 2.2.3 which provides a warning for this out of the box.

remote: -----> Installing dependencies using bundler 2.2.3
remote:        Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4
remote:        Your bundle only supports platforms ["x86_64-darwin-19"] but your local platform
remote:        is x86_64-linux. Add the current platform to the lockfile with `bundle lock
remote:        --add-platform x86_64-linux` and try again.
remote:        Bundler Output: Your bundle only supports platforms ["x86_64-darwin-19"] but your local platform
remote:        is x86_64-linux. Add the current platform to the lockfile with `bundle lock
remote:        --add-platform x86_64-linux` and try again.

Then customers will be unblocked in the short term (provided I don't have to rollback due to a yet-to-be-found regression).

Future musings

As a feature of bundler, I want to be able to either have bundler default to linux as a secondary platform or have some way of specifying it globally.

This change in behavior with 2.2.3 means a lot of existing tutorials for deploying Ruby apps now no longer work. Ideally, I would like to get them back to being valid as many are not maintained and will never be updated. For those that can be updated, it would be ideal to not have to add an extra step.

I'm happy to help brainstorm here or provide support if I can.

[deivid-rodriguez]

Right, I think I can probably restore the old behaviour as a last resort without losing any of the good stuff of locking the specific platform in the lockfile. I'll work on it tomorrow and keep you posted.

[schneems]

Thanks! We have code on the Heroku buildpack that deletes the Gemfile.lock if we see it's locked to the windows platform. It would be exciting to get rid of that functionality and get deterministic deploys for devs using windows. I appreciate you working on this area.

[deivid-rodriguez]

Hi @schneems!

So, I've been sleeping on this issue, and I think what we have now is quite good to be honest. Let me try to explain why we did this, and why I believe it's a good thing.

Context

In previous bundler versions, bundler didn't consider platforms for resolution at all. What bundler would previously do is to resolve dependencies without considering platforms, and then at installation time, pick up platform specific variants with the same version as the resolved version if they exist. That approach had several problems:

• Resolution correctness. Resolution would be incorrect sometimes, since it can happen that for the same gem and version number, a platform specific variant can have different dependencies than the standard variant. Without considering the specific platform for resolution, we might not get a valid set of dependencies, since new unconsidered constraints can be introduced after resolution. This is rare, but not that rare. For example, nokogiri recently released prerelease versions meeting this condition.

• Safety. Not only resolution is invalid, but the previous approach meant resolution was not actually fully locked. Say you're using foo-1.2.0 in your application. You have a lockfile specifically locking foo to 1.2.0, you test it in CI and in your staging environment, and it's all good. Then someone pushes foo-1.2.0-x86_64-linux to rubygems.org, a platform specific variant for foo 1.2.0, that, for example, has a critical bug, or that was pushed by a malicious actor. Then when you deploy to production, foo-1.2.0-x86_64-linux will be installed because it matches the running platform more closely. To me this is totally unexpected, if you have a Gemfile.lock file, no third party release should be able to change the set of third party code that you run.

Resolving for the specific running platform and recording the exact resolution in the lockfile fixes the above issues.

Current situation

• I believe we can improve on it but the current error message is not too bad, and allows for easily fixing the issue.

• This error will only happen if you are using bundler in frozen mode (with BUNDLE_FROZEN or BUNDLE_DEPLOYMENT). If you're not using frozen mode, then bundler will automatically re-resolve using the running platform if it's not already in the lockfile. If you are using frozen mode, I believe it means that you explicitly want to avoid the second issue I mentioned above, so... an error if that can't be guaranteed seems appropriate.

• Yes, some tutorials could be broken by this, but only under some circumstances:

  • If the tutorial provides a Gemfile.lock then bundler will respect that. Meaning, if the lockfile was generated with previous bundler versions, bundler will still fallback to how it worked in previous versions. But if you generate a lockfile from scratch, then it will use the more secure and correct mode.
  • If the tutorial does not set frozen mode, then bundler will re-resolve and just work as mentioned above.
  • If the two above are not met, yes, things can break, but just like they could break if there's a new Rails release or whatever dependencies the tutorial uses. I believe that's acceptable, but I'm willing to help updating any popular tutorials that we detect to be affected by this.

Please let me know what you think.

[schneems]

I think that's reasonable to keep this in there. As you mentioned it's more specific and safer. I am interested though in polishing the "out of the box" experience.

I'm thinking to how bundler was added to Ruby by default so the annoyance of hitting the no such command "bundle" doesn't get hit. It's a relatively small improvement as it only needs to be installed once per ruby version, but the "out of the box" experience is dramatically improved (in my opinion).

In my ideal world, a project with a Gemfile.lock would "just work ™️" on deployment, so it seems that ideally, we could find some way to get the x86_64-linux platform in there either with lower or no cost to the developer.

Spitballing some ideas:

• Prompt the user when there's only one platform and it's windows or mac
• Allow for setting a global config: bundle config --global set deploy_platform_default x86_64-linux
• Add x86_64-linux by default out of the box and allow users to overwrite the default
• Some combination of the above?

Do any of those sound remotely appealing or workable?

What I'm striving for is that if a developer does every step to create an app and deploy it "by the book" that it works "out of the box". Like this sentiment: https://twitter.com/searls/status/1344277289840873472.

If not I would be interested to hear more about your considerations. Also if you prefer we could move this over to slack or a zoom/hangout chat if you want higher fidelity. I'm happy to bounce ideas around if you think it could be helpful.

[schneems]

Also some feedback from a customer who develops on Linux already.

I asked if this fixed their issue:

bundle lock --add-platform x86_64-linux
bundle install
git add . ; git commit -m fix

They responded:

No, this doesn't fix it as it doesn't change my Gemfile.lock at all. I'm working locally on Ubuntu, so this platform is already there. The one that is missing is "ruby" and this fixes it for me: "bundle lock --add-platform ruby"

BTW, I tested with bundler 2.2.5

Before (does not work on heroku)

PLATFORMS
x86_64-linux

After (works on heorku)

PLATFORMS
ruby
x86_64-linux

[deivid-rodriguez]

@schneems I'll respond to your comments as soon as I can, but if you can report that issue separately in the mean time, I appreciate it.

[schneems]

Can do. Thanks!

[deivid-rodriguez]

Hi @schneems.

I'm not opposed to adding some configuration so that users can specify the set of platforms they want to support. I had the idea of adding a platforms DSL in the Gemfile where people can specify the target platforms they want to support.

We currently have the issue though that the resolver can handle platforms correctly, but at the same time, resolving for multiple platforms by default is sometimes very inefficient. That's why we currently resolve only for the current platform and if the user wants to add more platforms, for example, to get ready for deploying under a different platform that the one they are developing on, then she can add them manually. I'm currently working on fixing these issues so that I can at least restore resolving for the RUBY platform by default as a fallback.

Until I can fix these issues, would adding the deployment platform on the fly to the lockfile if not present work for you? Basically, run bundle lock --add-platform x86_86-linux before setting deployment mode and running bundle install, so that you make sure the platform we are deploying too is present in the lockfile, no matter what. This solution would also work for Gemfiles generated on Windows I believe.

[schneems]

That's do-able. We've got some deploy freezes right now, but I can work something up. My concern is that a temporary workaround ends up being permanent. Like the same way we have to strip out BUNDLE WITH information from the Gemfile.lock.

I think adding this platform on our end defeats the security and resolution-correctness purpose you mentioned before as they can get different versions in prod than at development time. I would perhaps be easier to just not enable BUNDLE_DEPLOYMENT=1 env var. Though worse (for safety/predictability concerns). In the long run I do want to be able to support windows deployment without deleting the Gemfile.lock. If I'm adding x86_64-linux still then I think I'm pretty much back to where we are today.

Options on my end

Here's some brainstorming about options I can take moving forwards with the consequences (good and bad) listed below each. For reference deploying, CodeTriage takes 2m0.919s which also includes running a release phase

Do nothing (zeroeth option)

• Preserves dev/prod parity behavior
• Requires the user take an extra step for successful deploy
• Adds ~0 seconds to deploy
• No feedback to the user is frustrating, a very bad experience

Turn off "deployment mode" on Heroku

• Does NOT preserve dev/prod dependency behavior
• Does NOT Require the user take an extra step for successful deploy
• Adds ~0 seconds to deploy
• Introduces other security holes such as someone updating their gemfile but not running bundle install and trying to deploy. A bad experience.

Auto-add x86_64-linux before bundle install on Heroku

• Does NOT preserve dev/prod dependency behavior
• Does NOT Require the user take an extra step for successful deploy
• Gets us close to where we are today for < bundler 2.2 generated lockfiles for user experience, also for security dev/prod behavior
• Adds ~4 seconds to deploy (for CodeTriage sized Gemfile.lock) [this is a 3.0% increase in deploy time]

$ time bundle lock --add-platform x86_64-linux --add-platform ruby
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies.....
Writing lockfile to /Users/rschneeman/Documents/projects/codetriage/Gemfile.lock

real  0m4.263s

• We possibly could only run this conditionally based on the output of bundle platform which takes 0.3 seconds to execute

Upgrade bundler to 2.2.5 for all users on Heroku

• Preserves dev/prod dependency behavior
• Does Require the user take an extra step for successful deploy
• Adds ~0 seconds to deploy
• Gives users a better error message telling them what to do
• Bundler upgrades historically cause platform issues which is why we restrict bundler versions
  • We want to eventually upgrade users, but need for 1 in a million bugs to be sussed out and fixed (as they happen quite often for us).

Detect bundle platform and error/warn on Heroku

• Preserves dev/prod dependency behavior
• Does NOT Require the user take an extra step for successful deploy
• Adds 0.3 seconds to every deploy [0.25% increase in deploy time]
• Gives users a better error message telling them what to do
• Output is not delivered in a machine readable standard, so there may be complications regexing it

Recap

I'm not sure. There's tradeoffs with all of these. I think I'll need to explore multiple options forward. Likely some combination of:

• Upgrading bundler version
• Detecting/warning on bundle platform
• Auto-adding some platforms based off of bundle platform though I'm still hesitant to do this for fear it becomes a long term expected feature of the platform. I'll have to couple this with heavy warnings.