Can't bundle `sidekiq-ent` with latest Bundler

[mperham]

Please see sidekiq/sidekiq#6317, I wonder if this is related to the new bundler/setup autoinstall code when trying to use private gem repositories?

[wata727]

It seems to be an issue caused by #7560.

First of all, in my environment, the version of Bundler installed on the system and the version of Bundler installed in vendor/bundle are different:

# cat .bundle/config
---
BUNDLE_CLEAN: "true"
BUNDLE_PATH: "vendor/bundle"
# cat test.rb 
require "bundler"
puts Bundler::VERSION
# ruby test.rb
2.4.19
# bundle exec ruby test.rb
2.5.11

In this situation, if you call a binstub that does require "bundler/setup", an authenticated gem (e.g. sidekiq-pro) will not load:

# DEBUG=true bin/rails 
Could not find gem 'sidekiq-pro' in locally installed gems
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:910:in `block in verify_changed_sources!'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:908:in `each'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:908:in `verify_changed_sources!'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:899:in `source_requirements'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:513:in `resolution_packages'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:495:in `resolver'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:570:in `start_resolution'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:301:in `resolve'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:523:in `materialize'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:200:in `specs'
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:266:in `specs_for'
/usr/local/lib/ruby/3.2.0/bundler/runtime.rb:18:in `setup'
/usr/local/lib/ruby/3.2.0/bundler.rb:162:in `setup'
/usr/local/lib/ruby/3.2.0/bundler/setup.rb:10:in `block in <top (required)>'
/usr/local/lib/ruby/3.2.0/bundler/ui/shell.rb:159:in `with_level'
/usr/local/lib/ruby/3.2.0/bundler/ui/shell.rb:111:in `silence'
/usr/local/lib/ruby/3.2.0/bundler/setup.rb:10:in `<top (required)>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
/app/config/boot.rb:3:in `<top (required)>'
/app/bin/rails:4:in `require_relative'
/app/bin/rails:4:in `<top (required)>'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client/rails.rb:28:in `load'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client/rails.rb:28:in `call'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client/command.rb:7:in `call'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client.rb:30:in `run'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/bin/spring:49:in `<top (required)>'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/binstub.rb:11:in `load'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/binstub.rb:11:in `<top (required)>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
/app/bin/spring:10:in `block in <top (required)>'
<internal:kernel>:90:in `tap'
/app/bin/spring:7:in `<top (required)>'
bin/rails:2:in `load'
bin/rails:2:in `<main>'
Run `bundle install` to install missing gems.

Setting frozen=true makes the cause obvious. It seems that a gem cannot be resolved because Gemfile.lock is rewritten by Bundler v2.4.19 (not v2.5.11).

# bundle config set --local frozen true
# DEBUG=true bin/rails
The list of sources changed, but the lockfile can't be updated because frozen mode is set

You have changed in the Gemfile:
* sidekiq-ent from `rubygems repository https://enterprise.contribsys.com/` to `rubygems repository https://<REDACTED>@enterprise.contribsys.com/`
* sidekiq-pro from `rubygems repository https://enterprise.contribsys.com/` to `rubygems repository https://<REDACTED>@enterprise.contribsys.com/`

Run `bundle install` elsewhere and add the updated Gemfile.lock to version control.
If this is a development machine, remove the Gemfile freeze by running `bundle config set frozen false`.
/usr/local/lib/ruby/3.2.0/bundler/definition.rb:418:in `ensure_equivalent_gemfile_and_lockfile'
/usr/local/lib/ruby/3.2.0/bundler/runtime.rb:13:in `setup'
/usr/local/lib/ruby/3.2.0/bundler.rb:162:in `setup'
/usr/local/lib/ruby/3.2.0/bundler/setup.rb:10:in `block in <top (required)>'
/usr/local/lib/ruby/3.2.0/bundler/ui/shell.rb:159:in `with_level'
/usr/local/lib/ruby/3.2.0/bundler/ui/shell.rb:111:in `silence'
/usr/local/lib/ruby/3.2.0/bundler/setup.rb:10:in `<top (required)>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
/app/config/boot.rb:3:in `<top (required)>'
/app/bin/rails:4:in `require_relative'
/app/bin/rails:4:in `<top (required)>'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client/rails.rb:28:in `load'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client/rails.rb:28:in `call'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client/command.rb:7:in `call'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/client.rb:30:in `run'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/bin/spring:49:in `<top (required)>'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/binstub.rb:11:in `load'
/app/vendor/bundle/ruby/3.2.0/gems/spring-4.2.1/lib/spring/binstub.rb:11:in `<top (required)>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:86:in `require'
/app/bin/spring:10:in `block in <top (required)>'
<internal:kernel>:90:in `tap'
/app/bin/spring:7:in `<top (required)>'
bin/rails:2:in `load'
bin/rails:2:in `<main>'

[deivid-rodriguez]

Thanks for the nice investigation @wata727. I was able to reproduce the problem. It's not expected that Bundler fails to auto switch to the proper version in your case (binstubs + local bundler path). I created #7719 to fix this problem.