Apparently, there's a BUNDLE_BIN option to automatically create binstubs for installed gems. I was not aware of this setting, but we need to turn it off when setting up the custom bundle or else we accidentally create binstubs for the application.