common: assume vendored resources are binary

[thepwagner]

Previously Dependabot included a list of binary mime types and encoded
only binary files.
Invert to include a list of acceptable text mime types to leave
un-encoded.

Per the included test, the edge case here is iso-8859-1 strings, which
aren't binary but also aren't valid utf-8.

Related

• Closes [ruby:bundler] Failing to create PRs with JSON::GeneratorError #3074

[jurre]

Nice one! I actually thought we'd already done this at some point 😄

[filipkis]

Thank you @thepwagner for this fix. Any estimate on when this will be deployed?

[thepwagner]

@filipkis sorry for the delay, this slipped over a long weekend here!

This is now live, as of: 0.133.3-23c9359e16a1afa23c9e39b780870bd72a429895 (since 2021-02-16 16:10:48 +0000)

[filipkis]

No worries, not a big delay :)

I run the dependabot on the test repo and while PRs do get create they have many files (instead of just one or couple of gems). It seems that gems somehow get unpacked. Here is an example

[thepwagner]

@filipkis isn't that expected?

Since json_vat uses a git dependency, "vendoring" means copying that repository's contents at the target commit.

$ docker run --rm -it ruby:2.5.5 bash
root@25d83f8cdd02:/# git clone https://github.com/filipkis/test-dependabot-bundler.git
root@25d83f8cdd02:/# cd test-dependabot-bundler/

# ... hack to remove bundler2 dependency, otherwise unchanged
root@25d83f8cdd02:/test-dependabot-bundler# git diff Gemfile.lock
diff --git a/Gemfile.lock b/Gemfile.lock
index b9c039f..d3911f0 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -689,6 +689,3 @@ DEPENDENCIES

 RUBY VERSION
    ruby 2.5.5p157
-
-BUNDLED WITH
-   2.1.4

# `bundle cache --all` vendors git dependencies, regardless of Dependabot:
root@25d83f8cdd02:/test-dependabot-bundler# bundle cache --all
NOTE: Gem::Specification#has_rdoc= is deprecated with no replacement. It will be removed on or after 2018-12-01.
Gem::Specification#has_rdoc= called from /usr/local/bundle/bundler/gems/premailer-214543a3b214/premailer.gemspec:8.
Updating files in vendor/cache
/test-dependabot-bundler/vendor/cache/premailer-214543a3b214/lib/premailer/version.rb:3: warning: already initialized constant Premailer::VERSION
/usr/local/bundle/bundler/gems/premailer-214543a3b214/lib/premailer/version.rb:3: warning: previous definition of VERSION was here
NOTE: Gem::Specification#has_rdoc= is deprecated with no replacement. It will be removed on or after 2018-12-01.
Gem::Specification#has_rdoc= called from /test-dependabot-bundler/vendor/cache/premailer-214543a3b214/premailer.gemspec:8.
root@25d83f8cdd02:/test-dependabot-bundler# git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   Gemfile.lock

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	vendor/cache/json-vat-6f6f3420f7ce/
	vendor/cache/libv8-8.4.255.0-x86_64-linux.gem
	vendor/cache/nokogiri-1.11.0-x86_64-linux.gem
	vendor/cache/premailer-214543a3b214/
	vendor/cache/spring-commands-rspec-434b4e7616b9/

no changes added to commit (use "git add" and/or "git commit -a")

Treating vendored git dependencies differently than vendored gems is a separate concern. Dependabot assumes all dependencies are fully vendored, or none are.

[filipkis]

Well, maybe :)

In our workflow I was before relying on only things that were added to vendor/cache on regular bundle install and that didn't use to add git dependencies. And then what confused me here is that these git dependencies are not related to this PR (but since they were not cached before, they appear in every PR created by dependabot).

So yes, I guess this is correct behaviour. And we'll just need to do the initial commit of these (or maybe add them to .gitignore).

Btw. is there any chance down the road this will be configurable (to exclude git dependancies)?