Having Psych 5 installed raises an error during another gem's C-extension installation when parsing YAML

[tombruijn]

Transferring this issue from the Ruby issue tracker: https://bugs.ruby-lang.org/issues/19371

Describe the problem as clearly as you can

Summary

There's an issue on Ruby versions with Psych 4 installed by default (Ruby 2.6 through 3.1) after installing the Psych gem version 5. This problem occurs when a Ruby gem has a C-extension installation script that parses a YAML string.

I'm reporting it here and not with on the Psych gem repo, because it looks more like an issue with which Ruby C-extension is load during other gem's C-extension installation.

Background

I have a gem that parses a YAML string in the C-extension installation script, or it calls Gem.configuration[:http_proxy], which parses the .gemrc file as YAML. This triggers the error mentioned below.

This YAML parsing is done in the gem's ext/extconf.rb file. An example gem can be found in this repository: https://github.com/tombruijn/yaml-dummy-gem, see the ext/extconf.rb file.

The problem

On Ruby 3.1.3 Psych version 4 is installed by default. When it parses the YAML file, it will use Psych 4.

When Psych 5 is also installed on Ruby 3.1.3, it is no longer be able to parse the YAML file. The following error is raised:

$ bundle install
Fetching https://github.com/tombruijn/yaml-dummy-gem.git
Resolving dependencies...
Using bundler 2.3.7
Using yaml-dummy-gem 1.0.0 from https://github.com/tombruijn/yaml-dummy-gem.git (at main@a48852d)
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d/ext
/usr/local/bin/ruby -I /usr/local/lib/ruby/3.1.0 -r ./siteconf20230123-730-rmbnnl.rb extconf.rb
/usr/local/lib/ruby/3.1.0/psych.rb:459:in `parse_stream': undefined method `parse' for #<Psych::Parser:0x0000ffff8078c7f8
@handler=#<Psych::Handlers::DocumentStream:0x0000ffff8078c910 @stack=[], @last=nil, @root=nil, @start_line=nil, @start_column=nil, @end_line=nil, @end_column=nil,
@block=#<Proc:0x0000ffff8078c848 /usr/local/lib/ruby/3.1.0/psych.rb:399>>, @external_encoding=0> (NoMethodError)

      parser.parse yaml, filename
            ^^^^^^
	from /usr/local/lib/ruby/3.1.0/psych.rb:399:in `parse'
	from extconf.rb:3:in `<main>'

extconf failed, exit code 1

Gem files will remain installed in /usr/local/bundle/bundler/gems/yaml-dummy-gem-a48852dac33d for inspection.
Results logged to /usr/local/bundle/bundler/gems/extensions/aarch64-linux/3.1.0/yaml-dummy-gem-a48852dac33d/gem_make.out

  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build'
  /usr/local/lib/ruby/3.1.0/tempfile.rb:317:in `open'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in build_extensions'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each'
  /usr/local/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions'
  /usr/local/lib/ruby/3.1.0/bundler/source/path/installer.rb:34:in `post_install'
  /usr/local/lib/ruby/3.1.0/bundler/source/path.rb:244:in `generate_bin'
  /usr/local/lib/ruby/3.1.0/bundler/source/git.rb:194:in `install'
  /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in `do_install'
  /usr/local/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue'
  /usr/local/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in create_threads'

An error occurred while installing yaml-dummy-gem (1.0.0), and Bundler cannot continue.

In Gemfile:
  yaml-dummy-gem

Debugging results

The error is raised because the Psych::Parser#parse method cannot be found. In Psych version 4, this method is defined by the Psych C-extension.

In Psych version 5 the parse method is defined in the gem's Ruby code. This method calls a C function registered as the private _native_parse method, which is the renamed version of the parse C-function in Psych version 4.

From what I can tell, the Psych version 4 C-extension is no longer loaded when Psych version 5 is installed in this scenario. There is a mix up in which Psych gem version's C-extension is loaded during my dummy gem's C-extension installation. It load the Psych 4 Ruby code, with the Psych 5 C-extension∂.

I confirmed this by modifying the standard installed Psych gem's code on Ruby 3.1 (Docker image ruby:3.1), with the following the change, which prints true on error. This means the Psych 4 gem has the Psych 5 C-extension loaded where _native_parse is defined.

diff --git lib/psych.rb lib/psych.rb
index 42d79ef..1a690d2 100644
--- lib/psych.rb
+++ lib/psych.rb
@@ -452,6 +452,9 @@ def self.parser
   def self.parse_stream yaml, filename: nil, &block
     if block_given?
       parser = Psych::Parser.new(Handlers::DocumentStream.new(&block))
+      # This returns `true`, but it should be `false`. The `_native_parse`
+      # method is defined in the Psych 5 C-extension, not Psych 4.
+      puts parser.respond_to? :_native_parse, true # => true
       parser.parse yaml, filename

This error only occurs during the gem's extension installation in ext/extconf.rb. If the gem parses YAML when a Ruby app is running, it will not produce the same error with Psych version 5 installed.

This issue does not occur on Ruby 3.2, where Psych version 5 is installed by default.

I have confirmed this error occurs on the latest patch releases of the following Ruby versions: 3.1, 3.0, 2.7 and 2.6.

$ gem list psych
psych (5.0.2, default: 4.0.3)

(Also confirmed to error on psych version 5.1.0.)

Did you try upgrading RubyGems?

Yes.

Post steps to reproduce the problem

Here is a basic Ruby gem that only parses a YAML file during extension installation: https://github.com/tombruijn/yaml-dummy-gem

Here is a small project that triggers the error: https://github.com/tombruijn/yaml-dummy-ruby-app

A GitHub actions workflow shows the results for all affected Ruby versions: https://github.com/tombruijn/yaml-dummy-ruby-app/actions/runs/3969088933

The example app repo also has instructions to run the example app locally. Please follow the instructions in the README to see the error.

[deivid-rodriguez]

I think this may have been fixed by #6490?

[hsbt]

I try to replace Psych to Bundler::YAMLSerializer for Gem::ConfigFile now. or I'll add compatiblity layer for psych.rb. But It's only workaround.

[deivid-rodriguez]

Just to confirm that #6490 has fixed this? @tombruijn you said that you tried upgrading RubyGems but your demo repository does not show any upgrading of RubyGems? If this issue was the same thing as #6490, upgrading RubyGems should've fixed this because the issue was specific to the default copy of RubyGems bundled with Ruby (the upgraded copy of RubyGems is installed on a different path that would not suffer from the problem).

Maybe this is a completely different issue, but I just want to confirm because they seem related to me.

[tombruijn]

@deivid-rodriguez you're right the update fixed it! I updated it on the test app CI and it no longer fails for the affected Ruby versions. Thanks!

[cyclingzealot]

I try to replace Psych to Bundler::YAMLSerializer for Gem::ConfigFile now. or I'll add compatiblity layer for psych.rb. But It's only workaround.

May I ask how you achieved this workaround exactly? Is it by doing an entry in ~/.gemrc ? If so, what setting and what value may I ask?

Looking at https://docs.ruby-lang.org/en/master/Gem/ConfigFile.html, it's not clear which setting needs to be set and to what value.