Ruby google-protobuf linux precompiled gem crash with Segmentation fault on linux-musl

[ntkme]

What version of protobuf and what language are you using?
Version: 3.22.2 or higher on aarch64-linux-musl / 4.29.3 or higher on x86_64-linux-musl
Language: Ruby

What operating system (Linux, Windows, ...) and version?
Alpine Linux any version

What runtime / compiler are you using (e.g., python version or gcc version)
Any ruby version

What did you do?
Steps to reproduce the behavior:

/ # gem install google-protobuf
Successfully installed google-protobuf-4.26.1-aarch64-linux
1 gem installed
/ # ruby -e "require 'google/protobuf'"
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf_c (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:15:in `rescue in <top (required)>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:12:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': Error loading shared library ld-linux-aarch64.so.1: No such file or directory (needed by /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so) - /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/3.3/protobuf_c.so (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf_native.rb:13:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:57:in `<module:Protobuf>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:15:in `<module:Google>'
	from /usr/local/bundle/gems/google-protobuf-4.26.1-aarch64-linux/lib/google/protobuf.rb:14:in `<top (required)>'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
	from -e:1:in `<main>'
<internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- google/protobuf (LoadError)
	from <internal:/usr/local/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from -e:1:in `<main>'

What did you expect to see
google-protobuf should work on linux-musl just like how it works on linux-gnu

What did you see instead?
google-protobuf prebuilt library is strictly linked to glibc and it's not compatible with musl-libc

Anything else we should know about your project / environment
Please see rake-compiler's documentation regarding recent changes in rubygems in regarding to -linux, -linux-gnu, and -linux-musl platforms: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

[ntkme]

As a workaround, uninstalling the prebuilt gem and reinstall platform ruby to build from source fixes the issue:

/ # gem uninstall google-protobuf
Successfully uninstalled google-protobuf-4.26.1-aarch64-linux
/ # apk add alpine-sdk
...
/ # gem install --platform ruby google-protobuf
Fetching google-protobuf-4.26.1.gem
Building native extensions. This could take a while...
Successfully installed google-protobuf-4.26.1
1 gem installed
/ # ruby -e "require 'google/protobuf'"

For bundler, the following in Gemfile would work:

gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

[ntkme]

There are a few options here:

1. Do not ship aarch64-linux gem, and let both aarch64-linux-gnu and aarch64-linux-musl platforms fallback to platform ruby and compile from source.
2. Ship the current flavor, but label it as aarch64-linux-gnu, so that it does not get installed incorrectly on aarch64-linux-musl.
3. Upgrade rake-compiler, and ship separate aarch64-linux-gnu and aarch64-linux-musl gems.

Again, please refer to rake-compiler's documentation on details: https://github.com/rake-compiler/rake-compiler-dock?tab=readme-ov-file#linux-gnu-and-musl-important-details

[JasonLunn]

@dazuma - what's the expected platform support for linux-musl?

[ntkme]

As said early, even if we don't "officially" support linux-musl, we can avoid it from breaking out of box. - Just ship the gem with label -linux-gnu instead of -linux using latest rake-compiler.

[henrahmagix]

I'd really appreciate if @ntkme's comment could be actioned pls ☺️

ship the gem with label -linux-gnu instead of -linux using latest rake-compiler

[ntkme]

This issue used to be aarch64-linux-musl only.

However, with 4.29.3 release, that x86_64-linux prebuilt now triggers Segmentation fault on x86_64-linux-musl:

• google-protobuf 4.29.2 (x86_64-linux) worked on musl: https://github.com/sass-contrib/sass-embedded-host-ruby/actions/runs/12606460485/job/35136617853
• google-protobuf 4.29.3 (x86_64-linux) crashed on musl with Segmentation fault: https://github.com/sass-contrib/sass-embedded-host-ruby/actions/runs/12712732880/job/35438995154

This issue can be worked around by force compiling from source by adding the following in Gemfile:

gem 'google-protobuf', force_ruby_platform: true if RUBY_PLATFORM.include?('linux-musl')

@JasonLunn I highly suggest that we take action now to update rake-compiler and split linux-gnu and linux-musl gems as soon as possible.

[ntkme]

The statistics from my own gem that has 10M+ total download shows that roughly 10% of x86_64-linux are from linux-musl, which is definitely not a small number. E.g. As of writing, the download counts for the latest version show:

• https://rubygems.org/gems/sass-embedded/versions/1.83.1-x86_64-linux-gnu - 24,734
• https://rubygems.org/gems/sass-embedded/versions/1.83.1-x86_64-linux-musl - 2,983

The statistics for google-protobuf might be different, but the total download for google-protobuf is way higher that even only 5% would affect tons of users. I know lots of users in ruby community uses docker.io/library/ruby:alpine image, some even use alpine for production, thus it's critical to not break linux-musl out of box even if we decide to not ship linux-musl prebuilt - an error from bundler asking user to install a compiler is way better than seeing Segmentation fault.

[ntkme]

I was probably the earliest adaptor of splitting linux-gnu and linux-musl prebuilt, as I introduced it way before rake-compiler added support, using my own custom cross-platform build scripts. I have been through lots of trouble with rubygems/bundler's bug around this area in terms of platform detection and even helped fixing a few bugs in rubygems/bundler.

From my own experience with it, I can say that as of now it's reliable with ruby >=3.1 and the default rubygems/bundler or higher version from them. In fact, nokogiri, one of the most downloaded gem of all time, introduced split -linux-gnu and -linux-musl with 1.18.0 release this month: https://rubygems.org/gems/nokogiri/versions
sparklemotion/nokogiri@4c15c5c

Now it's time to adopt -linux-gnu and -linux-musl gems.