`rb_protect` is incompatible with fiber scheduler.

[ioquatix]

openssl/ext/openssl/ossl_pkey.c

Line 72 in c310840

obj = rb_protect((VALUE (*)(VALUE))pkey_new0, (VALUE)pkey, &status);

In some OpenSSL tests in Async::IO we sometimes create certificates in non-blocking fiber. Because of this, some operations will internally call Fiber#transfer. In this case, it will fail:

 0.35s    error: Async::Task [oid=0x12e8] [ec=0x12fc] [pid=64428] [2021-05-08 12:22:16 +1200]
               |   FiberError: fiber called across stack rewinding barrier
               |   → /Users/samuel/Documents/socketry/async/lib/async/scheduler.rb:114 in `transfer'
               |     /Users/samuel/Documents/socketry/async/lib/async/scheduler.rb:114 in `block'
               |     /Users/samuel/.gem/ruby/3.1.0/gems/async-rspec-1.16.0/lib/async/rspec/ssl.rb:118 in `initialize'
               |     /Users/samuel/.gem/ruby/3.1.0/gems/async-rspec-1.16.0/lib/async/rspec/ssl.rb:118 in `new'
               |     /Users/samuel/.gem/ruby/3.1.0/gems/async-rspec-1.16.0/lib/async/rspec/ssl.rb:118 in `block (3 levels) in <module
               |     /Users/samuel/.gem/ruby/3.1.0/gems/async-rspec-1.16.0/lib/async/rspec/ssl.rb:118 in `collect'
               |     /Users/samuel/.gem/ruby/3.1.0/gems/async-rspec-1.16.0/lib/async/rspec/ssl.rb:118 in `block (2 levels) in <module

The line in Async::RSpec that fails is here:

https://github.com/socketry/async-rspec/blob/88f72bcb6df56c5f13f799484219b0bd8f66b703/lib/async/rspec/ssl.rb#L118

It seems calling OpenSSL::PKey::RSA.new(2048) in a non-blocking fiber will fail.

cc @ko1 @rhenium

[rhenium]

Duplicate of #415? I suspect it has more to do with rb_thread_call_without_gvl().

Note that OpenSSL::PKey::RSA.new(num) was recently rewritten around OpenSSL::PKey.generate_key("RSA", rsa_keygen_bits: num). It still does the same thing, though.

[ioquatix]

Oh yeah I think it's a duplicate.