Support OpenSSL 3.0

[rhenium]

OpenSSL 3.0 is scheduled to be released later this year. It is a major version bump from 1.1 and contains architecture changes that affect Ruby/OpenSSL.

From https://www.openssl.org/policies/releasestrat.html:

The following alpha and beta releases for OpenSSL 3.0 are currently scheduled. Note that these dates are subject to change and alpha or beta releases may be inserted or removed as required:

alpha1, 2020-03-31: Basic functionality plus basic FIPS module
alpha2, 2020-04-21: Complete external provider support (serialization, support for new algs, support for providers which only include operations in a class)
alpha3, 2020-05-21: Aiming to test the API completeness before beta1 freezes it)
beta1, 2020-06-02: Code complete (API stable, feature freeze)
betaN: Other beta releases TBD
Final: 2020 early Q4

The design is outlined in the web page:

https://www.openssl.org/docs/OpenSSL300Design.html

Unlike OpenSSL 1.0 -> 1.1, not so many changes are required to make it just compile, but a lot of deprecation warnings are generated while compiling and many test cases are currently failing when compiled against OpenSSL's master.

• OpenSSL::HMAC needs a rewrite with the EVP API as it currently uses the low-level HMAC_*() functions.
• Subclasses of OpenSSL::PKey, such as RSA or DSA, provide access to those low-level functions. The following methods need rewrite:
  • PKey::*#generate (and an overload of .new)
    • This can probably be implemented in pure-Ruby with PKey.generate_parameters and PKey.generate_key.
  • Low-level sign/verify methods that take prehashed values.
    • RSA#{private,public}_{encrypt,decrypt}
    • DSA#syssign and #sysverify
    • EC#dsa_sign_asn1 and #dsa_verify_asn1
• The ENGINE API is deprecated in favor of "Provider"s.

[rhenium]

Updated TODO list as of alpha14:

• OpenSSL::HMAC, Use the EVP API

  • No incompatibilities expected for Ruby programs.
  • Done by hmac: migrate from the low-level HMAC API to the EVP API #371.

• OpenSSL::PKey, Parameters/key generation with the EVP API

  • No incompatibilities expected.
  • Done by pkey: use high level EVP interface to generate parameters and keys #397.
    • Added OpenSSL::PKey.generate_key and OpenSSL::PKey.generate_parameters.
    • Rewritten OpenSSL::PKey::{RSA,DSA,DH}.{new,generate}.

• OpenSSL::PKey::*, Low-level methods for signature/encryption/key agreement

  • RSA#{private,public}_{encrypt,decrypt}, DSA#syssign, DSA#sysverify, EC#dsa_sign_asn1, and EC#dsa_verify_asn1.
  • No incompatibilities expected.
  • Done by pkey: implement PKey#encrypt, #decrypt, #sign_raw, #verify_raw, and #verify_recover #382.
    • Added OpenSSL::PKey.{encrypt,decrypt,sign_raw,verify_raw,verify_recover}.

• OpenSSL::PKey::*, Getters for parameters/key components

  • RSA#{n,e,d,p,...}, etc.
  • No incompatibilities expected.
  • Needs a rewrite with EVP_PKEY_get_params() family (only exists in 3.0)

• OpenSSL::PKey::*, Setters for parameters/key components

  • {RSA,DSA,DH}#set_* and EC#{private_key=,public_key=,group=}
  • Feature removed without replacement. Keys are now immutable once created - all components must be specified at once.
  • Use cases definitely exist, a new interface is required.
    • EVP_PKEY_fromdata() requires the caller to specify what the pkey is: parameters only, public key only, or private key?
    • OpenSSL::PKey.new_private_key("RSA", n: 123, e: 456, d: 789)?

• OpenSSL::PKey::{RSA,DSA}, Private/public key decoders for unpopular/non-standard formats

  • OpenSSL::PKey::DSA.new accepts PEM encoded "DSAPublicKey" format ("DSA PUBLIC KEY" header).
  • OpenSSL::PKey::RSA.new accepts DER/PEM encoded "RSAPublicKey" format ("RSA PUBLIC KEY" header).
  • ruby-openssl has no ability to export keys in such formats as of [Bug #4422].
  • Would it be OK to drop them? We need to write our decoder using OpenSSL::ASN1.decode if we want to keep it.

• OpenSSL::PKey::*, Methods that need a rewrite, which can be done with existing (<= 1.1.1) API

  • #to_text
  • #public_key (the method to create a copy with private components removed)
  • DH#params_ok? and EC#check_key
  • Done in Use EVP API in more places #436.

• OpenSSL::PKey::*, Methods that need a rewrite, which can be done in a compatible way, but requires new functions in 3.0.0

  • #params
  • #initialize_copy - annoying because EVP_PKEY_dup() only exists in OpenSSL 3.0.
  • #private? and #public?

• OpenSSL::PKey::{DH,EC}, #generate_key!

  • Feature removed without replacement. Keys are now immutable once created.
  • Print deprecation warning and suggest using OpenSSL::PKey.generate_key(pkey_obj_without_key_components).

• OpenSSL::PKey::EC::Group and OpenSSL::PKey::EC::Point

  • Effectively removed without replacement, though most of the functions are not explicitly marked as deprecated.
  • Because the low-level key generation is deprecated and the EVP API can't create keys using an EC_GROUP object, there is no use for this class.
  • Information that used to be obtained through EC_GROUP object (curve's name or parameters) are directly obtained from EVP_PKEY object with EVP_PKEY_get_params().

• OpenSSL::Engine

  • The entire ENGINE API is deprecated in favor of Providers, but it's new and is a completely different concept.
  • ENGINE should work if we ignore the deprecation warnings, but I doubt if there will be any ENGINE implementations that work with OpenSSL 3.0 anyway.

• OpenSSL::SSL, tmp_dh_callback

  • The callback used on the server side for ephemeral DH. Called during the handshake, but there is no reason for it to be a callback (anymore).
  • Replaced by SSL_CTX_set_tmp_dh() or SSL_CTX_set_dh_auto().
  • Added SSLContext#tmp_dh= by ssl: add SSLContext#tmp_dh=  #459.

[zzak]

This is awesome, thanks for your hard work @rhenium!

[pvalena]

I've tried to apply PR #399 on top of Ruby 3.0.1, and I'm experiencing a segfault on running test suite (failed when reading key). Any ideas how to fix that? Is that still expected?

I'll appreciate any help.