WIP: ALPN negotiation

[HoneyryderChuck]

This MR adds support for using the ALPN extension from bouncycastle (and when using jsse.BCSSLEngine). It:

• updates BC to 1.69;
• adds an alpn negotiation test from ruby-openssl;
• implements the necessary SSLSocket and SSLContext ALPN ruby APIs:
  • OpenSSL::SSL::SSLSocket#alpn_protocol
  • OpenSSL::SSL::SSLContext#alpn_protocols (rw)
  • OpenSSL::SSL::SSLContext#alpn_select_cb
• adds the BCSSLEngine client/server APIs.

Pending

• Fix OpenSSL::SSL::SSLContext#initialize (OpenSSL::SSL::SSLContext.new("Unsupported") should raise exception instead)
• Fix SSL provider selection (should jruby switch support to jsse only?)
• Fix cipher suite (more below)

how to reproduce

After building the jar for tests (via rake test_prepare):

> JAVA_OPTS="-Djruby.openssl.ssl.provider=BC" bundle exec ruby -Itest src/test/ruby/ssl/test_session.rb --name /alpn_protocol_selection_ary/

why it's not working yet

The test doesn't run yet. The error I get is:

org.bouncycastle.tls.TlsFatalAlert: handshake_failure(40); No selectable cipher suite
        at org.bouncycastle.tls.AbstractTlsServer.getSelectedCipherSuite(AbstractTlsServer.java:438)
        at org.bouncycastle.jsse.provider.ProvTlsServer.getSelectedCipherSuite(ProvTlsServer.java:529)
        at org.bouncycastle.tls.TlsServerProtocol.generateServerHello(TlsServerProtocol.java:570)
        at org.bouncycastle.tls.TlsServerProtocol.handleHandshakeMessage(TlsServerProtocol.java:902)
        at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(TlsProtocol.java:635)
        at org.bouncycastle.tls.TlsProtocol.processRecord(TlsProtocol.java:524)
        at org.bouncycastle.tls.RecordStream.readFullRecord(RecordStream.java:207)
        at org.bouncycastle.tls.TlsProtocol.safeReadFullRecord(TlsProtocol.java:830)
        at org.bouncycastle.tls.TlsProtocol.offerInput(TlsProtocol.java:1210)
        at org.bouncycastle.tls.TlsProtocol.offerInput(TlsProtocol.java:1178)
        at org.bouncycastle.jsse.provider.ProvSSLEngine.unwrap(ProvSSLEngine.java:464)
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634)
        at org.jruby.ext.openssl.SSLSocket.readAndUnwrap(SSLSocket.java:715)
        at org.jruby.ext.openssl.SSLSocket.doHandshake(SSLSocket.java:580)
        at org.jruby.ext.openssl.SSLSocket.acceptImpl(SSLSocket.java:374)
        at org.jruby.ext.openssl.SSLSocket.accept(SSLSocket.java:326)

I suspect this has to do with the advertised cipher suites in jruby-openssl (OpenSSL::SSL::SSLContext.new(:"TLSv1_2").ciphers.each { |c| p c}.sort only returns "TLSv1/SSLv3" ciphers). I looked at the implementation in CipherStrings.java, and a lot seems to be outdated in regards to mapping with openssl's "SSL_CIPHER", since the constants are matching v1.0.0 .

As this definitely increases the scope of what I thought was necessary to implement ALPN negotiation, would greatly welcome some pointers on how to best overcome this, as I'm not so familiar with Java SSL APIs, and the cipher selection part of openssl in general.

cc @kares @headius

Closes #217

[HoneyryderChuck]

@kares any feedback? Understand you might be busy, just wanted to deal with any feedback while I didn't lose the context yet :)

[kares]

Hey Tiago, sorry I wanted to get back to you but too much stuff going on.

Basically I have 3 things here:

• BC update isn't related to ALPN, is it? there are issues with 1.69 so we should do that separately
• having the BC SSL provider part of JOSSL was intended to eventually be casted down (if used) to be able to impl OpenSSL APIs. so I have no problem with that -> but default (Sun) Java SSL engine should continue to work
• having said the previous I believe ALPN can be implemented with the official javax.net.SSLSocket APIs including the callback (haven't tried yet), wdyt?

A blocker for the last might be that we still compile against Java 7 but there's a branch (next) where this is updated already - still trying to get CI in shape.

[HoneyryderChuck]

hey @kares , thx for getting back.

BC update isn't related to ALPN, is it? there are issues with 1.69 so we should do that separately

It is, at least 1.69 release notes mentions "extended ALPN support recently added to Java 8". Which issues exactly? I could definitely make it a separate PR, if you think it's best.

but default (Sun) Java SSL engine should continue to work

I think it still does, i.e. if you don't opt in to JSSE provider, it'll just work as before, minus the ALPN support.

having said the previous I believe ALPN can be implemented with the official javax.net.SSLSocket APIs including the callback (haven't tried yet), wdyt?

Can you point me to the docs? I didn't think that's possible when I checked, but can reassess my patch if it means supporting ALPN for all providers.

A blocker for the last might be that we still compile against Java 7 ...

Perhaps there's a way to feature flag it by java version? Don't know exactly how to do it though.

There's also the question of TLS 1.2 compatible cipher suites, which I have no idea how to patch. Does any of your ongoing changes perhaps deal with it?

[kares]

Hey, finally managed to look into ALPN and try your PR.
Code base was recently updated to Java 8 so that allowes us to rely on some of the added SSL APIs - which pretty much do what the BC APIs are to achieve here.

I moved the work into a separate PR #247 let me know if you see anything missing or if you have any feedback/concens. I am okay with the BC update but would prefer to do it separately atm.

[HoneyryderChuck]

@kares thx for the update! I just had a glance at the PR and it looks 👌 ready to go. One test is failing, but it seems unrelated?

About the BC update, agree it can be done separately, I just thought it was necessary for Alpn initially.

[kares]

Thanks for the initial implementation Tiago, I am closing this WiP PR is favor of #247