Add the fallback logic to use 'ssl_engine' if 'ssl_handler' attribute is not available / compatible

[reta]

Description

Originally was posted on the OpenSearch forum [1], when transport-reactor-netty4 plugin is activated, some functionality related to opensearch-security plugin is not working properly.

$ ./plugins/opensearch-security/tools/securityadmin.sh -dg -cacert config/root-ca.pem  -cert config/node1.pem -w
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java
Listening for transport dt_socket at address: 5005
Security Admin v7
Will connect to localhost:9200 ... done
Listening for transport dt_socket at address: 5005
ERR: An unexpected ResponseException occured: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/2.0 403 Forbidden]
No security data
Trace:
org.opensearch.client.ResponseException: method [GET], host [https://localhost:9200], URI [/_plugins/_security/whoami], status line [HTTP/2.0 403 Forbidden]
No security data
        at org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:541)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:154)

It turned out the that issue was related to the SslHandler that opensearch-security tries to locate by name. The transport-reactor-netty4 plugin does use different naming convention and as such, the SslHandler has not being detected. Further testing uncovered that merely returning SslHandler may not be enough due to classloader issues (we have at least 3 places where SslHandler could be loaded from):

$ curl https://localhost:9200/_plugins/_security/whoami -kv
* Host localhost:9200 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:9200...
* Connected to localhost (::1) port 9200
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=localhost
*  start date: Sep 29 13:14:41 2025 GMT
*  expire date: Sep 29 13:14:41 2027 GMT
*  issuer: C=AU; ST=Some-State; O=Internet Widgits Pty Ltd; CN=localhost
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost:9200/_plugins/_security/whoami
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: localhost:9200]
* [HTTP/2] [1] [:path: /_plugins/_security/whoami]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /_plugins/_security/whoami HTTP/2
> Host: localhost:9200
> User-Agent: curl/8.5.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 500
< x-opensearch-version: OpenSearch/3.2.0 (opensearch)
< content-type: application/json; charset=UTF-8
< content-length: 705
<
* Connection #0 to host localhost left intact
{"error":{"root_cause":[{"type":"class_cast_exception","reason":"class io.netty.handler.ssl.SslHandler cannot be cast to class io.netty.handler.ssl.SslHandler (io.netty.handler.ssl.SslHandler is in unnamed module of loader java.net.URLClassLoader @58164e9a; io.netty.handler.ssl.SslHandler is in unnamed module of loader java.net.URLClassLoader @2d74cbbd)"}],"type":"class_cast_exception","reason":"class io.netty.handler.ssl.SslHandler cannot be cast to class io.netty.handler.ssl.SslHandler (io.netty.handler.ssl.SslHandler is in unnamed module of loader java.net.URLClassLoader @58164e9a; io.netty.handler.ssl.SslHandler is in unnamed module of loader java.net.URLClassLoader @2d74cbbd)"},"status":500}

To address that, adding the fallback attribute ssl_engine that return the instance of the SSLEngine directly (if available) since the SslHandler is not really needed. The fix works as expected, the output is identical to the default transport (netty4):

$ ./plugins/opensearch-security/tools/securityadmin.sh -dg -cacert config/root-ca.pem  -cert config/node1.pem -w
WARNING: nor OPENSEARCH_JAVA_HOME nor JAVA_HOME is set, will use /usr/bin/java
Security Admin v7
Will connect to localhost:9200 ... done
Connected as null
ERR: null is not an admin user
Seems you use a client certificate but this one is not registered as admin_dn
Make sure opensearch.yml on all nodes contains:
plugins.security.authcz.admin_dn:
  - "null"

[1] https://forum.opensearch.org/t/pods-not-coming-up-after-using-transport-reactor-netty4-plugin-for-mcp-server/26990

Issues Resolved

Complimentary to opensearch-project/OpenSearch#19458, see please https://forum.opensearch.org/t/pods-not-coming-up-after-using-transport-reactor-netty4-plugin-for-mcp-server/26990

Testing

Tested SNAPSHOT distribution, default implementation (netty4) is not affected

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwperks]

TY @reta ! FYI The CI checks are failing until #5659 is merged.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.93%. Comparing base (853c501) to head (ccd6f3b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5667      +/-   ##
==========================================
- Coverage   72.95%   72.93%   -0.03%     
==========================================
  Files         414      414              
  Lines       25869    25873       +4     
  Branches     3934     3934              
==========================================
- Hits        18874    18870       -4     
- Misses       5082     5093      +11     
+ Partials     1913     1910       -3     

Files with missing lines	Coverage Δ
.../opensearch/security/filter/OpenSearchRequest.java	78.94% <100.00%> (+5.61%)	⬆️

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[reta]

@derek-ho @DarshitChanpura @nibix may I ask you folks to take a look please? thank you