[BUG] OpenID certificate improperly validated causing 401 Unauthorized due to openjdk bug

[cyanidium]

Describe the bug

If you have an OpenID provider that is using a certificate which contains at least one nameConstraint with a leading . (likely in the issuer chain rather than the leaf certificate), then java will fail to validate the certificate and OpenID authorization in OpenSearch will fail, returning 401 Unauthorized.

I have been troubleshooting this issue for more than a week and read through dozens of conversations where people report OpenID authorization issues with OpenSearch, many of which were never fully resolved. In some cases, the same configuration appears to work for some users but not others. With sufficient logging enabled (see replication below) there are two stack traces that appeared for me. Most conversations report similar certificate path validation exceptions, but that is the second stack trace in this case. The earlier exception that gives this case away is: CertPathValidatorException: name constraints check failed.

In my case I have an internal certificate authority for the internal domain, and have cert-manager configured as an intermediate authority for the app.internal domain with a nameConstraint of DNS:.app.internal. That then gets issued to the ingress/gateway/proxy that sits in front of a KeyCloak instance at sso.app.internal. Setting the openid_connect_url value to https://sso.app.internal/realms/internal/.well-known/openid-configuration and configuring opensearch-dashboards to use the same provider results in the dashboards correctly redirecting, but receiving a 401 Unauthorized after being redirected back from the OpenID login.

I verified that the internal users, roles, and role mappings were all configured correctly, that the certificate authority certificates were provided correctly in the configuration, and dozens of other suggestions. The single change that enabled OpenID to work correctly was to reissue the intermediate authority certificate without the nameConstraint and then reissue the KeyCloak certificate. SSO started working without needing any changes to OpenSearch once the certificate was deployed to KeyCloak.

Related component

Build

To Reproduce

1. Configure an OIDC provider with a HTTPS certificate that contains at least one nameConstraints with a leading . such as DNS:.example.com in the issuer chain (detailed instructions in the upstream fix)
2. Configure OpenSearch to use the provider with openid_connect_url
3. Attempt to log in using OpenID
4. Receive 401 Unauthorized

Additionally, you can configure log4j2.properties to see more details from the cluster:

status = error
appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n
rootLogger.level = info
rootLogger.appenderRef.console.ref = console
logger.securityjwt.name = com.amazon.dlic.auth.http.jwt
logger.securityjwt.level = trace

And you should see at least one stack trace (I usually see 2 for each failed log in), the first includes the error CertPathValidatorException: name constraints check failed. I unfortunately don't have a full copy of this any more.

Expected behavior

Users from OpenID providers hosted with certificates containing nameConstraints in any part of the issuer chain should be able to authenticate with OpenSearch.

Additional Details

Additional context
This is an issue with upstream openjdk that has been fixed and backported to to 22+. The container images for OpenSearch 2.17.1 indicate the openjdk 21.0.4 is bundled, which does not contain the fix.

What needs to happen now?
This will continue to be an issue until a patched openjdk is bundled with OpenSearch. In the meantime, hopefully this bug appears in search results for others hitting the same issue so they don't have to spend days troubleshooting it.

[dblock]

[Catch All Triage - 1, 2, 3, 4, 5]