TLS support for auxiliary transports

[finnegancarroll]

Description

In core auxiliary transports may implement secure variants to provide security plugin functionality such as EIT, authN, and authZ. Today auxiliary transports are provided this interface in core but it remains un-implemented in security plugin. These changes enable security plugin to provide basic TLS support such that secure auxiliary transports can encrypt data in transit.

Configuration settings for each aux transport exist under distinct namespaces, where the identifier for the transport is the same as the key for enabling it under setting aux.transport.types.

plugins.security.ssl.aux.arrow.<setting option...>
plugins.security.ssl.aux.grpc.<setting option...>
plugins.security.ssl.aux.other.<setting option...>

Note that these changes provide no authorization component and only work to deliver an SSL configuration. Users can configure clientauth_mode to allow/disallow certificates access to a particular transport, but no permissions are enforced and no user is authenticated when a connection is successfully established.

Testing

These changes add mock aux transports to unit tests for SslContextHandler and include the OpenSearch core native gRPC transport plugin within integration test suites to verify TLS is functioning as expected. These changes can also be tested manually by compiling this feature branch and installing it on an OpenSearch distribution.

Env variables for convenience:

export PLAT=darwin
export SNAP_VERSION=3.2.0
export REPOS_DIR=<path to clone OS and security feature branches>

export OS_REPO=${REPOS_DIR}OpenSearch
export SEC_REPO=${REPOS_DIR}security
export SEC_TAR=${SEC_REPO}/build/distributions/opensearch-security-${SNAP_VERSION}.0-SNAPSHOT.zip
export GRPC_TAR=${OS_REPO}/plugins/transport-grpc/build/distributions/transport-grpc-${SNAP_VERSION}-SNAPSHOT.zip
export OS_INSTALL=${OS_REPO}/distribution/archives/${PLAT}-tar/build/install/opensearch-${SNAP_VERSION}-SNAPSHOT

Build OpenSearch/security plugin/gRPC plugin:

cd ${OS_REPO} && ./gradlew :distribution:archives:${PLAT}-tar:assemble 
cd ${OS_REPO} && ./gradlew plugins:transport-grpc:assemble
cd ${OS_REPO} && ./gradlew publishToMavenLocal
cd ${SEC_REPO} && ./gradlew :assemble

Install both plugins:

${OS_INSTALL}/bin/opensearch-plugin install file://${GRPC_TAR}
${OS_INSTALL}/bin/opensearch-plugin install file://${SEC_TAR}

Create demo certs for security plugin - Respond yes yes no (keeping cluster mode disabled):

export OPENSEARCH_INITIAL_ADMIN_PASSWORD=PrTestPass369char! && \
chmod +x "${OS_INSTALL}/plugins/opensearch-security/tools/install_demo_configuration.sh" && \
"${OS_INSTALL}/plugins/opensearch-security/tools/install_demo_configuration.sh"

Enable the secure gRPC auxiliary transport:

echo "aux.transport.types: experimental-secure-transport-grpc" >> ${OS_INSTALL}/config/opensearch.yml
echo "aux.transport.experimental-secure-transport-grpc.port: '9400-9500'" >> ${OS_INSTALL}/config/opensearch.yml

Configure security settings with demo certs (.pem format):
Note: This example configures clientauth_mode: REQUIRE.

echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.enabled: true" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.pemcert_filepath: esnode.pem" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.pemkey_filepath: esnode-key.pem" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.clientauth_mode: REQUIRE" >> ${OS_INSTALL}/config/opensearch.yml
echo "plugins.security.ssl.aux.experimental-secure-transport-grpc.pemtrustedcas_filepath: root-ca.pem" >> ${OS_INSTALL}/config/opensearch.yml

Start OpenSearch:

${OS_INSTALL}/bin/opensearch

Test plaintext connection:

grpcurl -plaintext localhost:9400 list

Test insecure TLS with no client certificate provided:

grpcurl -insecure localhost:9400 list

Test TLS with client certificate provided:
Note: -insecure to skip host name verification.

grpcurl -insecure -cert ${OS_INSTALL}/config/esnode.pem -key ${OS_INSTALL}/config/esnode-key.pem localhost:9400 list

Issues Resolved

#17795

Testing

Unit tests and integration tests for mock auxiliary transport CertTypes.

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

❌ Patch coverage is 77.58621% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.75%. Comparing base (8ad2641) to head (6be11f5).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...h/security/dlic/rest/api/ssl/CertificatesInfo.java	29.62%	17 Missing and 2 partials ⚠️
...nsearch/security/ssl/ExternalSecurityKeyStore.java	0.00%	10 Missing ⚠️
...rg/opensearch/security/ssl/SslSettingsManager.java	90.12%	3 Missing and 5 partials ⚠️
...a/org/opensearch/security/ssl/config/CertType.java	79.48%	6 Missing and 2 partials ⚠️
...lic/rest/api/ssl/CertificatesInfoNodesRequest.java	66.66%	1 Missing and 1 partial ⚠️
...org/opensearch/security/ssl/SslContextHandler.java	33.33%	1 Missing and 1 partial ⚠️
...ensearch/security/ssl/util/SSLConfigConstants.java	92.85%	0 Missing and 2 partials ⚠️
.../opensearch/security/ssl/config/SslParameters.java	92.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5375      +/-   ##
==========================================
+ Coverage   72.70%   72.75%   +0.05%     
==========================================
  Files         398      398              
  Lines       24657    24765     +108     
  Branches     3752     3760       +8     
==========================================
+ Hits        17927    18018      +91     
- Misses       4898     4912      +14     
- Partials     1832     1835       +3     

Files with missing lines	Coverage Δ
.../api/ssl/TransportCertificatesInfoNodesAction.java	95.34% <100.00%> (-0.95%)	⬇️
...ensearch/security/ssl/DefaultSecurityKeyStore.java	30.53% <100.00%> (ø)
.../security/ssl/OpenSearchSecureSettingsFactory.java	55.38% <100.00%> (+7.17%)	⬆️
...arch/security/ssl/OpenSearchSecuritySSLPlugin.java	87.50% <100.00%> (+0.10%)	⬆️
.../opensearch/security/ssl/config/SslParameters.java	81.48% <92.30%> (+4.55%)	⬆️
...lic/rest/api/ssl/CertificatesInfoNodesRequest.java	73.68% <66.66%> (ø)
...org/opensearch/security/ssl/SslContextHandler.java	96.80% <33.33%> (-2.10%)	⬇️
...ensearch/security/ssl/util/SSLConfigConstants.java	93.75% <92.85%> (+15.97%)	⬆️
...rg/opensearch/security/ssl/SslSettingsManager.java	82.38% <90.12%> (+4.17%)	⬆️
...a/org/opensearch/security/ssl/config/CertType.java	79.48% <79.48%> (-20.52%)	⬇️
... and 2 more

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[finnegancarroll]

@cwperks can you take a look when convenient?

[cwperks]

Thank you for enhancing (and fixing) the bwc tests in this repo with this change :)

[cwperks]

@DarshitChanpura @nibix wdyt about having aux transport tests in this repo and testing for EIT? I would be happy to accept the maintenance in this case and if there are other aux transports in the future (f.e. Apache arrow as an alternative to default netty) then we can find another appropriate place for the tests at a later time.

The changes in this PR should get merged to supply the settings, so the current point of contention is where the appropriate place for the tests is.

[nibix]

@cwperks

wdyt about having aux transport tests in this repo and testing for EIT? I would be happy to accept the maintenance in this case and if there are other aux transports in the future (f.e. Apache arrow as an alternative to default netty) then we can find another appropriate place for the tests at a later time.

Well, as a temporary solution, it's okay. Still, I would also think that it won't be sustainable in the long run, as each further transport has these effects:

• increase of dependencies, and thus increased chance of issues with dependencies (as we see right now)
• increase of test turnaround times
• increased chance of test failures due to external factors
• it's unclear what shall be done when a test fails

IMHO, the infrastructure that is provided by the security plugin should be tested by a minimal test implementation that is co-located with the tests.

Wouldn't it be possible to have the integration tests with security on the GRPC plugin side?

[cwperks]

Wouldn't it be possible to have the integration tests with security on the GRPC plugin side?

gRPC plugin is in core and currently none of the core plugins test with security...not saying its not possible but typically only plugins with their own repo test with security.

I see 2 solutions:

1. Introduce testing w/ security in core plugins
2. Move gRPC out of core to a separate repo

[cwperks]

@finnegancarroll any thoughts on the latter of moving grpc plugin out of core and to its own repo?

[nibix]

Ah, I did not know that gRPC is part of the core project. That changes the situation a bit, I thought it was a separate repo.

[nibix]

I am wondering: what are the criteria to decide what goes to the core repo and what goes to its own repo?

[cwperks]

I am wondering: what are the criteria to decide what goes to the core repo and what goes to its own repo?

That's an art, not a science ;). In all seriousness, I agree that we should have criteria somewhere to help people distinguish when to do either.

My general guidance is this: if you want a separate set of maintainers go with a separate repo. If the core maintainers agree to do the maintenance then it can go in core. So really it boils down to who does the maintenance..I think.

[cwperks]

IMO Workload Management is another area that was introduced in the core that might be better as a separate plugin repo as they are also talking about doing testing with security.

For example, they are assigning requests to workload groups and some of the matching rules involve user attributes like username and backend roles.

CC: @kkhatua @kaushalmahi12 @jainankitk you may be interested in this conversation

[finnegancarroll]

@finnegancarroll any thoughts on the latter of moving gRPC plugin out of core and to its own repo?

Are there BWC implications of doing this since it would no longer be available natively?

I think there's competing interests here as one long term goal for gRPC is to become more natively available in OpenSearch. We expect this transport to, in theory, be more performant across the board and not just a couple key operations. We also expect plugins to more frequently take a dependency on gRPC as support is expanded to plugin APIs (I know @karenyrx is running into a similar issue with KNN query support).

I think an issue will be raised in core soon to discuss the placement of the gRPC plugin which will be very relevant to this conversation of taking it as a test dependency. Will copy this discussion and related issue there.

[cwperks]

Are there BWC implications of doing this since it would no longer be available natively?

No issues with bwc. Only functional changes to plugins have bwc considerations..not where the plugin actually resides. If a plugin is the same code, then it would produce the same binary whether located in its own repo or in the core.

[kaushalmahi12]

@cwperks I believe even if we move authN/authZ into or out of the core, it will still execute before the actual RestHandlers hence I assume rest layer will continue to access the same authN/authZ info via ThreadContext/SomeOtherMechanism in future.

Is my understanding incorrect on this ?

Regarding testing with security, yes we will need it if we enable the security attribute based autotagging but then again we have following options as you mentioned

• Either the security layer needs to be present in core
• We add the plugin specific tests in security plugin