[BUG][Performance] Permission Check needs many CPU Resources for User with all_access/admin

[HenryTheSir]

What is the bug?
Migrating from Opensearch 2.9 with disabled security to Opensearch 2.9 with enabled security we see a huge rise in cpu usage on our coordinating nodes.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Switch from Opensearch w/o security to security
2. Ingest heavy amounts of data with a user which has all_access
3. Observe high cpu usage in coordinating nodes (2-10% without cpu usage -> 100%+ with security)
4. High latency for all requests going to OS

What is the expected behavior?
Fast permission Checks if user has all_access
Possible Solution would be to add here a quick check if user is admin and then return true here:

https://github.com/opensearch-project/security/blob/2.9/src/main/java/org/opensearch/security/privileges/PrivilegesEvaluator.java#L522

Could be extended to some build in roles. (e.g. all actions are write actions -> user has role 'write' return permitted)

Audit Logging should not be affected by this as there are indices resolved again. (Auditing was disabled while enabling security)

What is your host/environment?

• OS: RHEL
• Opensearch Version 2.9
• Plugins vanilla Opensearch package (.tgz - https://opensearch.org/versions/opensearch-2-9-0.html )

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Output of hot_threads of one of the coordinating nodes:

   47.3% (236.6ms out of 500ms) cpu usage by thread 'opensearch[hostname][transport_worker][T#12]'
     6/10 snapshots sharing following 110 elements
       app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:233)
       app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:160)
       org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:827)
       org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:791)
       org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$4(ConfigModelV7.java:1062)
       org.opensearch.security.securityconf.ConfigModelV7$$Lambda$5257/0x00000008013f3040.apply(Unknown Source)
       java.base@11.0.16.1/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
       java.base@11.0.16.1/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1621)
       java.base@11.0.16.1/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
       java.base@11.0.16.1/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
       java.base@11.0.16.1/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
       java.base@11.0.16.1/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
       java.base@11.0.16.1/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
       org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1063)
       org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:516)
       org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:524)
       org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:377)
       org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)

[cwperks]

[Triage] Thank you for filing an issue @HenryTheSir. Would you please provide a little more detail around the types of queries being run where you are experiencing increased latencies? The security plugin has a performance tax, but in my experience it is closer to <10% than the latencies you are experiencing.

[HenryTheSir]

Hi @cwperks ,

we are doing bulk indexing with up to 5000 docs per request against an cluster with 3 leader, 4 coordination and around 300 data nodes (split up in hot/warm). In the bulk request can be up to 340 different target aliases which map to write indices. In the cluster itself are around 8.000 indices.

Spikes are only seen on coordinating nodes.
Find attached a screenshot from one coordinating node while security was enabled :

If more input is needed do not hesitate to contact me.

[stephen-crawford]

[Triage] Hi @HenryTheSir, thank you for filing this issue. The actionable task for this issue would be to look into the hotthread report shared above and determining whether a shortcut--such as suggested by Henry-would circumvent the CPU time.

We would be happy to review a PR addressing this issue.

[shwetathareja]

Index resolution in security filter is pretty expensive affair

 app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:276)
       app//org.opensearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:197)
       org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.getResolvedIndexPattern(ConfigModelV7.java:882)
       org.opensearch.security.securityconf.ConfigModelV7$IndexPattern.attemptResolveIndexNames(ConfigModelV7.java:845)
       org.opensearch.security.securityconf.ConfigModelV7.lambda$impliesTypePerm$3(ConfigModelV7.java:1074)
       org.opensearch.security.securityconf.ConfigModelV7$$Lambda/0x0000000502f0b1c8.apply(Unknown Source)
       java.base@21.0.4/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
       java.base@21.0.4/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1715)
       java.base@21.0.4/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
       java.base@21.0.4/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
       java.base@21.0.4/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
       java.base@21.0.4/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
       java.base@21.0.4/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
       org.opensearch.security.securityconf.ConfigModelV7.impliesTypePerm(ConfigModelV7.java:1075)
       org.opensearch.security.securityconf.ConfigModelV7$SecurityRoles.impliesTypePermGlobal(ConfigModelV7.java:551)
       org.opensearch.security.privileges.PrivilegesEvaluator.evaluate(PrivilegesEvaluator.java:572)
       org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:381)
       org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)
       app//org.opensearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:663)
       app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
       app//org.opensearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:825)
       app//org.opensearch.action.bulk.TransportBulkAction.doInternalExecute(TransportBulkAction.java:378)
       app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:237)
       app//org.opensearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:125)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220)
       org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395)
       org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:77)
       app//org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:190)
       app//org.opensearch.action.support.TransportAction.execute(TransportAction.java:109)
       app//org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:112)
       app//org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:99)
       app//org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476)
       app//org.opensearch.client.support.AbstractClient.bulk(AbstractClient.java:552)
       app//org.opensearch.rest.action.document.RestBulkAction.lambda$prepareRequest$0(RestBulkAction.java:112)
       app//org.opensearch.rest.action.document.RestBulkAction$$Lambda/0x00000005030668b0.accept(Unknown Source)
       app//org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:127)
       org.opensearch.security.filter.SecurityRestFilter$AuthczRestHandler.handleRequest(SecurityRestFilter.java:195)
       app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:382)
       app//org.opensearch.rest.RestController.tryAllHandlers(RestController.java:473)
       app//org.opensearch.rest.RestController.dispatchRequest(RestController.java:294)
       org.opensearch.network.commons.filter.FilterChainManager$FilterChainImpl.doFilter(FilterChainManager.java:70)
       org.opensearch.network.commons.logger.OpenSearchServiceLogAdapter.doFilter(OpenSearchServiceLogAdapter.java:44)
       org.opensearch.network.commons.filter.FilterChainManager$FilterChainImpl.doFilter(FilterChainManager.java:67)
       org.opensearch.network.commons.redact.HttpResponseAdapter.doFilter(HttpResponseAdapter.java:110)
       org.opensearch.network.commons.filter.FilterChainManager$FilterChainImpl.doFilter(FilterChainManager.java:67)
       org.opensearch.network.commons.filter.FilterChainManager.dispatchRequest(FilterChainManager.java:42)
       org.opensearch.http.AbstractOpenSearchHttpServerTransport.dispatchRequest(AbstractOpenSearchHttpServerTransport.java:434)
       org.opensearch.http.AbstractOpenSearchHttpServerTransport.handleIncomingRequest(AbstractOpenSearchHttpServerTransport.java:502)
       org.opensearch.http.AbstractOpenSearchHttpServerTransport.incomingRequest(AbstractOpenSearchHttpServerTransport.java:364)
       org.opensearch.http.netty4.OpenSearchNetty4HttpRequestHandler.channelRead0(OpenSearchNetty4HttpRequestHandler.java:74)
       org.opensearch.http.netty4.OpenSearchNetty4HttpRequestHandler.channelRead0(OpenSearchNetty4HttpRequestHandler.java:52)
       io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
       org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:72)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
       io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
       io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
       io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
       io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61)
       io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:425)
       io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
       io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
       io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
       io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
       io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
       io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       java.base@21.0.4/java.lang.Thread.runWith(Thread.java:1596)
       java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)

[shikharj05]

This is fixed and improved in in 2.19+ by #3870