Open
Description
Description
In our customer production environment. One cluster normally search response time is around 80ms. After starting some wrong password client servers, cluster search response avg up to 300+ms. From cpu analysis, we can see the password verification caused a lot cpu resource(50%+). So we might need to improve this auth verify performance, special to the fail fast case.
7.5.1 JStack info:
"elasticsearch[node][generic][T#62]" #968 daemon prio=5 os_prio=0 cpu=93909.91ms elapsed=5616.47s allocated=2200M defined_classes=0 tid=0x00007f1d17cab000 nid=0x6f23 runnable [0x00007f1dd42b8000]
java.lang.Thread.State: RUNNABLE
at org.elasticsearch.xpack.core.security.authc.support.BCrypt.key(BCrypt.java:562)
at org.elasticsearch.xpack.core.security.authc.support.BCrypt.crypt_raw(BCrypt.java:626)
at org.elasticsearch.xpack.core.security.authc.support.BCrypt.hashpw(BCrypt.java:708)
at org.elasticsearch.xpack.core.security.authc.support.BCrypt.checkpw(BCrypt.java:789)
at org.elasticsearch.xpack.core.security.authc.support.Hasher.verifyBcryptHash(Hasher.java:561)
at org.elasticsearch.xpack.core.security.authc.support.Hasher.access$100(Hasher.java:27)
at org.elasticsearch.xpack.core.security.authc.support.Hasher$1.verify(Hasher.java:38)
at org.elasticsearch.xpack.core.security.authc.support.Hasher.verifyHash(Hasher.java:497)
at org.elasticsearch.xpack.security.authc.file.FileUserPasswdStore.verifyPassword(FileUserPasswdStore.java:84)
at org.elasticsearch.xpack.security.authc.file.FileRealm.doAuthenticate(FileRealm.java:39)
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.authenticateWithCache(CachingUsernamePasswordRealm.java:166)
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$authenticateWithCache$2(CachingUsernamePasswordRealm.java:153)
at org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm$$Lambda$4270/0x00000017ff09ac40.accept(Unknown Source)
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.9.1-ga/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.9.1-ga/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.9.1-ga/Thread.java:834)
I have done pressure test on 7.14, 8.11, both have auth perf issue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment