KerberosAuthenticationIT authentication test failures

[benwtrent]

Build scan:

7.6 https://gradle-enterprise.elastic.co/s/3olzbtxfidvds
7.8 https://gradle-enterprise.elastic.co/s/22optb3r7tq24
7.7 https://gradle-enterprise.elastic.co/s/4225d2fztqtfg

Repro line:

Three tests fail:

./gradlew ':x-pack:qa:kerberos-tests:integTestRunner' --tests "org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab" -Dtests.seed=461C01080EE1DB8B -Dtests.security.manager=true -Dtests.locale=es-EC -Dtests.timezone=IET -Dcompiler.java=13


./gradlew ':x-pack:qa:kerberos-tests:integTestRunner' --tests "org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByUsernamePassword" -Dtests.seed=461C01080EE1DB8B -Dtests.security.manager=true -Dtests.locale=es-EC -Dtests.timezone=IET -Dcompiler.java=13


./gradlew ':x-pack:qa:kerberos-tests:integTestRunner' --tests "org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testGetOauth2TokenInExchangeForKerberosTickets" -Dtests.seed=461C01080EE1DB8B -Dtests.security.manager=true -Dtests.locale=es-EC -Dtests.timezone=IET -Dcompiler.java=13

Reproduces locally?:
No
Applicable branches:

7.6
7.7
7.8
master

Failure excerpt:

All tests fail with a similar NPE.

org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT > testLoginByKeytab FAILED
    java.lang.RuntimeException: javax.security.auth.login.LoginException: java.lang.NullPointerException
    	at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:149)
    	at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159)
    	at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
    	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310)
    	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
    	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:754)
    	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
    	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
    	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
    	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
    	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
    	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.lambda$login$1(SpnegoHttpClientConfigCallbackHandler.java:168)
    	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
    	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.login(SpnegoHttpClientConfigCallbackHandler.java:155)
    	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.setupSpnegoAuthSchemeSupport(SpnegoHttpClientConfigCallbackHandler.java:126)
    	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.customizeHttpClient(SpnegoHttpClientConfigCallbackHandler.java:115)
    	at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:215)
    	at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
    	at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:191)
    	at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.buildRestClientForKerberos(KerberosAuthenticationIT.java:190)
    	at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.executeRequestAndVerifyResponse(KerberosAuthenticationIT.java:156)
    	at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab(KerberosAuthenticationIT.java:98)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
    	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
    	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
    	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
    	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
    	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
    	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
    	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
    	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
    	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    	at java.base/java.lang.Thread.run(Thread.java:832)
        at __randomizedtesting.SeedInfo.seed([461C01080EE1DB8B:297FA9053AE957CC]:0)
        at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.setupSpnegoAuthSchemeSupport(SpnegoHttpClientConfigCallbackHandler.java:141)
        at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.customizeHttpClient(SpnegoHttpClientConfigCallbackHandler.java:115)
        at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:215)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
        at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:191)
        at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.buildRestClientForKerberos(KerberosAuthenticationIT.java:190)
        at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.executeRequestAndVerifyResponse(KerberosAuthenticationIT.java:156)
        at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab(KerberosAuthenticationIT.java:98)

        Caused by:
        javax.security.auth.login.LoginException: java.lang.NullPointerException
        	at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:149)
        	at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159)
        	at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
        	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310)
        	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
        	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:754)
        	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
        	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
        	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
        	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
        	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
        	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
        	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
        	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.lambda$login$1(SpnegoHttpClientConfigCallbackHandler.java:168)
        	at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
        	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.login(SpnegoHttpClientConfigCallbackHandler.java:155)
        	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.setupSpnegoAuthSchemeSupport(SpnegoHttpClientConfigCallbackHandler.java:126)
        	at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.customizeHttpClient(SpnegoHttpClientConfigCallbackHandler.java:115)
        	at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:215)
        	at java.base/java.security.AccessController.doPrivileged(AccessController.java:312)
        	at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:191)
        	at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.buildRestClientForKerberos(KerberosAuthenticationIT.java:190)
        	at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.executeRequestAndVerifyResponse(KerberosAuthenticationIT.java:156)
        	at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab(KerberosAuthenticationIT.java:98)
        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
        	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        	at java.base/java.lang.Thread.run(Thread.java:832)
            at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:821)
            at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
            at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
            at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
            at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
            at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
            at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.lambda$login$1(SpnegoHttpClientConfigCallbackHandler.java:168)
            at java.base/java.security.AccessController.doPrivileged(AccessController.java:554)
            at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.login(SpnegoHttpClientConfigCallbackHandler.java:155)
            at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.setupSpnegoAuthSchemeSupport(SpnegoHttpClientConfigCallbackHandler.java:126)
            ... 7 more

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[jkakavas]

These started failing in May 9 only with runtime java set to 15 ( this is why @benwtrent can't reproduce them I guess, the repro line above is missing the -Druntime.java=15 but the build plugin output has this as it should ) . Now, JDK 15 build 22 was released on May 6 and I guess we picked it up around May 9 we actually picked it up on May 7. This is an NPE in the sun.security.krb5.KrbKdcRep class so I guess it's most probably a JDK bug rather one of our own. I will look at this over the next week or so and file the bug upstream if needed.

[benwtrent]

Here is a different test failure, but also JDK15 + kerberos NPE: https://gradle-enterprise.elastic.co/s/hgrh76kyapze2

java.io.IOException: Login failure for hdfs/hdfs.build.elastic.co@BUILD.ELASTIC.CO from keytab /dev/shm/elastic+elasticsearch+7.x+matrix-java-periodic/ES_RUNTIME_JAVA/openjdk15/nodes/general-purpose/test/fixtures/krb5kdc-fixture/testfixtures_shared/shared/hdfs/keytabs/hdfs_hdfs.build.elastic.co.keytab: javax.security.auth.login.LoginException: java.lang.NullPointerException
    	at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:149)
    	at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159)
    	at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
    	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310)
    	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
    	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:754)
    	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
    	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
    	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
    	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
    	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
    	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
    	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
    	at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
    	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
    	at hdfs.MiniHDFS.main(MiniHDFS.java:119)
    
    	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1066)
    	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
    	at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
    	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
    	at hdfs.MiniHDFS.main(MiniHDFS.java:119)
    Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException
    	at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:149)
    	at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159)
    	at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
    	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310)
    	at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498)
    	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:754)
    	at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592)
    	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
    	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
    	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
    	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
    	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
    	at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
    	at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
    	at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
    	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
    	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
    	at hdfs.MiniHDFS.main(MiniHDFS.java:119)
    
    	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:821)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
    	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
    	at java.base/java.security.AccessController.doPrivileged(AccessController.java:691)
    	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
    	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
    	at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
    	... 9 more

[markharwood]

Another JDK15 + Kerberos NPE today https://gradle-enterprise.elastic.co/s/w46qi55v5ga2k

REPRODUCE WITH: ./gradlew ':x-pack:qa:kerberos-tests:integTestRunner' --tests "org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab" -Dtests.seed=5473F4144376DD6D -Dtests.security.manager=true -Dtests.locale=en-ZA -Dtests.timezone=Hongkong -Dcompiler.java=13 |  

java.lang.RuntimeException: javax.security.auth.login.LoginException: java.lang.NullPointerException |  
-- | --
  | at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:149) |  
  | at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159) |  
  | at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) |  
  | at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310) |  
  | at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498) |  
  | at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:754) |  
  | at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592) |  
  | at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726) |  
  | at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665) |  
  | at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663) |  
  | at java.base/java.security.AccessController.doPrivileged(AccessController.java:691) |  
  | at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663) |  
  | at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.lambda$login$1(SpnegoHttpClientConfigCallbackHandler.java:168) |  
  | at java.base/java.security.AccessController.doPrivileged(AccessController.java:554) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.login(SpnegoHttpClientConfigCallbackHandler.java:155) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.setupSpnegoAuthSchemeSupport(SpnegoHttpClientConfigCallbackHandler.java:126) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.customizeHttpClient(SpnegoHttpClientConfigCallbackHandler.java:115) |  
  | at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:215) |  
  | at java.base/java.security.AccessController.doPrivileged(AccessController.java:312) |  
  | at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:191) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.buildRestClientForKerberos(KerberosAuthenticationIT.java:190) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.executeRequestAndVerifyResponse(KerberosAuthenticationIT.java:156) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab(KerberosAuthenticationIT.java:98) |  
  | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) |  
  | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) |  
  | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |  
  | at java.base/java.lang.reflect.Method.invoke(Method.java:564) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) |  
  | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  | at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) |  
  | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) |  
  | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |  
  | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |  
  | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |  
  | at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |  
  | at java.base/java.lang.Thread.run(Thread.java:832) |  
  | at __randomizedtesting.SeedInfo.seed([5473F4144376DD6D:3B105C19777E512A]:0) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.setupSpnegoAuthSchemeSupport(SpnegoHttpClientConfigCallbackHandler.java:141) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.SpnegoHttpClientConfigCallbackHandler.customizeHttpClient(SpnegoHttpClientConfigCallbackHandler.java:115) |  
  | at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:215) |  
  | at java.base/java.security.AccessController.doPrivileged(AccessController.java:312) |  
  | at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:191) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.buildRestClientForKerberos(KerberosAuthenticationIT.java:190) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.executeRequestAndVerifyResponse(KerberosAuthenticationIT.java:156) |  
  | at org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT.testLoginByKeytab(KerberosAuthenticationIT.java:98) |  
  |   |  
  | Caused by: |  
  | javax.security.auth.login.LoginException: java.lang.NullPointerException |  
  | at java.security.jgss/sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:149) |  
  | at java.security.jgss/sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159) |  
  | at java.security.jgss/sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121) |  
  | at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:310) |  
  | at java.security.jgss/sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:498) |  
  | at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:754) |  
  | at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:592) |  
  | at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726) |  
  | at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665) |  
  | at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)

[jkakavas]

I spent 30 mins on it today out of curiosity. It seems this is actually a bug in sun.security.krb5.KrbKdcRep that was surfaced by the changes to ReferralsState in https://hg.openjdk.java.net/jdk/jdk/rev/142d090cab77 . I don't know enough about the mentioned RFCs in order to make a proper bug request upstream now, and given this might be a JDK15 regression that doesn't affect us now, I will take the time at some point over the next few weeks to do so