[CI] Searchable snapshots tests fail to start secureHdfsFixture on Java 17

[mark-vieira]

Looks like this just started happening on Sep 26th. I suspect this is auto security related in some way, but don't quote me on that. Looks ot be specific to running on Java 17.

Build scan:
https://gradle-enterprise.elastic.co/s/us3iizodx5g3k/console-log?task=:x-pack:plugin:searchable-snapshots:qa:hdfs:secureHdfsFixture

Repro line:
./gradlew :x-pack:plugin:searchable-snapshots:qa:hdfs:integTestSecure -Druntime.java=17

Reproduces locally?:
Yes

Applicable branches:
master

Failure history:
https://gradle-enterprise.elastic.co/scans/failures?failures.failureClassification=all_failures&failures.failureMessage=Execution%20failed%20for%20task%20*%0A%3E%20Failed%20to%20start%20secureHdfsFixture&search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles

Failure excerpt:

    java.io.IOException: Login failure for hdfs/hdfs.build.elastic.co@BUILD.ELASTIC.CO from keytab /home/mark/workspaces/elasticsearch/test/fixtures/krb5kdc-fixture/testfixtures_shared/shared/hdfs/keytabs/hdfs_hdfs.build.elastic.co.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1066)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
        at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
        at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
        at hdfs.MiniHDFS.main(MiniHDFS.java:114)
    Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:880)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:743)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
        at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:734)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
        at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:580)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
        ... 9 more
    2021-09-28 16:55:47,664 INFO  [main] hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1789)) - Shutting down the Mini HDFS Cluster
    Exception in thread "main" java.io.IOException: Login failure for hdfs/hdfs.build.elastic.co@BUILD.ELASTIC.CO from keytab /home/mark/workspaces/elasticsearch/test/fixtures/krb5kdc-fixture/testfixtures_shared/shared/hdfs/keytabs/hdfs_hdfs.build.elastic.co.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1066)
        at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:286)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1081)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:376)
        at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:1027)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:830)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:485)
        at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
        at hdfs.MiniHDFS.main(MiniHDFS.java:114)
    Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
    
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:880)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:743)
        at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
        at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:734)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:672)
        at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:670)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:670)
        at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:580)
        at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1057)
        ... 9 more

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[masseyke]

This looks suspiciously like what I was seeing on my machine. The root cause was that the keytab was using the deprecated des3-cbc-sha1-kd encryption type. I fixed it locally by switching the keytab to use aes256-cts-hmac-sha1-96 instead. But since CI wasn't failing I didn't commit this (since I wasn't sure what the FIPS implications would be). Sounds like it's time to make this change though.

[mark-vieira]

But since CI wasn't failing I didn't commit this (since I wasn't sure what the FIPS implications would be). Sounds like it's time to make this change though.

Yeah, we don't run the full Java support matrix for every pull request. We generally just run with the minimum runtime Java version (which for master at the moment is Java 11).

[masseyke]

I created #78703 with the changes I had made locally. This is outside of my area of expertise though, so glad to close that one if there's a better solution.

[ywangd]

Another one: https://gradle-enterprise.elastic.co/s/64cq7tjqhnaay

[markharwood]

Another https://gradle-enterprise.elastic.co/s/nes4gncakuwea