ERROR cannot connect to kerberized kyuubi using beeline #1638

turtoise · 2021-12-28T08:38:24Z

turtoise
Dec 28, 2021

My environment is a kerberized HDP ,when i deployed kyuubi with HDP Spark3 ,it seems kyuubi server started successfully.But when i using below command 'beeline -u "jdbc:hive2://client-xxx.edp:10009/;principal=hive_test/client-xxx.edp@HADOOP.COM" 'to connect kyuubi,it shows errors:

Caused by: org.apache.kyuubi.KyuubiSQLException: 21/12/28 15:59:47 INFO RetryInvocationHandler: org.apache.hadoop.security.authorize.AuthorizationException: User: hive_test/client-xxx.edp@HADOOP.COM is not allowed to impersonate hive_test, while invoking ApplicationClientProtocolPBClientImpl.getClusterMetrics over null after 7 failover attempts. Trying to failover after sleeping for 32753ms.
 See more: /home/spark/kyuubi-1.4.0/work/hive_test/kyuubi-spark-sql-engine.log.2
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
        at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:140)
        ... 1 more

Please, can someone provide the suggestions for this problem? Also, share the details or steps is highly appreciated.
In kyuubi-defaults.conf file:

kyuubi.authentication=KERBEROS
kyuubi.kinit.principal=hive_test/client-xxx.edp@HADOOP.COM
kyuubi.kinit.keytab=hive_test.keytab

kyuubi.engine.share.level=SERVER
kyuubi.frontend.bind.host=x.x.x.x
kyuubi.frontend.bind.port=10009
kyuubi.ha.zookeeper.quorum=datanode-xxx.edp:2181
kyuubi.ha.zookeeper.namespace=kyuubi
kyuubi.session.engine.idle.timeout=PT10H
spark.master=yarn
spark.submit.deployMode=cluster

Answered by zhouyifan279

Jan 4, 2022

@turtoise The KMS provider exception is probably a bug of Kyuubi's HadoopCredentialsManager. Could you set kyuubi.credentials.hadoopfs.enabled to false in kyuubi-defaults.conf, restart kyuubi server and try command:

beeline -u "jdbc:hive2://controlnode-205.edp:10009/;principal=hive_test/client-209.edp@HADOOP.COM;

Then paste kyuubi server log and engine log here.

Thanks.

View full answer

yaooqinn · 2021-12-28T08:46:42Z

yaooqinn
Dec 28, 2021
Collaborator

looks like you need to add these in $HADOOP_CONF_DIR/core-site.xml

hadoop.proxyuser.hive_test.hosts *
hadoop.proxyuser.hive_test.groups *

5 replies

turtoise Dec 28, 2021
Author

It seems works after add these in $HADOOP_CONF_DIR/core-site.xml. Now I can connect to kyuubi,but another problem appeared.
I execute command select * from test in sparksql successfully,but it failed in kyuubi .Error like this:

Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: http://controlnode-205.edp:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fcontrolnode-205.edp%40HADOOP.COM
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:232)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:219)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:317)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:189)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:395)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1035)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1029)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        ... 60 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:360)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:204)
        ... 70 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:148)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:336)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:310)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:310)
        ... 71 more (state=,code=0)

ulysses-you Dec 28, 2021
Collaborator

could you also provide the spark-submit command log ?

turtoise Dec 29, 2021
Author

OK.This is what spark-submit command log shows:
First when I execute command beeline -u jdbc:hive2://controlnode-205.edp:10009/;principal=hive_test/client-209.edp@HADOOP.COM
I can connect to kyuubi successfully.

Connecting to jdbc:hive2://controlnode-205.edp:10009/;principal=hive_test/client-209.edp@HADOOP.COM
21/12/29 11:22:39 INFO hive.Utils: Supplied authorities: controlnode-205.edp:10009
21/12/29 11:22:39 INFO hive.Utils: Resolved authority: controlnode-205.edp:10009
21/12/29 11:22:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Apache Kyuubi (Incubating) (version 1.4.0-incubating)
Driver: Kyuubi Project Hive JDBC Shaded Client (version 1.4.0-incubating)
Beeline version 1.4.0-incubating by Apache Kyuubi (Incubating)
0: jdbc:hive2://controlnode-205.edp:10009/>

The server log is:

21/12/29 11:22:39 INFO server.KyuubiThriftBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
21/12/29 11:22:39 INFO operation.LaunchEngine: Processing hive_test's query[a33b9b65-6587-4db2-809a-9c245dbd4f94]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE
21/12/29 11:22:39 INFO operation.LaunchEngine: Processing hive_test's query[a33b9b65-6587-4db2-809a-9c245dbd4f94]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE
21/12/29 11:22:39 INFO imps.CuratorFrameworkImpl: Starting
21/12/29 11:22:39 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=datanode-206.edp:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@3561046f
21/12/29 11:22:39 INFO zookeeper.ClientCnxn: Opening socket connection to server datanode-206.edp/172.16.40.206:2181. Will not attempt to authenticate using SASL (unknown error)
21/12/29 11:22:39 INFO zookeeper.ClientCnxn: Socket connection established to datanode-206.edp/172.16.40.206:2181, initiating session
21/12/29 11:22:39 INFO zookeeper.ClientCnxn: Session establishment complete on server datanode-206.edp/172.16.40.206:2181, sessionid = 0x17ddaf384370089, negotiated timeout = 60000
21/12/29 11:22:39 INFO state.ConnectionStateManager: State change: CONNECTED
21/12/29 11:22:39 INFO engine.EngineRef: Launching engine:
/usr/hdp/3.1.5.0-152/spark3/bin/spark-submit \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.kyuubi.session.engine.idle.timeout=PT10H \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.ha.zookeeper.quorum=datanode-206.edp:2181 \
        --conf spark.kyuubi.kinit.principal=hive_test/client-209.edp@HADOOP.COM \
        --conf spark.app.name=kyuubi_CONNECTION_SPARK_SQL_hive_test_ed83620c-36c9-4fd4-99d5-28f26cce08b4 \
        --conf spark.kyuubi.ha.engine.ref.id=ed83620c-36c9-4fd4-99d5-28f26cce08b4 \
        --conf spark.submit.deployMode=cluster \
        --conf spark.master=yarn \
        --conf spark.yarn.tags=KYUUBI \
        --conf spark.kyuubi.engine.share.level=CONNECTION \
        --conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_CONNECTION_SPARK_SQL/hive_test/ed83620c-36c9-4fd4-99d5-28f26cce08b4 \
        --conf spark.kyuubi.kinit.keytab=/etc/security/keytabs/hive_test.keytab \
        --proxy-user hive_test /root/kyuubi-1.4.0/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.0-incubating.jar
21/12/29 11:22:39 INFO engine.ProcBuilder: Logging to /root/kyuubi-1.4.0/work/hive_test/kyuubi-spark-sql-engine.log.9
21/12/29 11:22:58 INFO client.ServiceDiscovery: Get service instance:datanode-207.edp:39988 and version:Some(1.4.0-incubating) under /kyuubi_CONNECTION_SPARK_SQL/hive_test/ed83620c-36c9-4fd4-99d5-28f26cce08b4
21/12/29 11:22:58 INFO session.KyuubiSessionImpl: [hive_test:172.16.40.209] SessionHandle [19c9845a-8a6f-4c1e-93fe-24c8694d3f02] - Connected to engine [datanode-207.edp:39988]
21/12/29 11:22:58 INFO session.KyuubiSessionImpl: [hive_test:172.16.40.209] SessionHandle [19c9845a-8a6f-4c1e-93fe-24c8694d3f02] - Opened engine session[SessionHandle [098df4f7-650e-42df-9a7b-f694775baf31]]
21/12/29 11:22:58 INFO imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
21/12/29 11:22:58 INFO zookeeper.ZooKeeper: Session: 0x17ddaf384370089 closed
21/12/29 11:22:58 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x17ddaf384370089
21/12/29 11:22:58 INFO operation.LaunchEngine: Processing hive_test's query[a33b9b65-6587-4db2-809a-9c245dbd4f94]: RUNNING_STATE -> FINISHED_STATE, statement: LAUNCH_ENGINE, time taken: 19.279 seconds
21/12/29 11:22:58 INFO session.KyuubiSessionManager: hive_test's session with SessionHandle [19c9845a-8a6f-4c1e-93fe-24c8694d3f02] is opened, current opening sessions 1

But when i execute simple sql statement like select * from test:

The server log is:

21/12/29 11:42:27 INFO log.OperationLog: Creating operation log file /root/kyuubi-1.4.0/server_operation_logs/19c9845a-8a6f-4c1e-93fe-24c8694d3f02/fbe314cd-03f5-460f-ae62-b7ef314f7b5a
21/12/29 11:42:27 INFO session.KyuubiSessionImpl: [hive_test:172.16.40.209] SessionHandle [19c9845a-8a6f-4c1e-93fe-24c8694d3f02] - Starting to wait the launch engine operation finished
21/12/29 11:42:27 INFO session.KyuubiSessionImpl: [hive_test:172.16.40.209] SessionHandle [19c9845a-8a6f-4c1e-93fe-24c8694d3f02] - Engine has been launched, elapsed time: 0 s
21/12/29 11:42:27 INFO operation.ExecuteStatement: Processing hive_test's query[fbe314cd-03f5-460f-ae62-b7ef314f7b5a]: INITIALIZED_STATE -> PENDING_STATE, statement: select * from test
21/12/29 11:42:27 INFO operation.ExecuteStatement: Processing hive_test's query[fbe314cd-03f5-460f-ae62-b7ef314f7b5a]: PENDING_STATE -> RUNNING_STATE, statement: select * from test
21/12/29 11:42:27 INFO operation.ExecuteStatement: Query[fbe314cd-03f5-460f-ae62-b7ef314f7b5a] in ERROR_STATE
21/12/29 11:42:27 INFO operation.ExecuteStatement: Processing hive_test's query[fbe314cd-03f5-460f-ae62-b7ef314f7b5a]: RUNNING_STATE -> ERROR_STATE, statement: select * from test, time taken: 0.737 seconds
21/12/29 11:42:27 INFO operation.ExecuteStatement: Processing hive_test's query[fbe314cd-03f5-460f-ae62-b7ef314f7b5a]: ERROR_STATE -> CLOSED_STATE, statement: select * from test
21/12/29 11:42:27 INFO client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:5D 29 6B 78 34 26 4F E3 AC F9 93 FC 84 C7 65 46, secret:D3 98 C7 C8 8C 3B 43 D2 A2 8D 9D 16 28 AA D2 54), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side

the client log is :

Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: http://controlnode-205.edp:9292/kms/v1/?op=GETDELEGATIONTOKEN&renewer=rm%2Fcontrolnode-205.edp%40HADOOP.COM
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:232)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:219)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:317)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:189)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:395)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1035)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1029)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        ... 60 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:360)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:204)
        ... 70 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
        at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:148)
        at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
        at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
        at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:336)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator$1.run(KerberosAuthenticator.java:310)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:310)
        ... 71 more (state=,code=0)

In addition,if i execute submit command separately like this:

/usr/hdp/3.1.5.0-152/spark3/bin/spark-submit \
        --queue default \
		--principal hive_test/client-209.edp@HADOOP.COM \
		--keytab /etc/security/keytabs/hive_test.keytab \
        --class org.apache.kyuubi.engine.spark.SparkSQLEngine \
        --conf spark.kyuubi.session.engine.idle.timeout=PT10H \
        --conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
        --conf spark.kyuubi.ha.zookeeper.quorum=datanode-206.edp:2181 \
        --conf spark.kyuubi.kinit.principal=hive_test/client-209.edp@HADOOP.COM \
        --conf spark.app.name=kyuubi_USER_SPARK_SQL_hive_test_default_67864b66-7ba5-4412-8752-54bca456ff89 \
        --conf spark.kyuubi.ha.engine.ref.id=67864b66-7ba5-4412-8752-54bca456ff89 \
        --conf spark.submit.deployMode=cluster \
        --conf spark.master=yarn \
        --conf spark.yarn.tags=KYUUBI \
        --conf spark.kyuubi.engine.share.level=USER \
        --conf spark.kyuubi.ha.zookeeper.namespace=/kyuubi_USER_SPARK_SQL/hive_test/default \
        --conf spark.kyuubi.kinit.keytab=/etc/security/keytabs/hive_test.keytab \
        /root/kyuubi-1.4.0/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.0-incubating.jar

21/12/29 12:01:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/12/29 12:01:16 INFO Client: Kerberos credentials: principal = hive_test/client-209.edp@HADOOP.COM, keytab = /etc/security/keytabs/hive_test.keytab
21/12/29 12:01:16 INFO RMProxy: Connecting to ResourceManager at controlnode-205.edp/x.x.x.x:8050
21/12/29 12:01:17 INFO Client: Requesting a new application from cluster with 3 NodeManagers
21/12/29 12:01:17 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
21/12/29 12:01:17 INFO Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.5.0-152/0/resource-types.xml
21/12/29 12:01:17 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (55296 MB per container)
21/12/29 12:01:17 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
21/12/29 12:01:17 INFO Client: Setting up container launch context for our AM
21/12/29 12:01:17 INFO Client: Setting up the launch environment for our AM container
21/12/29 12:01:17 INFO Client: Preparing resources for our AM container
21/12/29 12:01:17 INFO Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
21/12/29 12:01:17 INFO Client: Uploading resource file:/etc/security/keytabs/hive_test.keytab -> hdfs://controlnode-205.edp:8020/user/hive_test/.sparkStaging/application_1640746942612_0004/hive_test.keytab
21/12/29 12:01:17 INFO Client: Source and destination file systems are the same. Not copying hdfs://controlnode-205.edp:8020/services/spark/sparkjars.zip
21/12/29 12:01:17 INFO Client: Uploading resource file:/root/kyuubi-1.4.0/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.4.0-incubating.jar -> hdfs://controlnode-205.edp:8020/user/hive_test/.sparkStaging/application_1640746942612_0004/kyuubi-spark-sql-engine_2.12-1.4.0-incubating.jar
21/12/29 12:01:17 INFO Client: Uploading resource file:/tmp/spark-f81955c1-e1c0-4f5d-98e4-572216f41230/__spark_conf__2048312595093835193.zip -> hdfs://controlnode-205.edp:8020/user/hive_test/.sparkStaging/application_1640746942612_0004/__spark_conf__.zip
21/12/29 12:01:17 INFO SecurityManager: Changing view acls to: root,hive_test
21/12/29 12:01:17 INFO SecurityManager: Changing modify acls to: root,hive_test
21/12/29 12:01:17 INFO SecurityManager: Changing view acls groups to:
21/12/29 12:01:17 INFO SecurityManager: Changing modify acls groups to:
21/12/29 12:01:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users  with view permissions: Set(root, hive_test); groups with view permissions: Set(); users  with modify permissions: Set(root, hive_test); groups with modify permissions: Set()
21/12/29 12:01:18 INFO HadoopDelegationTokenManager: Attempting to login to KDC using principal: hive_test/client-209.edp@HADOOP.COM
21/12/29 12:01:18 INFO HadoopDelegationTokenManager: Successfully logged into KDC.
21/12/29 12:01:18 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-2070321598_1, ugi=hive_test/client-209.edp@HADOOP.COM (auth:KERBEROS)]] with renewer rm/controlnode-205.edp@HADOOP.COM
21/12/29 12:01:18 INFO DFSClient: Created token for hive_test: HDFS_DELEGATION_TOKEN owner=hive_test/client-209.edp@HADOOP.COM, renewer=yarn, realUser=, issueDate=1640750478043, maxDate=1641355278043, sequenceNumber=368, masterKeyId=35 on x.x.x.x:8020
21/12/29 12:01:18 INFO KMSClientProvider: New token created: (Kind: kms-dt, Service: kms://http@controlnode-205.edp:9292/kms, Ident: (kms-dt owner=hive_test, renewer=yarn, realUser=, issueDate=1640750478101, maxDate=1641355278101, sequenceNumber=3, masterKeyId=2))
21/12/29 12:01:18 INFO HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-2070321598_1, ugi=hive_test/client-209.edp@HADOOP.COM (auth:KERBEROS)]] with renewer hive_test/client-209.edp@HADOOP.COM
21/12/29 12:01:18 INFO DFSClient: Created token for hive_test: HDFS_DELEGATION_TOKEN owner=hive_test/client-209.edp@HADOOP.COM, renewer=hive_test, realUser=, issueDate=1640750478285, maxDate=1641355278285, sequenceNumber=369, masterKeyId=35 on x.x.x.x:8020
21/12/29 12:01:18 INFO KMSClientProvider: New token created: (Kind: kms-dt, Service: kms://http@controlnode-205.edp:9292/kms, Ident: (kms-dt owner=hive_test, renewer=hive_test, realUser=, issueDate=1640750478301, maxDate=1641355278301, sequenceNumber=4, masterKeyId=2))
21/12/29 12:01:18 INFO HadoopFSDelegationTokenProvider: Renewal interval is 86400020 for token kms-dt
21/12/29 12:01:18 INFO HadoopFSDelegationTokenProvider: Renewal interval is 86400043 for token HDFS_DELEGATION_TOKEN
21/12/29 12:01:19 INFO HiveConf: Found configuration file file:/etc/spark3/3.1.5.0-152/0/hive-site.xml
21/12/29 12:01:19 INFO Client: Submitting application application_1640746942612_0004 to ResourceManager
21/12/29 12:01:19 INFO YarnClientImpl: Submitted application application_1640746942612_0004
21/12/29 12:01:20 INFO Client: Application report for application_1640746942612_0004 (state: ACCEPTED)
21/12/29 12:01:20 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: AM container is launched, waiting for AM container to Register with RM
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1640750479503
         final status: UNDEFINED
         tracking URL: http://controlnode-205.edp:8088/proxy/application_1640746942612_0004/
         user: hive_test
21/12/29 12:01:21 INFO Client: Application report for application_1640746942612_0004 (state: ACCEPTED)
21/12/29 12:01:22 INFO Client: Application report for application_1640746942612_0004 (state: ACCEPTED)
21/12/29 12:01:23 INFO Client: Application report for application_1640746942612_0004 (state: ACCEPTED)
21/12/29 12:01:24 INFO Client: Application report for application_1640746942612_0004 (state: ACCEPTED)
21/12/29 12:01:25 INFO Client: Application report for application_1640746942612_0004 (state: ACCEPTED)
21/12/29 12:01:26 INFO Client: Application report for application_1640746942612_0004 (state: RUNNING)
21/12/29 12:01:26 INFO Client:
         client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
         diagnostics: N/A
         ApplicationMaster host: datanode-206.edp
         ApplicationMaster RPC port: 46150
         queue: default
         start time: 1640750479503
         final status: UNDEFINED
         tracking URL: http://controlnode-205.edp:8088/proxy/application_1640746942612_0004/
         user: hive_test
21/12/29 12:01:27 INFO Client: Application report for application_1640746942612_0004 (state: RUNNING)
21/12/29 12:01:28 INFO Client: Application report for application_1640746942612_0004 (state: RUNNING)

ulysses-you Dec 29, 2021
Collaborator

can you try this command to connect with Kyuubi ?

beeline -u "jdbc:hive2://controlnode-205.edp:10009/;principal=hive_test/client-209.edp@HADOOP.COM;#spark.kerberos.principal=hive_test/client-209.edp@HADOOP.COM;spark.kerberos.keytab=/etc/security/keytabs/hive_test.keytab"

turtoise Dec 29, 2021
Author

It works, didn't report any errors to execute sql command. However,in server log, it reports errors every 1 minute like this:

21/12/29 15:04:08 INFO credentials.HadoopFsDelegationTokenProvider: getting token owned by hive_test for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1740124464_43, ugi=hive_test (auth:PROXY) via hive_test/client-209.edp@HADOOP.COM (auth:KERBEROS)]]
21/12/29 15:04:08 INFO hdfs.DFSClient: Created token for hive_test: HDFS_DELEGATION_TOKEN owner=hive_test, renewer=hive_test, realUser=hive_test/client-209.edp@HADOOP.COM, issueDate=1640761448127, maxDate=1641366248127, sequenceNumber=429, masterKeyId=35 on x.x.x.x:8020
21/12/29 15:04:08 WARN kms.LoadBalancingKMSClientProvider: KMS provider at [http://controlnode-205.edp:9292/kms/v1/] threw an IOException:
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationToken(KMSClientProvider.java:1051)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:255)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:252)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.getDelegationToken(LoadBalancingKMSClientProvider.java:252)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:107)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$2(HadoopFsDelegationTokenProvider.scala:67)
        at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:65)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:107)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:106)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:58)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationToken(KMSClientProvider.java:1029)
        ... 31 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: http://controlnode-205.edp:9292/kms/v1/?op=GETDELEGATIONTOKEN&doAs=hive_test&renewer=hive_test
        at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:232)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:219)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:350)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:316)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:189)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:393)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1035)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1029)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        ... 32 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://controlnode-205.edp:9292/kms/v1/?op=GETDELEGATIONTOKEN&doAs=hive_test&renewer=hive_test&user.name=hive_test, status: 403, message: Forbidden
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:403)
        at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:74)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:213)
        ... 42 more
21/12/29 15:04:08 ERROR kms.LoadBalancingKMSClientProvider: Aborting since the Request has failed with all KMS providers(depending on hadoop.security.kms.client.failover.max.retries=1 setting and numProviders=1) in the group OR the exception is not recoverable
21/12/29 15:04:08 WARN credentials.HadoopCredentialsManager: Failed to update tokens for hive_test, try again in 60000 ms
java.io.IOException: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationToken(KMSClientProvider.java:1051)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:255)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$1.call(LoadBalancingKMSClientProvider.java:252)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
        at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.getDelegationToken(LoadBalancingKMSClientProvider.java:252)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:95)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.collectDelegationTokens(DelegationTokenIssuer.java:107)
        at org.apache.hadoop.security.token.DelegationTokenIssuer.addDelegationTokens(DelegationTokenIssuer.java:76)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$2(HadoopFsDelegationTokenProvider.scala:67)
        at scala.collection.immutable.Set$Set1.foreach(Set.scala:141)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.$anonfun$obtainDelegationTokens$1(HadoopFsDelegationTokenProvider.scala:65)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$$anon$1.run(HadoopFsDelegationTokenProvider.scala:107)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider$.doAsProxyUser(HadoopFsDelegationTokenProvider.scala:106)
        at org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.obtainDelegationTokens(HadoopFsDelegationTokenProvider.scala:58)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1(HadoopCredentialsManager.scala:210)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.$anonfun$run$1$adapted(HadoopCredentialsManager.scala:210)
        at scala.collection.Iterator.foreach(Iterator.scala:943)
        at scala.collection.Iterator.foreach$(Iterator.scala:943)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
        at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:214)
        at org.apache.kyuubi.credentials.HadoopCredentialsManager$$anon$1.run(HadoopCredentialsManager.scala:210)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.UndeclaredThrowableException
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider.getDelegationToken(KMSClientProvider.java:1029)
        ... 31 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Error while authenticating with endpoint: http://controlnode-205.edp:9292/kms/v1/?op=GETDELEGATIONTOKEN&doAs=hive_test&renewer=hive_test
        at sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.wrapExceptionWithMessage(KerberosAuthenticator.java:232)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:219)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:350)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:316)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:189)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:393)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1035)
        at org.apache.hadoop.crypto.key.kms.KMSClientProvider$3.run(KMSClientProvider.java:1029)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        ... 32 more
Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, URL: http://controlnode-205.edp:9292/kms/v1/?op=GETDELEGATIONTOKEN&doAs=hive_test&renewer=hive_test&user.name=hive_test, status: 403, message: Forbidden
        at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:403)
        at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:74)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:143)
        at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:213)
        ... 42 more
21/12/29 15:04:08 INFO credentials.HadoopCredentialsManager: Scheduling renewal in 60000 ms.

could explain the reason why this happens?

yaooqinn · 2021-12-28T09:47:38Z

yaooqinn
Dec 28, 2021
Collaborator

can you try kyuubi.engine.share.level=CONNECTION

1 reply

turtoise Dec 28, 2021
Author

I tried what you said,but it won't work. The same error happens. Also,i checked the doc https://kyuubi.readthedocs.io/en/latest/client/kerberized_kyuubi.html, not found anything wrong. Here is also a situation,my HDP environment has integrated with ranger, i don't know if it has any influence.

yaooqinn · 2021-12-28T10:03:25Z

yaooqinn
Dec 28, 2021
Collaborator

https://kyuubi.readthedocs.io/en/latest/client/kerberized_kyuubi.html FYI, hope this doc helps

0 replies

pan3793 · 2021-12-29T04:58:25Z

pan3793
Dec 29, 2021
Collaborator

cc @zhouyifan279

0 replies

zhouyifan279 · 2022-01-04T06:46:35Z

zhouyifan279
Jan 4, 2022
Collaborator

@turtoise The KMS provider exception is probably a bug of Kyuubi's HadoopCredentialsManager. Could you set kyuubi.credentials.hadoopfs.enabled to false in kyuubi-defaults.conf, restart kyuubi server and try command:

beeline -u "jdbc:hive2://controlnode-205.edp:10009/;principal=hive_test/client-209.edp@HADOOP.COM;

Then paste kyuubi server log and engine log here.

Thanks.

6 replies

zhouyifan279 Jan 17, 2022
Collaborator

@turtoise After reading related code in both kyuubi and spark, I find that this error occurrs due to absence of proxy user configurations in Hadoop KMS conf file.

You can edit kms-site.xml according to Hadoop KMS guide to allow Kyuubi server principal (set in kyuubi.kinit.principal) to impersonate other users.

Then Kyuubi server will be able to get KMS delegation tokens for SQL engine when kyuubi.credentials.hadoopfs.enabled=true.

Note: --principal and --keytab args in spark-submit are not necessary after you enabled Kyuubi server principal to impersonate other users.

turtoise Jan 17, 2022
Author

I edit the kms-site.xml file and restart the services, But it doesn't work.
the kyuubi server log reports same error as before.

turtoise Jan 24, 2022
Author

Ps: Recently, I tried add these three parameters in ranger-kms/conf/kms-site.xml of ranger-kms. It takes an effect. However,when i configure it in both kyuubi and hadoop,it does not work. Now,I can connect to kyuubi without --principal and --keytab args, even though I set kyuubi.credentials.hadoopfs.enabled=true in kyuubi-defaults.conf.

zhouyifan279 Jan 26, 2022
Collaborator

@turtoise Thanks for the info.

By your description, I think your HDFS is using Ranger KMS instead of Hadoop KMS.
As the proxy user parameters only takes effect at KMS server side, what you metioned about is the normal behavior.

xcl1102 Jun 26, 2024

@zhouyifan279 Will setting kyuubi.credentials.hadoopfs.enabled to false have any impact?
My service log error is as follows. Just change kyuubi.credentials.hadoopfs.enabled to false, but will it have any impact?

2024-06-26 19:13:41.284 ERROR KyuubiSessionManager-exec-pool: Thread-112 org.apache.kyuubi.session.KyuubiSessionImpl: Failed to renew engine credentials for SessionHandle [02225d5e-191b-4b6f-9cac-6c8c32d4325f] org.apache.kyuubi.KyuubiException: Exception thrown in awaitResult: at org.apache.kyuubi.util.ThreadUtils$.awaitResult(ThreadUtils.scala:78) at org.apache.kyuubi.credentials.CredentialsRef.$anonfun$waitUntilReady$1(CredentialsRef.scala:70) at org.apache.kyuubi.credentials.CredentialsRef.$anonfun$waitUntilReady$1$adapted(CredentialsRef.scala:70) at scala.Option.foreach(Option.scala:407) at org.apache.kyuubi.credentials.CredentialsRef.waitUntilReady(CredentialsRef.scala:70) at org.apache.kyuubi.credentials.HadoopCredentialsManager.getOrCreateUserCredentialsRef(HadoopCredentialsManager.scala:226) at org.apache.kyuubi.credentials.HadoopCredentialsManager.renewCredentials(HadoopCredentialsManager.scala:161) at org.apache.kyuubi.session.KyuubiSessionImpl.renewEngineCredentials(KyuubiSessionImpl.scala:240) at org.apache.kyuubi.session.KyuubiSessionImpl.engineCredentials$lzycompute(KyuubiSessionImpl.scala:73) at org.apache.kyuubi.session.KyuubiSessionImpl.engineCredentials(KyuubiSessionImpl.scala:73) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:126) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49) at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:123) at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$2(LaunchEngine.scala:60) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [60000 milliseconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:259) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:263) at org.apache.kyuubi.util.ThreadUtils$.awaitResult(ThreadUtils.scala:75) ... 21 more

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR cannot connect to kerberized kyuubi using beeline #1638

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 12 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ERROR cannot connect to kerberized kyuubi using beeline #1638

turtoise Dec 28, 2021

Replies: 5 comments · 12 replies

yaooqinn Dec 28, 2021 Collaborator

turtoise Dec 28, 2021 Author

ulysses-you Dec 28, 2021 Collaborator

turtoise Dec 29, 2021 Author

ulysses-you Dec 29, 2021 Collaborator

turtoise Dec 29, 2021 Author

yaooqinn Dec 28, 2021 Collaborator

turtoise Dec 28, 2021 Author

yaooqinn Dec 28, 2021 Collaborator

pan3793 Dec 29, 2021 Collaborator

zhouyifan279 Jan 4, 2022 Collaborator

zhouyifan279 Jan 17, 2022 Collaborator

turtoise Jan 17, 2022 Author

turtoise Jan 24, 2022 Author

zhouyifan279 Jan 26, 2022 Collaborator

xcl1102 Jun 26, 2024

turtoise
Dec 28, 2021

Replies: 5 comments 12 replies

yaooqinn
Dec 28, 2021
Collaborator

turtoise Dec 28, 2021
Author

ulysses-you Dec 28, 2021
Collaborator

turtoise Dec 29, 2021
Author

ulysses-you Dec 29, 2021
Collaborator

turtoise Dec 29, 2021
Author

yaooqinn
Dec 28, 2021
Collaborator

turtoise Dec 28, 2021
Author

yaooqinn
Dec 28, 2021
Collaborator

pan3793
Dec 29, 2021
Collaborator

zhouyifan279
Jan 4, 2022
Collaborator

zhouyifan279 Jan 17, 2022
Collaborator

turtoise Jan 17, 2022
Author

turtoise Jan 24, 2022
Author

zhouyifan279 Jan 26, 2022
Collaborator