You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JIRA:YARN-9708. Yarn Router Support DelegationToken.
In the YARN non-federation mode, RMDelegationTokenIdentifier is generated, updated, and deleted by RM.
In the YARN federation mode, the Router replaces the RM to interact with the Client, so the operation of the DelegationToken originally performed by the RM will be replaced by the Router.
In this process, it is necessary to ensure that the original generation method of DelegationToken remains unchanged.
1.RM's management of DelegationToken
RMDelegationTokenIdentifier is managed by RMDelegationTokenSecretManager , RMDelegationTokenSecretManager inherits from AbstractDelegationTokenSecretManager .
AbstractDelegationTokenSecretManager is responsible for the generation of MasterKey and Token.
RMDelegationTokenSecretManager is responsible for storing MasterKey and Token.
RMDelegationTokenSecretManager contains 5 important methods.
storeNewMasterKey
removeStoredMasterKey
storeNewToken
updateStoredToken
removeStoredToken
In the process of persisting Token, RM mainly relies on RMStateStore for storage, among which there are two key implementations, MemoryRMStateStore, ZKRMStateStore.
2.Router's management of DelegationToken
Similar to RM's RMDelegationTokenSecretManager, the Router will implement RouterDelegationTokenSecretManager.
When Storing MasterKey and Token, use the FederationStateStore of Yarn Federation directly, and plan to implement a storage solution based on memory, ZK, and MySQL.
This pr only implements the memory-based storage scheme. The storage scheme based on ZK and Mysql will be submitted separately for pr implementation.
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗
mvndep
15m 17s
Maven dependency ordering for branch
+1 💚
mvninstall
28m 40s
trunk passed
+1 💚
compile
4m 16s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
compile
3m 29s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
checkstyle
1m 29s
trunk passed
+1 💚
mvnsite
2m 47s
trunk passed
+1 💚
javadoc
2m 30s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
javadoc
2m 16s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
spotbugs
4m 46s
trunk passed
+1 💚
shadedclient
23m 20s
branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗
mvndep
0m 30s
Maven dependency ordering for patch
+1 💚
mvninstall
2m 7s
the patch passed
+1 💚
compile
4m 31s
the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
Suspicious comparison of Integer references in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:[line 123]
Suspicious comparison of Long references in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:[line 123]
Possible null pointer dereference of identifier in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setIdentifier(YARNDelegationTokenIdentifier) Dereferenced at RouterStoreTokenPBImpl.java:identifier in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setIdentifier(YARNDelegationTokenIdentifier) Dereferenced at RouterStoreTokenPBImpl.java:[line 91]
Possible null pointer dereference of renewDate in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setRenewDate(Long) Dereferenced at RouterStoreTokenPBImpl.java:renewDate in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setRenewDate(Long) Dereferenced at RouterStoreTokenPBImpl.java:[line 100]
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗
mvndep
14m 50s
Maven dependency ordering for branch
+1 💚
mvninstall
30m 43s
trunk passed
+1 💚
compile
4m 30s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
compile
3m 35s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
checkstyle
1m 26s
trunk passed
+1 💚
mvnsite
2m 52s
trunk passed
+1 💚
javadoc
2m 28s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
javadoc
2m 17s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
spotbugs
5m 38s
trunk passed
+1 💚
shadedclient
24m 22s
branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗
mvndep
0m 26s
Maven dependency ordering for patch
+1 💚
mvninstall
2m 10s
the patch passed
+1 💚
compile
4m 21s
the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
Suspicious comparison of Integer references in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:[line 123]
Suspicious comparison of Long references in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:[line 123]
Possible null pointer dereference of identifier in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setIdentifier(YARNDelegationTokenIdentifier) Dereferenced at RouterStoreTokenPBImpl.java:identifier in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setIdentifier(YARNDelegationTokenIdentifier) Dereferenced at RouterStoreTokenPBImpl.java:[line 91]
Possible null pointer dereference of renewDate in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setRenewDate(Long) Dereferenced at RouterStoreTokenPBImpl.java:renewDate in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setRenewDate(Long) Dereferenced at RouterStoreTokenPBImpl.java:[line 100]
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗
mvndep
15m 1s
Maven dependency ordering for branch
+1 💚
mvninstall
25m 44s
trunk passed
+1 💚
compile
4m 6s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
compile
3m 31s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
checkstyle
1m 32s
trunk passed
+1 💚
mvnsite
3m 18s
trunk passed
+1 💚
javadoc
2m 33s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
javadoc
2m 28s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
spotbugs
4m 56s
trunk passed
+1 💚
shadedclient
22m 9s
branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗
mvndep
0m 33s
Maven dependency ordering for patch
+1 💚
mvninstall
2m 11s
the patch passed
+1 💚
compile
3m 53s
the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
Suspicious comparison of Integer references in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:[line 123]
Suspicious comparison of Long references in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:in org.apache.hadoop.yarn.server.federation.store.records.RouterMasterKey.equals(Object) At RouterMasterKey.java:[line 123]
Possible null pointer dereference of identifier in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setIdentifier(YARNDelegationTokenIdentifier) Dereferenced at RouterStoreTokenPBImpl.java:identifier in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setIdentifier(YARNDelegationTokenIdentifier) Dereferenced at RouterStoreTokenPBImpl.java:[line 91]
Possible null pointer dereference of renewDate in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setRenewDate(Long) Dereferenced at RouterStoreTokenPBImpl.java:renewDate in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreTokenPBImpl.setRenewDate(Long) Dereferenced at RouterStoreTokenPBImpl.java:[line 100]
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+0 🆗
mvndep
14m 21s
Maven dependency ordering for branch
+1 💚
mvninstall
28m 23s
trunk passed
+1 💚
compile
4m 13s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
compile
3m 28s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
checkstyle
1m 29s
trunk passed
+1 💚
mvnsite
2m 45s
trunk passed
+1 💚
javadoc
2m 30s
trunk passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
+1 💚
javadoc
2m 11s
trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚
spotbugs
4m 49s
trunk passed
+1 💚
shadedclient
23m 9s
branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+0 🆗
mvndep
0m 28s
Maven dependency ordering for patch
+1 💚
mvninstall
2m 7s
the patch passed
+1 💚
compile
4m 15s
the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRemoveStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRemoveStoredTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRemoveStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRemoveStoredTokenRequestPBImpl.java:[line 93]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreNewTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterStoreNewTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreNewTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterStoreNewTokenRequestPBImpl.java:[line 93]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenRequestPBImpl.java:[line 114]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenResponsePBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenResponsePBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenResponsePBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenResponsePBImpl.java:[line 114]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRemoveStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRemoveStoredTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRemoveStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRemoveStoredTokenRequestPBImpl.java:[line 98]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreNewTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterStoreNewTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterStoreNewTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterStoreNewTokenRequestPBImpl.java:[line 98]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenRequestPBImpl.java:[line 119]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenResponsePBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenResponsePBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterUpdateStoredTokenResponsePBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterUpdateStoredTokenResponsePBImpl.java:[line 114]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRMTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRMTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRMTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRMTokenRequestPBImpl.java:[line 97]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRMTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRMTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRMTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRMTokenRequestPBImpl.java:[line 97]
Null passed for non-null parameter of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRMTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRMTokenRequestPBImpl.java:of convertToProtoFormat(RouterStoreToken) in org.apache.hadoop.yarn.server.federation.store.records.impl.pb.RouterRMTokenRequestPBImpl.setRouterStoreToken(RouterStoreToken) Method invoked at RouterRMTokenRequestPBImpl.java:[line 97]
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA:YARN-9708. Yarn Router Support DelegationToken.
In the YARN non-federation mode,
RMDelegationTokenIdentifier
is generated, updated, and deleted by RM.In the YARN federation mode, the Router replaces the RM to interact with the Client, so the operation of the DelegationToken originally performed by the RM will be replaced by the Router.
In this process, it is necessary to ensure that the original generation method of DelegationToken remains unchanged.
1.RM's management of DelegationToken
RMDelegationTokenIdentifier
is managed byRMDelegationTokenSecretManager
,RMDelegationTokenSecretManager
inherits fromAbstractDelegationTokenSecretManager
.AbstractDelegationTokenSecretManager
is responsible for the generation of MasterKey and Token.RMDelegationTokenSecretManager
is responsible for storing MasterKey and Token.RMDelegationTokenSecretManager
contains 5 important methods.In the process of persisting Token, RM mainly relies on RMStateStore for storage, among which there are two key implementations, MemoryRMStateStore, ZKRMStateStore.
2.Router's management of DelegationToken
Similar to RM's
RMDelegationTokenSecretManager
, the Router will implementRouterDelegationTokenSecretManager
.When Storing MasterKey and Token, use the FederationStateStore of Yarn Federation directly, and plan to implement a storage solution based on memory, ZK, and MySQL.
This pr only implements the memory-based storage scheme. The storage scheme based on ZK and Mysql will be submitted separately for pr implementation.