HADOOP-16298. Manage/Renew delegation tokens for externally scheduled jobs

[cbaenziger]

Description of PR

This PR provides a means to reload authentication credentials from updated Hadoop delegation tokens similar to how one would reload credentials from an updated kerberos ticket.

How was this patch tested?

This patch has been tested via:

• The unit tests
• Via long-running (modified to use the refresh) HBase 1.x clients. Long-running means through many delegation token refresh cycles
• Further it has been used in non-long running Spark workloads which were delegation token based
• TODO: I would like to figure out how I can write a unit-test to validate [the reload for an HA HDFS|https://github.com/apache/hadoop/compare/trunk...cbaenziger:HADOOP-16298?expand=1#diff-e160685d647b1420f6c56264302c0e1c82ead52e75a31d91fb2a12f0ce9261c4R465-R481] but I can not find what seems like the correct layer to put such a test. I do not expect HDFS should be used in a hadoop-common layer test; is it okay to test UGI's behavior in hadoop-hdfs-project?

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [N/A] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [N/A] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 14s		trunk passed
+1 💚	compile	25m 1s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	compile	21m 3s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	1m 2s		trunk passed
+1 💚	mvnsite	1m 40s		trunk passed
+1 💚	javadoc	1m 8s		trunk passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	2m 32s		trunk passed
+1 💚	shadedclient	26m 1s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 4s		the patch passed
+1 💚	compile	24m 25s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javac	24m 25s		the patch passed
+1 💚	compile	21m 10s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚	javac	21m 10s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 59s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 11 new + 180 unchanged - 2 fixed = 191 total (was 182)
+1 💚	mvnsite	1m 36s		the patch passed
+1 💚	javadoc	1m 4s		the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 41s		the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
-1 ❌	spotbugs	2m 44s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
+1 💚	shadedclient	25m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	17m 50s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 49s		The patch does not generate ASF License warnings.
		214m 26s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Possible null pointer dereference of cred in org.apache.hadoop.security.UserGroupInformation.reloginFromDelegationTokens() Dereferenced at UserGroupInformation.java:cred in org.apache.hadoop.security.UserGroupInformation.reloginFromDelegationTokens() Dereferenced at UserGroupInformation.java:[line 892]
	Return value of Credentials.getAllTokens() ignored, but method has no side effect At UserGroupInformation.java:but method has no side effect At UserGroupInformation.java:[line 905]
Failed junit tests	hadoop.ipc.TestSaslRPC
	hadoop.ipc.TestIPC
	hadoop.security.TestUserGroupInformation

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4059/1/artifact/out/Dockerfile
GITHUB PR	#4059
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 870a57d85639 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29 06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 8075f9c
Default Java	Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4059/1/testReport/
Max. process+thread count	2202 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4059/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.