HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager #1894

ghanko · 2020-03-12T07:02:33Z

No description provided.

hadoop-yetus · 2020-07-31T13:05:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 2s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ trunk Compile Tests _
+1 💚	mvninstall	24m 25s	trunk passed
+1 💚	compile	22m 9s	trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	compile	20m 30s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	checkstyle	0m 46s	trunk passed
+1 💚	mvnsite	1m 33s	trunk passed
+1 💚	shadedclient	18m 52s	branch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 38s	hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚	javadoc	0m 59s	trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+0 🆗	spotbugs	2m 35s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	2m 32s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 58s	the patch passed
+1 💚	compile	22m 1s	the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
+1 💚	javac	22m 1s	the patch passed
+1 💚	compile	19m 5s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
+1 💚	javac	19m 5s	the patch passed
+1 💚	checkstyle	0m 49s	the patch passed
+1 💚	mvnsite	1m 39s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	16m 14s	patch has no errors when building and testing our client artifacts.
-1 ❌	javadoc	0m 36s	hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.
+1 💚	javadoc	0m 58s	the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09
-1 ❌	findbugs	2m 26s	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
		_ Other Tests _
+1 💚	unit	10m 14s	hadoop-common in the patch passed.
+1 💚	asflicense	0m 45s	The patch does not generate ASF License warnings.
		172m 49s

Reason	Tests
FindBugs	module:hadoop-common-project/hadoop-common
	Inconsistent synchronization of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.currentKey; locked 88% of time Unsynchronized access at AbstractDelegationTokenSecretManager.java:88% of time Unsynchronized access at AbstractDelegationTokenSecretManager.java:[line 379]

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/3/artifact/out/Dockerfile
GITHUB PR	#1894
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 71f762727e41 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `e756fe3`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09
javadoc	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
javadoc	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/3/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
findbugs	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/3/artifact/out/new-findbugs-hadoop-common-project_hadoop-common.html
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/3/testReport/
Max. process+thread count	2394 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/3/console
versions	git=2.17.1 maven=3.6.0 findbugs=4.0.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2021-03-11T15:21:19Z

...n/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java

      storeDelegationKey(currentKey);
    }
+    //Log must be invoked outside the lock on 'this'
+    logUpdateMasterKey(currentKey);


so this is now happening after the store? And both generateSecret and store are now synchronized?

Yes, but I'm not sure why it was placed between them. They have to be in the same synchronized block to be atomic, otherwise other threads may find that there's no key for the current id.

I see. Probably it's just the original order and then sync blocks went up around some of the ops

steveloughran · 2021-03-11T15:22:44Z

looking at this with a goal of getting it in to 3.3.1.

Is the race condition in generating the key IDs? If so, isn't that enough to sync and we can leave the rest alone?

ghanko · 2021-03-16T17:35:18Z

No, the problem is that key id generation and store happens in distinct synchronized blocks so between the two blocks other threads can access currentKeyId which doesn't correspond to any key because it's not stored yet.

steveloughran · 2021-03-17T11:10:00Z

OK., now I understand why we need this and why the PR does the right thing,

Can you do rebase and forced push of the PR to kick off yetus again?

…cretManager Change-Id: Id29a4e2471854659a7506d5d82da8e98827b6593

hadoop-yetus · 2021-03-17T18:19:47Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 6s		trunk passed
+1 💚	compile	27m 10s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	compile	23m 18s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	checkstyle	1m 11s		trunk passed
+1 💚	mvnsite	1m 46s		trunk passed
+1 💚	javadoc	1m 16s		trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 43s		trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	spotbugs	2m 44s		trunk passed
+1 💚	shadedclient	20m 34s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 5s		the patch passed
+1 💚	compile	26m 18s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javac	26m 18s		the patch passed
+1 💚	compile	22m 50s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
+1 💚	javac	22m 50s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 12s		the patch passed
+1 💚	mvnsite	1m 58s		the patch passed
+1 💚	javadoc	1m 11s		the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
+1 💚	javadoc	1m 47s		the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
-1 ❌	spotbugs	2m 48s	/new-spotbugs-hadoop-common-project_hadoop-common.html	hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	shadedclient	20m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	17m 58s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		217m 18s

Reason	Tests
SpotBugs	module:hadoop-common-project/hadoop-common
	Inconsistent synchronization of org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.currentKey; locked 88% of time Unsynchronized access at AbstractDelegationTokenSecretManager.java:88% of time Unsynchronized access at AbstractDelegationTokenSecretManager.java:[line 379]
Failed junit tests	hadoop.security.TestRaceWhenRelogin

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/6/artifact/out/Dockerfile
GITHUB PR	#1894
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux 0e96f88fb06a 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28 05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `759aa2b`
Default Java	Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/6/testReport/
Max. process+thread count	3134 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1894/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

This comment has been minimized.

Sign in to view

steveloughran reviewed Mar 11, 2021

View reviewed changes

HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSe…

759aa2b

…cretManager Change-Id: Id29a4e2471854659a7506d5d82da8e98827b6593

ghanko force-pushed the hadoop-16819 branch from a42478d to 759aa2b Compare March 17, 2021 14:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager #1894

HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager #1894

ghanko commented Mar 12, 2020

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jul 31, 2020

Uh oh!

steveloughran Mar 11, 2021

Uh oh!

ghanko Mar 16, 2021

Uh oh!

steveloughran Mar 17, 2021

Uh oh!

steveloughran commented Mar 11, 2021

Uh oh!

ghanko commented Mar 16, 2021

Uh oh!

steveloughran commented Mar 17, 2021

Uh oh!

hadoop-yetus commented Mar 17, 2021

Uh oh!

Uh oh!

HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager #1894

Are you sure you want to change the base?

HADOOP-16819 Possible inconsistent state of AbstractDelegationTokenSecretManager #1894

Conversation

ghanko commented Mar 12, 2020

Uh oh!

This comment has been minimized.

hadoop-yetus commented Jul 31, 2020

Uh oh!

steveloughran Mar 11, 2021

Choose a reason for hiding this comment

Uh oh!

ghanko Mar 16, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran Mar 17, 2021

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Mar 11, 2021

Uh oh!

ghanko commented Mar 16, 2021

Uh oh!

steveloughran commented Mar 17, 2021

Uh oh!

hadoop-yetus commented Mar 17, 2021

Uh oh!

Uh oh!