HADOOP-16653. S3Guard DDB overreacts to no tag access #1660

bgaborg · 2019-10-16T11:36:50Z

Change-Id: I4990e61dcf1baa7ddb593ac9a2195e9338060539

hadoop-yetus · 2019-10-16T12:41:23Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	81	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1254	trunk passed
+1	compile	32	trunk passed
+1	checkstyle	24	trunk passed
+1	mvnsite	36	trunk passed
+1	shadedclient	878	branch has no errors when building and testing our client artifacts.
+1	javadoc	25	trunk passed
0	spotbugs	59	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	57	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	32	the patch passed
+1	compile	28	the patch passed
+1	javac	28	the patch passed
-0	checkstyle	18	hadoop-tools/hadoop-aws: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
+1	mvnsite	31	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	876	patch has no errors when building and testing our client artifacts.
+1	javadoc	22	the patch passed
+1	findbugs	64	the patch passed
		_ Other Tests _
+1	unit	91	hadoop-aws in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3671

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/2/artifact/out/Dockerfile
GITHUB PR	#1660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux c36ba57754d3 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `bbcf0b9`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/2/testReport/
Max. process+thread count	338 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/2/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-10-16T12:42:16Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	104	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1421	trunk passed
+1	compile	38	trunk passed
+1	checkstyle	25	trunk passed
+1	mvnsite	37	trunk passed
+1	shadedclient	874	branch has no errors when building and testing our client artifacts.
+1	javadoc	26	trunk passed
0	spotbugs	59	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	57	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	34	the patch passed
+1	compile	27	the patch passed
+1	javac	27	the patch passed
-0	checkstyle	18	hadoop-tools/hadoop-aws: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
+1	mvnsite	30	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	879	patch has no errors when building and testing our client artifacts.
+1	javadoc	23	the patch passed
+1	findbugs	61	the patch passed
		_ Other Tests _
+1	unit	77	hadoop-aws in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3862

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/1/artifact/out/Dockerfile
GITHUB PR	#1660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 558e42d447a4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `bbcf0b9`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/1/testReport/
Max. process+thread count	421 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-10-16T14:38:42Z

...op-aws/src/main/java/org/apache/hadoop/fs/s3a/s3guard/DynamoDBMetadataStoreTableManager.java

+        LOG.info("Table {} contains no version marker item and tag. " +
+            "The table is empty, so the version marker will be added " +
+            "as TAG and ITEM.", tableName);
+        putVersionMarkerItemToTable();


you've swapped the order. Does this matter?

steveloughran · 2019-10-16T14:41:09Z

hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/auth/ITestAssumeRole.java

@@ -756,4 +756,25 @@ public void testBucketLocationForbidden() throws Throwable {
    Assertions.assertThat(info)
        .contains(S3GuardTool.BucketInfo.LOCATION_UNKNOWN);
  }
+  /**
+   * Turn off access to dynamo DB Tags and see how DDB table init copes.


maybe add: There's no testing of the codepath other than checking the logs -this test does make sure that no regression stops the tag permission failures from halting the client

steveloughran · 2019-10-16T14:41:44Z

Tested against?

steveloughran · 2019-10-16T14:42:57Z

LGTM, other than the detail of you haven't stated your test endpoint.

bgaborg · 2019-10-16T14:46:16Z

There are some failing itests:

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 152.399 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps
[ERROR] testConcurrentTableCreations(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps)  Time elapsed: 152.302 s  <<< ERROR!
java.lang.IllegalArgumentException: Table s3guard.test.testConcurrentTableCreations-670326112 is not deleted.
	at com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:505)
	at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps.deleteTable(ITestS3GuardConcurrentOps.java:87)
	at org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardConcurrentOps.testConcurrentTableCreations(ITestS3GuardConcurrentOps.java:178)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.waiters.WaiterTimedOutException: Reached maximum attempts without transitioning to the desired state
	at com.amazonaws.waiters.WaiterExecution.pollResource(WaiterExecution.java:86)
	at com.amazonaws.waiters.WaiterImpl.run(WaiterImpl.java:88)
	at com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:502)
	... 16 more

[ERROR] Tests run: 53, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 461.299 s <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore
[ERROR] testAncestorOverwriteConflict(org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore)  Time elapsed: 132.144 s  <<< ERROR!
java.lang.IllegalArgumentException: Table TEST_TABLE is not deleted.
	at com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:505)
	at org.apache.hadoop.fs.s3a.s3guard.ITestDynamoDBMetadataStore.setUp(ITestDynamoDBMetadataStore.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.waiters.WaiterTimedOutException: Reached maximum attempts without transitioning to the desired state
	at com.amazonaws.waiters.WaiterExecution.pollResource(WaiterExecution.java:86)
	at com.amazonaws.waiters.WaiterImpl.run(WaiterImpl.java:88)
	at com.amazonaws.services.dynamodbv2.document.Table.waitForDelete(Table.java:502)
	... 14 more

tested against ireland.
these test failures are not related, however these should be addressed.

Change-Id: I0cfe7b33ff9529442e66b64e6e4db653aa8d8131

hadoop-yetus · 2019-10-16T16:05:12Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	76	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1218	trunk passed
+1	compile	32	trunk passed
+1	checkstyle	23	trunk passed
+1	mvnsite	35	trunk passed
+1	shadedclient	864	branch has no errors when building and testing our client artifacts.
+1	javadoc	24	trunk passed
0	spotbugs	58	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	56	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	33	the patch passed
+1	compile	27	the patch passed
+1	javac	27	the patch passed
-0	checkstyle	18	hadoop-tools/hadoop-aws: The patch generated 2 new + 10 unchanged - 0 fixed = 12 total (was 10)
+1	mvnsite	32	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	886	patch has no errors when building and testing our client artifacts.
+1	javadoc	23	the patch passed
+1	findbugs	63	the patch passed
		_ Other Tests _
+1	unit	79	hadoop-aws in the patch passed.
+1	asflicense	29	The patch does not generate ASF License warnings.
		3615

Subsystem	Report/Notes
Docker	Client=19.03.3 Server=19.03.3 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/3/artifact/out/Dockerfile
GITHUB PR	#1660
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 27e31c54f44b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `375224e`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/3/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/3/testReport/
Max. process+thread count	421 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1660/3/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

bgaborg · 2019-10-24T09:24:16Z

is there anything needed to do to get this in?

steveloughran

LGTM
+1

…Contributed by Gabor Bota.

…Contributed by Gabor Bota. (cherry picked from commit d5e9971) Change-Id: I40212336785b5a7c1f8a194cb197558c58b0e1be

…Contributed by Gabor Bota.

bgaborg requested a review from steveloughran October 16, 2019 11:36

HADOOP-16653. S3Guard DDB overreacts to no tag access

dae0183

Change-Id: I4990e61dcf1baa7ddb593ac9a2195e9338060539

bgaborg force-pushed the HADOOP-16653-silent-notag-access branch from b1299cc to dae0183 Compare October 16, 2019 11:39

steveloughran reviewed Oct 16, 2019

View reviewed changes

additional javadoc

c3b145f

Change-Id: I0cfe7b33ff9529442e66b64e6e4db653aa8d8131

steveloughran self-assigned this Oct 24, 2019

steveloughran approved these changes Oct 24, 2019

View reviewed changes

bgaborg merged commit d5e9971 into apache:trunk Oct 28, 2019

bgaborg deleted the HADOOP-16653-silent-notag-access branch October 28, 2019 10:22

amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019

HADOOP-16653. S3Guard DDB overreacts to no tag access (apache#1660). …

06e8718

…Contributed by Gabor Bota.

RogPodge pushed a commit to RogPodge/hadoop that referenced this pull request Mar 25, 2020

HADOOP-16653. S3Guard DDB overreacts to no tag access (apache#1660). …

a364da0

…Contributed by Gabor Bota.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16653. S3Guard DDB overreacts to no tag access #1660

HADOOP-16653. S3Guard DDB overreacts to no tag access #1660

Uh oh!

bgaborg commented Oct 16, 2019

Uh oh!

hadoop-yetus commented Oct 16, 2019

Uh oh!

hadoop-yetus commented Oct 16, 2019

Uh oh!

steveloughran Oct 16, 2019

Uh oh!

steveloughran Oct 16, 2019

Uh oh!

steveloughran commented Oct 16, 2019

Uh oh!

steveloughran commented Oct 16, 2019

Uh oh!

bgaborg commented Oct 16, 2019

Uh oh!

hadoop-yetus commented Oct 16, 2019

Uh oh!

bgaborg commented Oct 24, 2019

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

HADOOP-16653. S3Guard DDB overreacts to no tag access #1660

HADOOP-16653. S3Guard DDB overreacts to no tag access #1660

Uh oh!

Conversation

bgaborg commented Oct 16, 2019

Uh oh!

hadoop-yetus commented Oct 16, 2019

Uh oh!

hadoop-yetus commented Oct 16, 2019

Uh oh!

steveloughran Oct 16, 2019

Choose a reason for hiding this comment

Uh oh!

steveloughran Oct 16, 2019

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Oct 16, 2019

Uh oh!

steveloughran commented Oct 16, 2019

Uh oh!

bgaborg commented Oct 16, 2019

Uh oh!

hadoop-yetus commented Oct 16, 2019

Uh oh!

bgaborg commented Oct 24, 2019

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!