HADOOP-16477. S3 delegation token tests fail if fs.s3a.encryption.key set #1210

steveloughran · 2019-08-01T18:33:46Z

Delegation Token In FileSystem tests unset more options and compare propagation of entries
Added a spurious ITestRoleDelegationInFilesystem override so I can debug things better.
SSEC tests fail too...need to understand that

It looks like the AWS Permissions Policy created for the Role DT (and passed into AssumeRole) isn't asking for the right KMS permissions for a PUT request to work. At least, it's mkdirs that fails first...we should verify read of existing data too). I don't currently understand what is wrong with the permissions I am asking for.

Or maybe it's actually the role which is blocked? That would explain things -the code is valid, but the role is too restricted. Needs more investigation, and no doubt some more entries in the troubleshooting docs

Change-Id: Icbd418f9aa6c72312d39b4d94a1f2a2854fca059

steveloughran · 2019-08-01T18:34:26Z

Testing: S3 Ireland with s3guard and KMS. Currently failing

steveloughran · 2019-08-01T18:44:02Z

Update. Yes, the IAM role I was assuming didn't have the right permissions. Plan

Revert the desperate attempts to widen the permissions
Add it to the DT troubleshooting docs.
make sure the tests are doing a full read of data created in the original FS, so verifying decryption propagates

steveloughran · 2019-08-02T21:19:50Z

Assumed role/DT tests all happy once IAM role is fixed up

SSE-C tests failing, with cause (As seen by debugger) that the kms key option isn't being unset. If the options were being set in a jceks file I'd understand this (you can't unset those & they take priority over Config map changes), but that's not it .Needs more debug time

steveloughran · 2019-10-24T13:06:52Z

HADOOP-16626 should help set things up reliably here.

hadoop-yetus · 2019-10-24T16:10:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	37	Docker mode activated.
		_ Prechecks _
+1	dupname	0	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 12 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1075	trunk passed
+1	compile	36	trunk passed
+1	checkstyle	28	trunk passed
+1	mvnsite	41	trunk passed
+1	shadedclient	827	branch has no errors when building and testing our client artifacts.
+1	javadoc	30	trunk passed
0	spotbugs	60	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	58	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	34	the patch passed
+1	compile	29	the patch passed
+1	javac	29	hadoop-tools_hadoop-aws generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16)
-0	checkstyle	20	hadoop-tools/hadoop-aws: The patch generated 3 new + 12 unchanged - 0 fixed = 15 total (was 12)
+1	mvnsite	33	the patch passed
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	817	patch has no errors when building and testing our client artifacts.
+1	javadoc	27	the patch passed
+1	findbugs	62	the patch passed
		_ Other Tests _
+1	unit	84	hadoop-aws in the patch passed.
+1	asflicense	34	The patch does not generate ASF License warnings.
		3369

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/11/artifact/out/Dockerfile
GITHUB PR	#1210
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 96f848993b43 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `ee699dc`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/11/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/11/testReport/
Max. process+thread count	449 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/11/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran · 2019-10-28T19:18:06Z

This PR also handles the case where the config explicitly sets the DT binding for the store, which needs to be unset for all tests playing with AWS auth chains

sidseth · 2019-10-30T23:18:19Z

Not sure if this is ready for review or not. I'm +1 if the tests are passing.
One thing that would be useful would be to document the change in the Roles which is mentioned in one of the comments.

steveloughran · 2019-11-12T11:03:07Z

thanks for the vote Sid, will review the docs and retest

… set * Delegation Token In FileSystem tests unset more options and compare propagation of entries * Added a spurious ITestRoleDelegationInFilesystem override so I can debug things better. SSEC tests fail too...need to understand that Change-Id: Icbd418f9aa6c72312d39b4d94a1f2a2854fca059

Change-Id: Iac491d15bcb088b93cea94401c6ec41076095b8d

Rebased to trunk with the changes of HADOOP-16626 stopping per-bucket options creeping back in. I believe that issue is why the patch failed to work before Change-Id: I4ab65cd99fe4c5e8b5594f436e50483bd63e7983

* all IAM roles need to include KMS R/W access * unset encryption for the checksum test * remove DT Binding when it interferes with tests of AWS credential providers. Change-Id: I2e532f3a272e76bb3f97af313d4454e28ce69e97

…MSE problem. Also changed `operation` on writes from `put` to `Writing Object` as that is what is actually happening. Change-Id: Ib3491984faaf9c15c6711a6c65b9c2eca28ddeb6

hadoop-yetus · 2019-11-12T16:24:19Z

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md

+
+```
+java.nio.file.AccessDeniedException: test/testDTFileSystemClient: Writing Object on test/testDTFileSystemClient:
+  com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; 


whitespace:end of line

hadoop-yetus · 2019-11-12T16:24:21Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	78	Docker mode activated.
		_ Prechecks _
+1	dupname	1	No case conflicting files found.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 12 new or modified test files.
		_ trunk Compile Tests _
+1	mvninstall	1259	trunk passed
+1	compile	32	trunk passed
+1	checkstyle	25	trunk passed
+1	mvnsite	37	trunk passed
+1	shadedclient	886	branch has no errors when building and testing our client artifacts.
+1	javadoc	25	trunk passed
0	spotbugs	59	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	56	trunk passed
		_ Patch Compile Tests _
+1	mvninstall	32	the patch passed
+1	compile	27	the patch passed
+1	javac	27	hadoop-tools_hadoop-aws generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16)
-0	checkstyle	18	hadoop-tools/hadoop-aws: The patch generated 3 new + 13 unchanged - 0 fixed = 16 total (was 13)
+1	mvnsite	32	the patch passed
-1	whitespace	0	The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1	shadedclient	934	patch has no errors when building and testing our client artifacts.
+1	javadoc	26	the patch passed
+1	findbugs	76	the patch passed
		_ Other Tests _
-1	unit	85	hadoop-aws in the patch failed.
+1	asflicense	31	The patch does not generate ASF License warnings.
		3744

Reason	Tests
Failed junit tests	hadoop.fs.s3a.s3guard.TestNullMetadataStore

Subsystem	Report/Notes
Docker	Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/12/artifact/out/Dockerfile
GITHUB PR	#1210
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux e5e887d9efc9 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / fb512f5
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/12/artifact/out/diff-checkstyle-hadoop-tools_hadoop-aws.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/12/artifact/out/whitespace-eol.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/12/artifact/out/patch-unit-hadoop-tools_hadoop-aws.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/12/testReport/
Max. process+thread count	421 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1210/12/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

steveloughran added bug fs/s3 changes related to hadoop-aws; submitter must declare test endpoint work in progress PRs still Work in Progress; reviews not expected but still welcome labels Aug 1, 2019

apache deleted a comment from hadoop-yetus Oct 24, 2019

steveloughran force-pushed the s3/HADOOP-16477-kms branch from 4224b06 to a8cb78b Compare October 24, 2019 13:39

steveloughran requested review from bgaborg and sidseth October 28, 2019 19:30

steveloughran added 5 commits November 12, 2019 14:15

HADOOP-16477 not managing to reset the SSE-C keys

6d7031a

Change-Id: Iac491d15bcb088b93cea94401c6ec41076095b8d

HADOOP-16477. make sure DT binding is cleared.

10c7e54

Rebased to trunk with the changes of HADOOP-16626 stopping per-bucket options creeping back in. I believe that issue is why the patch failed to work before Change-Id: I4ab65cd99fe4c5e8b5594f436e50483bd63e7983

HADOOP-16477: improve assumed role docs with troubleshooting on the K…

aca67f5

…MSE problem. Also changed `operation` on writes from `put` to `Writing Object` as that is what is actually happening. Change-Id: Ib3491984faaf9c15c6711a6c65b9c2eca28ddeb6

steveloughran force-pushed the s3/HADOOP-16477-kms branch from 9cf6f07 to aca67f5 Compare November 12, 2019 15:20

hadoop-yetus reviewed Nov 12, 2019

View reviewed changes

steveloughran closed this Nov 17, 2019

steveloughran deleted the s3/HADOOP-16477-kms branch October 15, 2021 19:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-16477. S3 delegation token tests fail if fs.s3a.encryption.key set #1210

HADOOP-16477. S3 delegation token tests fail if fs.s3a.encryption.key set #1210

Uh oh!

steveloughran commented Aug 1, 2019

Uh oh!

steveloughran commented Aug 1, 2019

Uh oh!

steveloughran commented Aug 1, 2019

Uh oh!

steveloughran commented Aug 2, 2019

Uh oh!

steveloughran commented Oct 24, 2019

Uh oh!

hadoop-yetus commented Oct 24, 2019

Uh oh!

steveloughran commented Oct 28, 2019

Uh oh!

sidseth commented Oct 30, 2019

Uh oh!

steveloughran commented Nov 12, 2019

Uh oh!

hadoop-yetus Nov 12, 2019

Uh oh!

hadoop-yetus commented Nov 12, 2019

Uh oh!

Uh oh!

HADOOP-16477. S3 delegation token tests fail if fs.s3a.encryption.key set #1210

HADOOP-16477. S3 delegation token tests fail if fs.s3a.encryption.key set #1210

Uh oh!

Conversation

steveloughran commented Aug 1, 2019

Uh oh!

steveloughran commented Aug 1, 2019

Uh oh!

steveloughran commented Aug 1, 2019

Uh oh!

steveloughran commented Aug 2, 2019

Uh oh!

steveloughran commented Oct 24, 2019

Uh oh!

hadoop-yetus commented Oct 24, 2019

Uh oh!

steveloughran commented Oct 28, 2019

Uh oh!

sidseth commented Oct 30, 2019

Uh oh!

steveloughran commented Nov 12, 2019

Uh oh!

hadoop-yetus Nov 12, 2019

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Nov 12, 2019

Uh oh!

Uh oh!