HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK #3292

mehakmeet · 2021-08-11T04:59:34Z

Region: ap-south-1.
mvn clean verify -Dparallel-tests -DtestsThreadCount=4 -Dscale

CSE ON, S3Guard ON

[INFO] Results:
[INFO] 
[WARNING] Tests run: 575, Failures: 0, Errors: 0, Skipped: 5

[INFO] Results:
[INFO] 
[WARNING] Tests run: 1476, Failures: 0, Errors: 0, Skipped: 1277

[INFO] Results:
[INFO] 
[WARNING] Tests run: 151, Failures: 0, Errors: 0, Skipped: 125

CSE ON, S3Guard OFF

[INFO] Results:
[INFO] 
[WARNING] Tests run: 575, Failures: 0, Errors: 0, Skipped: 5

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[ERROR]   ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[INFO] 
[ERROR] Tests run: 1476, Failures: 0, Errors: 2, Skipped: 641

[ERROR] Errors: 
[ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:267 » TestTimedOut
[INFO] 
[ERROR] Tests run: 151, Failures: 1, Errors: 1, Skipped: 28

CSE OFF, S3Guard ON

[INFO] Results:
[INFO] 
[WARNING] Tests run: 575, Failures: 0, Errors: 0, Skipped: 5

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestS3AFileSystemContract>FileSystemContractBaseTest.testLSRootDir:835->FileSystemContractBaseTest.assertListFilesFinds:850 » TestTimedOut
[ERROR]   ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[ERROR]   ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[INFO] 
[ERROR] Tests run: 1476, Failures: 0, Errors: 3, Skipped: 405

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestS3AContractRootDir.testListEmptyRootDirectory:82->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:196 » TestTimedOut
[ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:265 » TestTimedOut
[ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:101 » TestTimedOut
[INFO] 
[ERROR] Tests run: 11, Failures: 0, Errors: 3, Skipped: 0

CSE OFF, S3Guard OFF

[INFO] Results:
[INFO] 
[WARNING] Tests run: 575, Failures: 0, Errors: 0, Skipped: 5

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[ERROR]   ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut
[INFO] 
[ERROR] Tests run: 1476, Failures: 0, Errors: 2, Skipped: 467

[ERROR] Errors: 
[ERROR]   ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:267 » TestTimedOut
[INFO] 
[ERROR] Tests run: 151, Failures: 1, Errors: 1, Skipped: 28

apache#2706) This (big!) patch adds support for client side encryption in AWS S3, with keys managed by AWS-KMS. Read the documentation in encryption.md very, very carefully before use and consider it unstable. S3-CSE is enabled in the existing configuration option "fs.s3a.server-side-encryption-algorithm": fs.s3a.server-side-encryption-algorithm=CSE-KMS fs.s3a.server-side-encryption.key=<KMS_KEY_ID> You cannot enable CSE and SSE in the same client, although you can still enable a default SSE option in the S3 console. * Filesystem list/get status operations subtract 16 bytes from the length of all files >= 16 bytes long to compensate for the padding which CSE adds. * The SDK always warns about the specific algorithm chosen being deprecated. It is critical to use this algorithm for ranged GET requests to work (i.e. random IO). Ignore. * Unencrypted files CANNOT BE READ. The entire bucket SHOULD be encrypted with S3-CSE. * Uploading files may be a bit slower as blocks are now written sequentially. * The Multipart Upload API is disabled when S3-CSE is active. Contributed by Mehakmeet Singh

mukund-thakur · 2021-08-16T15:42:55Z

Ran tests using mvn -Dparallel-tests -DtestsThreadCount=8 clean verify with CSE on. All good .

mehakmeet · 2021-08-20T09:59:59Z

CC: @steveloughran

mehakmeet · 2021-09-16T07:04:23Z

Added the latest CSE Tuning. Tested successfully, but would be best if someone could once run the tests once in their setup before this goes in.

CC: @steveloughran @mukund-thakur

hadoop-yetus · 2021-09-16T07:45:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	11m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	6m 21s		Maven dependency ordering for branch
-1 ❌	mvninstall	1m 52s	/branch-mvninstall-root.txt	root in branch-3.3 failed.
-1 ❌	compile	0m 23s	/branch-compile-root.txt	root in branch-3.3 failed.
-0 ⚠️	checkstyle	0m 20s	/buildtool-branch-checkstyle-root.txt	The patch fails to run checkstyle in root
-1 ❌	mvnsite	0m 22s	/branch-mvnsite-hadoop-common-project_hadoop-common.txt	hadoop-common in branch-3.3 failed.
-1 ❌	mvnsite	0m 23s	/branch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in branch-3.3 failed.
-1 ❌	javadoc	0m 23s	/branch-javadoc-hadoop-common-project_hadoop-common.txt	hadoop-common in branch-3.3 failed.
-1 ❌	javadoc	0m 22s	/branch-javadoc-hadoop-tools_hadoop-aws.txt	hadoop-aws in branch-3.3 failed.
-1 ❌	spotbugs	0m 23s	/branch-spotbugs-hadoop-common-project_hadoop-common.txt	hadoop-common in branch-3.3 failed.
-1 ❌	spotbugs	0m 22s	/branch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in branch-3.3 failed.
+1 💚	shadedclient	3m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 20s		Maven dependency ordering for patch
-1 ❌	mvninstall	0m 23s	/patch-mvninstall-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	mvninstall	0m 22s	/patch-mvninstall-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	compile	0m 23s	/patch-compile-root.txt	root in the patch failed.
-1 ❌	javac	0m 23s	/patch-compile-root.txt	root in the patch failed.
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 20s	/buildtool-patch-checkstyle-root.txt	The patch fails to run checkstyle in root
-1 ❌	mvnsite	0m 22s	/patch-mvnsite-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	mvnsite	0m 22s	/patch-mvnsite-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	xml	0m 2s		The patch has no ill-formed XML file.
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	javadoc	0m 22s	/patch-javadoc-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
-1 ❌	spotbugs	0m 22s	/patch-spotbugs-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	spotbugs	0m 23s	/patch-spotbugs-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+1 💚	shadedclient	2m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 22s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
-1 ❌	unit	0m 22s	/patch-unit-hadoop-tools_hadoop-aws.txt	hadoop-aws in the patch failed.
+0 🆗	asflicense	0m 23s		ASF License check generated no output?
		32m 40s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/2/artifact/out/Dockerfile
GITHUB PR	#3292
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname	Linux f72610b2c132 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / 7778dced379d78da2e75229636e708afa38a518b
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/2/testReport/
Max. process+thread count	93 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/2/console
versions	git=2.17.1 maven=3.6.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

steveloughran · 2021-09-16T09:32:16Z

unable to create new native thread

looks like a container issue. Try rebase and resubmit

hadoop-yetus · 2021-09-16T13:20:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 32 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	12m 29s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 35s		branch-3.3 passed
+1 💚	compile	23m 7s		branch-3.3 passed
+1 💚	checkstyle	3m 15s		branch-3.3 passed
+1 💚	mvnsite	2m 50s		branch-3.3 passed
+1 💚	javadoc	2m 27s		branch-3.3 passed
+1 💚	spotbugs	4m 29s		branch-3.3 passed
+1 💚	shadedclient	42m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 33s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 50s		the patch passed
+1 💚	compile	21m 38s		the patch passed
-1 ❌	javac	21m 38s	/results-compile-javac-root.txt	root generated 1 new + 1945 unchanged - 0 fixed = 1946 total (was 1945)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	3m 13s	/results-checkstyle-root.txt	root: The patch generated 13 new + 157 unchanged - 39 fixed = 170 total (was 196)
+1 💚	mvnsite	2m 43s		the patch passed
+1 💚	xml	0m 1s		The patch has no ill-formed XML file.
+1 💚	javadoc	2m 30s		the patch passed
+1 💚	spotbugs	4m 15s		the patch passed
+1 💚	shadedclient	34m 31s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 30s		hadoop-common in the patch passed.
+1 💚	unit	2m 26s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 52s		The patch does not generate ASF License warnings.
		220m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/3/artifact/out/Dockerfile
GITHUB PR	#3292
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell xml markdownlint
uname	Linux c1bb628014b7 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / a677fa1159a1f3557cae1700e8717d69a252716c
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/3/testReport/
Max. process+thread count	3158 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/3/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mehakmeet · 2021-09-16T14:16:49Z

javac error seems unrelated to the patch and check styles is as discussed indentations.

mehakmeet · 2021-10-01T08:12:21Z

Going to make this as the backport for only the first commit, so that we could have a chain of commits rather than 1 commit with all the others squashed.

hadoop-yetus · 2021-10-01T11:04:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 45s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 21 new or modified test files.
			_ branch-3.3 Compile Tests _
+0 🆗	mvndep	3m 36s		Maven dependency ordering for branch
+1 💚	mvninstall	29m 4s		branch-3.3 passed
+1 💚	compile	17m 22s		branch-3.3 passed
+1 💚	checkstyle	2m 38s		branch-3.3 passed
+1 💚	mvnsite	2m 32s		branch-3.3 passed
+1 💚	javadoc	2m 33s		branch-3.3 passed
+1 💚	spotbugs	3m 48s		branch-3.3 passed
+1 💚	shadedclient	24m 21s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 35s		the patch passed
+1 💚	compile	18m 17s		the patch passed
-1 ❌	javac	18m 17s	/results-compile-javac-root.txt	root generated 2 new + 1945 unchanged - 1 fixed = 1947 total (was 1946)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 45s		root: The patch generated 0 new + 35 unchanged - 1 fixed = 35 total (was 36)
+1 💚	mvnsite	2m 23s		the patch passed
+1 💚	javadoc	2m 14s		the patch passed
+1 💚	spotbugs	4m 10s		the patch passed
+1 💚	shadedclient	24m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	17m 0s		hadoop-common in the patch passed.
+1 💚	unit	2m 29s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 54s		The patch does not generate ASF License warnings.
		170m 57s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/4/artifact/out/Dockerfile
GITHUB PR	#3292
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint
uname	Linux f8494a32cb6b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.3 / `e421824`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~18.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/4/testReport/
Max. process+thread count	3158 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3292/4/console
versions	git=2.17.1 maven=3.6.0 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mehakmeet · 2021-10-05T11:11:21Z

Merged in branch-3.3

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

mehakmeet force-pushed the HADOOP-13887-3.3 branch from 7778dce to a677fa1 Compare September 16, 2021 09:38

mehakmeet force-pushed the HADOOP-13887-3.3 branch from a677fa1 to e421824 Compare October 1, 2021 08:12

mehakmeet mentioned this pull request Oct 1, 2021

HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration (#3466) #3508

Closed

mehakmeet changed the title ~~HADOOP-13887, HADOOP-17817, HADOOP-17823. Support S3 client side encryption (S3-CSE) using AWS-SDK~~ HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK Oct 1, 2021

mehakmeet closed this Oct 5, 2021

balvisio reviewed Dec 16, 2021

View reviewed changes

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java Show resolved Hide resolved

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK #3292

HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK #3292

Uh oh!

mehakmeet commented Aug 11, 2021

Uh oh!

mukund-thakur commented Aug 16, 2021

Uh oh!

mehakmeet commented Aug 20, 2021

Uh oh!

mehakmeet commented Sep 16, 2021

Uh oh!

hadoop-yetus commented Sep 16, 2021

Uh oh!

steveloughran commented Sep 16, 2021

Uh oh!

hadoop-yetus commented Sep 16, 2021

Uh oh!

mehakmeet commented Sep 16, 2021

Uh oh!

mehakmeet commented Oct 1, 2021

Uh oh!

hadoop-yetus commented Oct 1, 2021

Uh oh!

mehakmeet commented Oct 5, 2021

Uh oh!

Uh oh!

Uh oh!

HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK #3292

HADOOP-13887. Support S3 client side encryption (S3-CSE) using AWS-SDK #3292

Uh oh!

Conversation

mehakmeet commented Aug 11, 2021

CSE ON, S3Guard ON

CSE ON, S3Guard OFF

CSE OFF, S3Guard ON

CSE OFF, S3Guard OFF

Uh oh!

mukund-thakur commented Aug 16, 2021

Uh oh!

mehakmeet commented Aug 20, 2021

Uh oh!

mehakmeet commented Sep 16, 2021

Uh oh!

hadoop-yetus commented Sep 16, 2021

Uh oh!

steveloughran commented Sep 16, 2021

Uh oh!

hadoop-yetus commented Sep 16, 2021

Uh oh!

mehakmeet commented Sep 16, 2021

Uh oh!

mehakmeet commented Oct 1, 2021

Uh oh!

hadoop-yetus commented Oct 1, 2021

Uh oh!

mehakmeet commented Oct 5, 2021

Uh oh!

Uh oh!

Uh oh!