HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration #3466

steveloughran · 2021-09-21T15:02:51Z

New unit test suite for propagation

adds TestBucketConfiguration for conf file propagation (succeeds)
one for jecks (fails; needs to sync up with the other patch)
moves over some test cases from ITestS3AConfiguration which
don't need an FS.

Change-Id: Ie1b6e0d1c655d00fc6d47ce054a1aeba01c71044

Description of PR

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?

new test and modified original yes: s3 london

steveloughran · 2021-09-28T12:59:12Z

checkstyle is unused import; fixed

hadoop-yetus · 2021-09-28T14:34:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 10 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 15s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 26s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 30s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 9s		trunk passed
+1 💚	shadedclient	21m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 14s		the patch passed
+1 💚	shadedclient	23m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 35s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		94m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/3/artifact/out/Dockerfile
GITHUB PR	#3466
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint
uname	Linux 89ea54edd52b 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 87e85baf79d53376e03d72b8a5f6a565db75873d
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/3/testReport/
Max. process+thread count	615 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mukund-thakur

Looks good overall. Just added a few clarification comments. And nice tests.

BTW, just to be sure once again. The key resolution follows the following order, right?

bucket level new key whether it is present in jceks or xml.
bucket level old key whether it is present in jceks or xml.
global new key whether it is present in jceks or xml.
global old key whether it is present in jceks or xml.

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AEncryptionMethods.java

hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java

hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md

hadoop-yetus · 2021-09-29T10:51:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 0s		Docker mode activated.
-1 ❌	patch	0m 21s		#3466 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.

Subsystem	Report/Notes
GITHUB PR	#3466
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/4/console
versions	git=2.17.1
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

New unit test suite for propagation * adds TestBucketConfiguration for conf file propagation (succeeds) * one for jecks (fails; needs to sync up with the other patch) * moves over some test cases from ITestS3AConfiguration which don't need an FS. Change-Id: Ie1b6e0d1c655d00fc6d47ce054a1aeba01c71044

Explicit ordering of resolution of current & deprecated encryption config keys when reading from configuration files, which ensures that semantics for jcecks-stored values match those of XML options. Test expanded for password resolution Docs updated Change-Id: I3bf9697ff07d8cb830dca5a1d2c224a79f89fe90

Change-Id: I4a1465b65c2b1c532e2ba5d2497c88b614247ccd

Change-Id: I840344205abdc70cadbe4635a0bec9a80bccd918

steveloughran · 2021-09-29T12:36:42Z

had to rebase

hadoop-yetus · 2021-09-29T14:10:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 11s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 10 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 29s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 27s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	javadoc	0m 22s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 31s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 10s		trunk passed
+1 💚	shadedclient	22m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 15s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 23s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 11s		the patch passed
+1 💚	shadedclient	22m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 31s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 31s		The patch does not generate ASF License warnings.
		92m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/5/artifact/out/Dockerfile
GITHUB PR	#3466
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint
uname	Linux 61d0a60a4726 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3e71af7`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/5/testReport/
Max. process+thread count	530 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3466/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

mukund-thakur

+1 All good.

mehakmeet · 2021-10-01T07:53:37Z

@steveloughran, Seems like one test method ITestS3AFileContextStatistics#verifyWrittenBytes() still uses the old conf.get(S3_ENCRYPTION_ALGORITHM, ""), so that test would fail if the test setup has per-bucket property set. I'll open a Jira for this test.

…apache#3466) The ordering of the resolution of new and deprecated s3a encryption options & secrets is the same when JCEKS and other hadoop credentials stores are used to store them as when they are in XML files: per-bucket settings always take priority over global values, even when the bucket-level options use the old option names. Contributed by Mehakmeet Singh and Steve Loughran

…#3466) The ordering of the resolution of new and deprecated s3a encryption options & secrets is the same when JCEKS and other hadoop credentials stores are used to store them as when they are in XML files: per-bucket settings always take priority over global values, even when the bucket-level options use the old option names. Contributed by Mehakmeet Singh and Steve Loughran Change-Id: I871672071efa2eb6b600cb2658fceeef57f658a3

steveloughran mentioned this pull request Sep 21, 2021

HADOOP-17922. Lookup old S3 encryption configs for JCEKS #3462

Closed

steveloughran force-pushed the s3/HADOOP-17922-per-bucket-deprecated-options branch from 693f9eb to 7793694 Compare September 27, 2021 17:36

mukund-thakur self-requested a review September 28, 2021 10:14

apache deleted a comment from hadoop-yetus Sep 28, 2021

mukund-thakur reviewed Sep 29, 2021

View reviewed changes

steveloughran changed the title ~~HADOOP-17922. move to fs.s3a.encryption.algorithm~~ HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration Sep 29, 2021

mehakmeet and others added 7 commits September 29, 2021 13:36

HADOOP-17922. Lookup old S3 encryption configs for JCEKS

5b2994e

HADOOP-17922. test fixes

d1893f5

HADOOP-17922. review comments

926012a

HADOOP-17922. checkstyle

355e1bf

Change-Id: I4a1465b65c2b1c532e2ba5d2497c88b614247ccd

HADOOP-17922. docs and javadocs tuning

3e71af7

Change-Id: I840344205abdc70cadbe4635a0bec9a80bccd918

steveloughran force-pushed the s3/HADOOP-17922-per-bucket-deprecated-options branch from 7ce5895 to 3e71af7 Compare September 29, 2021 12:36

steveloughran mentioned this pull request Sep 29, 2021

HADOOP-17851. Support user specified content encoding for S3A #3498

Merged

3 tasks

mukund-thakur approved these changes Sep 30, 2021

View reviewed changes

steveloughran merged commit d609f44 into apache:trunk Sep 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration #3466

HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration #3466

steveloughran commented Sep 21, 2021 •

edited

Loading

steveloughran commented Sep 28, 2021

hadoop-yetus commented Sep 28, 2021

mukund-thakur left a comment

hadoop-yetus commented Sep 29, 2021

steveloughran commented Sep 29, 2021

hadoop-yetus commented Sep 29, 2021

mukund-thakur left a comment

mehakmeet commented Oct 1, 2021

HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration #3466

HADOOP-17922. move to fs.s3a.encryption.algorithm - JCEKS integration #3466

Conversation

steveloughran commented Sep 21, 2021 • edited Loading

Description of PR

For code changes:

steveloughran commented Sep 28, 2021

hadoop-yetus commented Sep 28, 2021

mukund-thakur left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Sep 29, 2021

steveloughran commented Sep 29, 2021

hadoop-yetus commented Sep 29, 2021

mukund-thakur left a comment

Choose a reason for hiding this comment

mehakmeet commented Oct 1, 2021

steveloughran commented Sep 21, 2021 •

edited

Loading