HDDS-1043. Enable token based authentication for S3 api. #561

ajayydv · 2019-03-06T02:52:43Z

No description provided.

hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot

hadoop-yetus · 2019-03-06T04:59:31Z

hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot

+    ${hostname}=        Execute                    hostname
+    Execute             kinit -k testuser/${hostname}@EXAMPLE.COM -t /etc/security/keytabs/testuser.keytab
+    ${result} =         Execute                    ozone sh s3 getsecret
+    ${accessKey} =      Get Regexp Matches         ${result}     (?<=awsAccessKey=).*


whitespace:tabs in line

hadoop-yetus · 2019-03-06T04:59:32Z

hadoop-ozone/dist/src/main/smoketest/test.sh

@@ -45,7 +45,7 @@ wait_for_datanodes(){

         #Print it only if a number. Could be not a number if scm is not yet started
         if [[ "$datanodes" ]]; then
-            echo "$datanodes datanode is up and healhty (until now)"
+            echo "$datanodes datanode is up and healthy (until now)"


whitespace:tabs in line

hadoop-ozone/integration-test/hs_err_pid22439.log

hadoop-yetus · 2019-03-06T18:43:11Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	7	#561 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#561
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/4/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-03-06T18:49:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	8	#561 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#561
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/5/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/AWSV4AuthValidator.java

ajayydv · 2019-03-06T18:55:35Z

a few unit tests are failing (NPE in s3 token token related tests)

could you please share the failing tests. I can't find them in test report.

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/AWSV4AuthValidator.java

bharatviswa504 · 2019-03-06T18:55:52Z

hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/AWSV4AuthValidator.java

+   */
+  public static boolean validateRequest(String strToSign, String signature,
+      String userKey) {
+    String expectedSignature = Hex.encode(sign(getSignatureKey(


Can we move this line in to a method say getSignature()

As from the doc, this is signature.

Added java doc for, let me know if you feel strongly about having it in separate function as it is one liner code.

bharatviswa504 · 2019-03-06T19:08:42Z

...common/src/main/java/org/apache/hadoop/ozone/security/OzoneDelegationTokenSecretManager.java

+    } catch (IOException e) {
+      LOG.error("Error while validating S3 identifier:{}",
+          identifier, e);
+      throw new InvalidToken("No S3 secret found for S3 identifier:"


Now if InvalidToken is thrown as an exception during invalid/malformed header, then how this will be thrown to the end user s3 request? I don't see any code for it.

Now if token validation fails rpc connection will fail itself. S3 gateway will get an error. Error propagation to client will depend on S3g error handling.

hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneServiceProvider.java

hadoop-yetus · 2019-03-06T22:38:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	7	#561 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#561
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/6/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

ajayydv · 2019-03-06T22:44:49Z

forced reset to squashed commit as local rebase added spurious commits.

hadoop-yetus · 2019-03-07T00:43:51Z

hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot

+
+Secure S3 test Failure
+    Run Keyword         Install aws cli
+    ${rc}  ${result} =  Run And Return Rc And Output  aws s3api --endpoint-url ${ENDPOINT_URL} create-bucket --bucket bucket-test123


whitespace:tabs in line

hadoop-yetus · 2019-03-07T00:43:52Z

hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot

+    ${hostname}=        Execute                    hostname
+    Execute             kinit -k testuser/${hostname}@EXAMPLE.COM -t /etc/security/keytabs/testuser.keytab
+    ${result} =         Execute                    ozone sh s3 getsecret
+    ${accessKey} =      Get Regexp Matches         ${result}     (?<=awsAccessKey=).*


whitespace:tabs in line

hadoop-yetus · 2019-03-07T01:59:49Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	29	Docker mode activated.
		_ Prechecks _
0	yamllint	0	yamllint was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 5 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	27	Maven dependency ordering for branch
+1	mvninstall	1016	trunk passed
+1	compile	968	trunk passed
+1	checkstyle	199	trunk passed
-1	mvnsite	41	integration-test in trunk failed.
-1	mvnsite	37	ozone-manager in trunk failed.
+1	shadedclient	655	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/dist hadoop-ozone/integration-test
-1	findbugs	34	ozone-manager in trunk failed.
+1	javadoc	217	trunk passed
		_ Patch Compile Tests _
0	mvndep	24	Maven dependency ordering for patch
-1	mvninstall	19	dist in the patch failed.
-1	mvninstall	27	integration-test in the patch failed.
+1	compile	967	the patch passed
+1	cc	967	the patch passed
+1	javac	967	the patch passed
+1	checkstyle	237	the patch passed
-1	mvnsite	41	integration-test in the patch failed.
+1	shellcheck	1	There were no new shellcheck issues.
+1	shelldocs	30	There were no new shelldocs issues.
-1	whitespace	4	The patch has 75 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-1	whitespace	5	The patch 19851 line(s) with tabs.
+1	shadedclient	796	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/dist hadoop-ozone/integration-test
+1	findbugs	273	the patch passed
+1	javadoc	201	the patch passed
		_ Other Tests _
+1	unit	85	common in the patch passed.
+1	unit	45	common in the patch passed.
+1	unit	31	dist in the patch passed.
-1	unit	37	integration-test in the patch failed.
+1	unit	47	ozone-manager in the patch passed.
+1	unit	46	s3gateway in the patch passed.
+1	asflicense	43	The patch does not generate ASF License warnings.
		7266

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/Dockerfile
GITHUB PR	#561
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc yamllint shellcheck shelldocs
uname	Linux 28089b4aa12f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `45f976f`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/branch-mvnsite-hadoop-ozone_integration-test.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/branch-mvnsite-hadoop-ozone_ozone-manager.txt
shellcheck	v0.4.6
findbugs	v3.1.0-RC1
findbugs	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/patch-mvnsite-hadoop-ozone_integration-test.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/whitespace-eol.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/whitespace-tabs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/testReport/
Max. process+thread count	445 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/s3gateway U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/7/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-03-07T10:31:56Z

a few unit tests are failing (NPE in s3 token token related tests)

could you please share the failing tests. I can't find them in test report.

Yetus is still not running all the tests. check the small red X next to your commit id:

https://ci.anzix.net/job/ozone/427/testReport/


Test Name | Duration | Age
-- | -- | --
org.apache.hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient.testMultiBlockWrites | 1 min 44 sec | 1
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testContainerExclusionWithClosedContainerException | 2 min 7 sec | 1
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testMultiBlockWritesWithIntermittentDnFailures | 1 min 19 sec | 1
org.apache.hadoop.ozone.client.rpc.TestOzoneAtRestEncryption.testGetS3Secret | 58 ms | 1
org.apache.hadoop.ozone.client.rpc.TestOzoneRpcClient.testGetS3Secret | 49 ms | 1
org.apache.hadoop.ozone.client.rpc.TestSecureOzoneRpcClient.testGetS3Secret | 48 ms | 1
org.apache.hadoop.ozone.ozShell.TestOzoneShell.testS3Secret[0] | 2.5 sec | 1
org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient.testDatanodeExclusionWithMajorityCommit | 29 sec | 2
org.apache.hadoop.ozone.container.server.TestSecureContainerServer.testClientServerRatisGrpc | 1.8 sec | 2

java.lang.NullPointerException
 at org.apache.hadoop.ozone.om.OzoneManager.getS3Secret(OzoneManager.java:2387)
 at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getS3Secret(OzoneManagerRequestHandler.java:877)
 at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handle(OzoneManagerRequestHandler.java:308)
 at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:92)
 at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:73)
 at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
 at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

hadoop-yetus · 2019-03-08T02:09:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	7	#561 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#561
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/8/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-03-08T22:59:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	8	#561 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#561
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/9/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-03-09T01:09:49Z

hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot

+
+Secure S3 test Failure
+    Run Keyword         Install aws cli
+    ${rc}  ${result} =  Run And Return Rc And Output  aws s3api --endpoint-url ${ENDPOINT_URL} create-bucket --bucket bucket-test123


whitespace:tabs in line

hadoop-yetus · 2019-03-09T01:09:50Z

hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot

+    ${hostname}=        Execute                    hostname
+    Execute             kinit -k testuser/${hostname}@EXAMPLE.COM -t /etc/security/keytabs/testuser.keytab
+    ${result} =         Execute                    ozone sh s3 getsecret
+    ${accessKey} =      Get Regexp Matches         ${result}     (?<=awsAccessKey=).*


whitespace:tabs in line

hadoop-yetus · 2019-03-09T01:09:53Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	24	Docker mode activated.
		_ Prechecks _
0	yamllint	0	yamllint was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 8 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	29	Maven dependency ordering for branch
+1	mvninstall	1037	trunk passed
+1	compile	970	trunk passed
+1	checkstyle	197	trunk passed
-1	mvnsite	52	integration-test in trunk failed.
+1	shadedclient	721	branch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/dist hadoop-ozone/integration-test
+1	findbugs	280	trunk passed
+1	javadoc	279	trunk passed
		_ Patch Compile Tests _
0	mvndep	65	Maven dependency ordering for patch
-1	mvninstall	18	dist in the patch failed.
-1	mvninstall	23	integration-test in the patch failed.
+1	compile	904	the patch passed
+1	cc	904	the patch passed
+1	javac	904	the patch passed
+1	checkstyle	183	the patch passed
-1	mvnsite	40	integration-test in the patch failed.
+1	shellcheck	0	There were no new shellcheck issues.
+1	shelldocs	36	There were no new shelldocs issues.
-1	whitespace	0	The patch 2 line(s) with tabs.
+1	shadedclient	610	patch has no errors when building and testing our client artifacts.
0	findbugs	0	Skipped patched modules with no Java source: hadoop-ozone/dist hadoop-ozone/integration-test
+1	findbugs	285	the patch passed
+1	javadoc	226	the patch passed
		_ Other Tests _
+1	unit	86	common in the patch passed.
+1	unit	50	common in the patch passed.
+1	unit	34	dist in the patch passed.
-1	unit	39	integration-test in the patch failed.
+1	unit	47	ozone-manager in the patch passed.
+1	unit	47	s3gateway in the patch passed.
+1	asflicense	42	The patch does not generate ASF License warnings.
		7276

Subsystem	Report/Notes
Docker	Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/Dockerfile
GITHUB PR	#561
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc yamllint shellcheck shelldocs
uname	Linux e0e17f31813d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `c072458`
maven	version: Apache Maven 3.3.9
Default Java	1.8.0_191
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/branch-mvnsite-hadoop-ozone_integration-test.txt
shellcheck	v0.4.6
findbugs	v3.1.0-RC1
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
mvninstall	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/patch-mvninstall-hadoop-ozone_integration-test.txt
mvnsite	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/patch-mvnsite-hadoop-ozone_integration-test.txt
whitespace	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/whitespace-tabs.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/testReport/
Max. process+thread count	412 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/common hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/s3gateway U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/10/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-03-12T16:42:44Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	0	Docker mode activated.
-1	patch	7	#561 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.

Subsystem	Report/Notes
GITHUB PR	#561
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-561/11/console
Powered by	Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

elek

Thank you @ajayydv the patch and the continuous improvement. This is a real milestone for the S3 gateway.

I am doing a final local build and test (including checkstyle/findbugs/acceptance) and will commit it soon.

Closes #561 (cherry picked from commit dcb0de8)

…ob and cause unit tests to get skipped Author: Cameron Lee <calee@linkedin.com> Reviewers: Xinyu Liu <xinyu@apache.org> Closes apache#561 from cameronlee314/app_runner_main_exit

ajayydv requested review from bharatviswa504 and elek and removed request for bharatviswa504 March 6, 2019 02:52

ajayydv added the ozone label Mar 6, 2019