HDDS-738. Removing REST protocol support from OzoneClient #1329

elek · 2019-08-21T14:12:09Z

Since we have functional {{S3Gateway}} for Ozone which works on REST protocol, having REST protocol support in OzoneClient feels redundant and it will take a lot of effort to maintain it up to date.
As S3Gateway is in a functional state now, I propose to remove REST protocol support from OzoneClient.

Once we remove REST support from OzoneClient, the following will be the interface to access Ozone cluster

OzoneClient (RPC Protocol)
OzoneFS (RPC Protocol)
S3Gateway (REST Protocol)

See: https://issues.apache.org/jira/browse/HDDS-738

hadoop-yetus · 2019-08-21T17:53:37Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	43	Docker mode activated.
		_ Prechecks _
+1	dupname	5	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 44 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	26	Maven dependency ordering for branch
+1	mvninstall	574	trunk passed
+1	compile	374	trunk passed
+1	checkstyle	65	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	721	branch has no errors when building and testing our client artifacts.
+1	javadoc	151	trunk passed
0	spotbugs	422	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	616	trunk passed
		_ Patch Compile Tests _
0	mvndep	33	Maven dependency ordering for patch
+1	mvninstall	533	the patch passed
+1	compile	368	the patch passed
+1	javac	95	hadoop-hdds in the patch passed.
+1	javac	273	hadoop-ozone generated 0 new + 5 unchanged - 3 fixed = 5 total (was 8)
-0	checkstyle	39	hadoop-ozone: The patch generated 15 new + 0 unchanged - 0 fixed = 15 total (was 0)
+1	mvnsite	0	the patch passed
+1	shellcheck	0	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	7	The patch has no ill-formed XML file.
+1	shadedclient	627	patch has no errors when building and testing our client artifacts.
-1	javadoc	85	hadoop-ozone generated 1 new + 25 unchanged - 1 fixed = 26 total (was 26)
+1	findbugs	635	the patch passed
		_ Other Tests _
+1	unit	293	hadoop-hdds in the patch passed.
-1	unit	356	hadoop-ozone in the patch failed.
+1	asflicense	45	The patch does not generate ASF License warnings.
		5951

Reason	Tests
Failed junit tests	hadoop.ozone.s3.TestOzoneClientProducer

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/1/artifact/out/Dockerfile
GITHUB PR	#1329
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs
uname	Linux e0f3a4b70b0d 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `57f7370`
Default Java	1.8.0_222
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/1/artifact/out/diff-checkstyle-hadoop-ozone.txt
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/1/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/1/testReport/
Max. process+thread count	1306 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone hadoop-ozone/client hadoop-ozone/common hadoop-ozone/datanode hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozonefs hadoop-ozone/tools U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/1/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

anuengineer · 2019-08-21T23:10:27Z

There are some minor issues, otherwise, it looks quite good.

Unit test failure looks connected to this patch.

 [ERROR] Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 2.141 s <<< FAILURE! - in org.apache.hadoop.ozone.s3.TestOzoneClientProducer
[ERROR] testGetClientFailure[0](org.apache.hadoop.ozone.s3.TestOzoneClientProducer)  Time elapsed: 0.58 s  <<< FAILURE!
java.lang.AssertionError: 
Expected to find 'Couldn't create protocol ' but got unexpected exception: java.io.IOException: Couldn't create RpcClient protocol
  at org.apache.hadoop.ozone.client.OzoneClientFactory.getClientProtocol(OzoneClientFactory.java:207)

The Checkstyle issues -- mostly unused imports.
One minor JavaDoc.

hadoop-yetus · 2019-08-23T16:56:45Z

hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestOzoneClientProducer.java

 import static org.apache.hadoop.ozone.s3.AWSAuthParser.AUTHORIZATION_HEADER;
 import static org.apache.hadoop.ozone.s3.AWSAuthParser.CONTENT_MD5;
 import static org.apache.hadoop.ozone.s3.AWSAuthParser.CONTENT_TYPE;
 import static org.apache.hadoop.ozone.s3.AWSAuthParser.HOST_HEADER;
 import static org.apache.hadoop.ozone.s3.AWSAuthParser.X_AMAZ_DATE;
 import static org.apache.hadoop.ozone.s3.AWSAuthParser.X_AMZ_CONTENT_SHA256;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;


whitespace:end of line

elek · 2019-08-27T07:56:44Z

/retest

hadoop-yetus · 2019-08-27T20:44:01Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	40	Docker mode activated.
		_ Prechecks _
+1	dupname	5	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 48 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	35	Maven dependency ordering for branch
+1	mvninstall	614	trunk passed
+1	compile	397	trunk passed
+1	checkstyle	92	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	785	branch has no errors when building and testing our client artifacts.
+1	javadoc	187	trunk passed
0	spotbugs	427	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	626	trunk passed
		_ Patch Compile Tests _
0	mvndep	50	Maven dependency ordering for patch
+1	mvninstall	539	the patch passed
+1	compile	399	the patch passed
+1	javac	113	hadoop-hdds in the patch passed.
+1	javac	286	hadoop-ozone generated 0 new + 5 unchanged - 3 fixed = 5 total (was 8)
+1	checkstyle	101	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	1	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	7	The patch has no ill-formed XML file.
+1	shadedclient	694	patch has no errors when building and testing our client artifacts.
-1	javadoc	102	hadoop-ozone generated 1 new + 26 unchanged - 1 fixed = 27 total (was 27)
+1	findbugs	622	the patch passed
		_ Other Tests _
-1	unit	315	hadoop-hdds in the patch failed.
-1	unit	1836	hadoop-ozone in the patch failed.
+1	asflicense	64	The patch does not generate ASF License warnings.
		7918

Reason	Tests
Failed junit tests	hadoop.hdds.scm.block.TestBlockManager
	hadoop.ozone.client.rpc.TestContainerStateMachineFailures
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/8/artifact/out/Dockerfile
GITHUB PR	#1329
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs
uname	Linux 0e9b74bf9e01 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `66cfa48`
Default Java	1.8.0_222
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/8/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/8/artifact/out/patch-unit-hadoop-hdds.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/8/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/8/testReport/
Max. process+thread count	4787 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone hadoop-ozone/client hadoop-ozone/common hadoop-ozone/datanode hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozonefs hadoop-ozone/s3gateway hadoop-ozone/tools U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/8/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

elek · 2019-08-27T21:25:01Z

@nandakumar131 @anuengineer Finally it's ready to commit. All the tests are fixed.

There were two failures:

TestSecureContainerServer: Fixed by HDDS-2040
TestOzoneClientRetriesOnException: Passed locally:

[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.818 s - in org.apache.hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

The biggest change in this patch is the following:

Until now the ozone sh command converted the bucket/volume objects to the rest representation to print out as json. As the rest classes are removed I changed it to print out json directly the original java classes (in json format). Therefore same of the field names will be changed by this patch.

hadoop-yetus · 2019-08-28T02:52:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	77	Docker mode activated.
		_ Prechecks _
+1	dupname	4	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 48 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	70	Maven dependency ordering for branch
+1	mvninstall	633	trunk passed
+1	compile	435	trunk passed
+1	checkstyle	82	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	885	branch has no errors when building and testing our client artifacts.
+1	javadoc	191	trunk passed
0	spotbugs	476	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	690	trunk passed
		_ Patch Compile Tests _
0	mvndep	40	Maven dependency ordering for patch
+1	mvninstall	589	the patch passed
+1	compile	394	the patch passed
+1	javac	106	hadoop-hdds in the patch passed.
+1	javac	288	hadoop-ozone generated 0 new + 5 unchanged - 3 fixed = 5 total (was 8)
+1	checkstyle	85	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	0	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	xml	7	The patch has no ill-formed XML file.
+1	shadedclient	736	patch has no errors when building and testing our client artifacts.
-1	javadoc	94	hadoop-ozone generated 1 new + 27 unchanged - 1 fixed = 28 total (was 28)
+1	findbugs	670	the patch passed
		_ Other Tests _
+1	unit	364	hadoop-hdds in the patch passed.
-1	unit	2199	hadoop-ozone in the patch failed.
+1	asflicense	49	The patch does not generate ASF License warnings.
		8652

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/9/artifact/out/Dockerfile
GITHUB PR	#1329
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml shellcheck shelldocs
uname	Linux fa5a2924f78d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `b1eee8b`
Default Java	1.8.0_222
javadoc	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/9/artifact/out/diff-javadoc-javadoc-hadoop-ozone.txt
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/9/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/9/testReport/
Max. process+thread count	4859 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone hadoop-ozone/client hadoop-ozone/common hadoop-ozone/datanode hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozonefs hadoop-ozone/s3gateway hadoop-ozone/tools U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1329/9/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

anuengineer · 2019-08-28T16:46:24Z

I am going to commit this now. There some other changes we need to do in the future.

Fix documentation -- if you file a JIRA, feel free to assign it to me. It might need us to fix some images too.
There is a server side code, it might be useful for us to file a follow up JIRA to remove it.

This patch is huge, so I expect various systems to fail, but this is the start of the 0.5.0 cycle, so we are fine and we can stabilize issues after this is committed if needed.

anuengineer · 2019-08-28T16:58:27Z

Committed to the trunk. @elek Thank you for the contribution. @nandakumar131 Thanks for the review and discussion.

elek added the ozone label Aug 21, 2019

elek force-pushed the HDDS-738 branch from c59b0c2 to 53c0da4 Compare August 22, 2019 14:39

elek added 2 commits August 23, 2019 12:38

HDDS-738. Removing REST protocol support from OzoneClient

0a94d00

checkstyle + unit test fixes

a31efcb

elek force-pushed the HDDS-738 branch from 53c0da4 to 3adcd41 Compare August 23, 2019 14:47

unit fixes

3adcd41

hadoop-yetus reviewed Aug 23, 2019

View reviewed changes

elek added 7 commits August 23, 2019 19:15

checkstyle + rat fixes

185cca6

fix tests

7363e9f

fix unit tests

fbcf8c7

fix unit tests

fbfe134

unit test fixes

7c47487

fix acceptance tests

59ee275

fix testomacls

831a635

fix back the unit test

484b531

apache deleted a comment from hadoop-yetus Aug 27, 2019

fix acceptance tests

14f85e6

anuengineer closed this Aug 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-738. Removing REST protocol support from OzoneClient #1329

HDDS-738. Removing REST protocol support from OzoneClient #1329

elek commented Aug 21, 2019

hadoop-yetus commented Aug 21, 2019

anuengineer commented Aug 21, 2019

hadoop-yetus Aug 23, 2019

elek commented Aug 27, 2019

hadoop-yetus commented Aug 27, 2019

elek commented Aug 27, 2019

hadoop-yetus commented Aug 28, 2019

anuengineer commented Aug 28, 2019

anuengineer commented Aug 28, 2019

HDDS-738. Removing REST protocol support from OzoneClient #1329

HDDS-738. Removing REST protocol support from OzoneClient #1329

Conversation

elek commented Aug 21, 2019

hadoop-yetus commented Aug 21, 2019

anuengineer commented Aug 21, 2019

hadoop-yetus Aug 23, 2019

Choose a reason for hiding this comment

elek commented Aug 27, 2019

hadoop-yetus commented Aug 27, 2019

elek commented Aug 27, 2019

hadoop-yetus commented Aug 28, 2019

anuengineer commented Aug 28, 2019

anuengineer commented Aug 28, 2019