HDDS-1909. Use new HA code for Non-HA in OM. #1225

bharatviswa504 · 2019-08-04T22:30:46Z

No description provided.

bharatviswa504 · 2019-08-06T00:11:17Z

/retest

bharatviswa504 · 2019-08-06T06:05:14Z

/retest

bharatviswa504 · 2019-08-23T23:05:39Z

/retest

...r/src/main/java/org/apache/hadoop/ozone/om/request/security/OMGetDelegationTokenRequest.java

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

arp7 · 2019-08-30T17:42:16Z

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

+          omClientResponse.getFlushFuture().get();
+          LOG.trace("Future for {} is completed", request);
+        } catch (ExecutionException | InterruptedException ex) {
+          // Do we need to terminate OM here?


The only potential issue is if we update the cache but fail to update RocksDB. In that case self-terminate is likely the best option. Since Ratis is not enabled the request is not persistent.

So, terminate OM here?

Yes I would err on the side of safety and self-terminate the OM here.

...main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java

hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestS3BucketManager.java

arp7

Few more comments.

hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestS3BucketManager.java

...r/src/test/java/org/apache/hadoop/ozone/om/request/volume/acl/TestOMVolumeAddAclRequest.java

...r/src/test/java/org/apache/hadoop/ozone/om/request/volume/acl/TestOMVolumeSetAclRequest.java

bharatviswa504 · 2019-08-30T19:17:57Z

Thank You @arp7 for the review.
I have addressed the review comments.

bharatviswa504 · 2019-08-30T19:18:18Z

/retest

bharatviswa504 · 2019-08-30T22:04:22Z

Fixed s3 test failure.

bharatviswa504 · 2019-08-30T22:04:26Z

/retest

hadoop-yetus · 2019-08-30T23:50:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	40	Docker mode activated.
		_ Prechecks _
+1	dupname	2	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 25 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	76	Maven dependency ordering for branch
+1	mvninstall	589	trunk passed
+1	compile	399	trunk passed
+1	checkstyle	95	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	790	branch has no errors when building and testing our client artifacts.
+1	javadoc	194	trunk passed
0	spotbugs	425	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	636	trunk passed
		_ Patch Compile Tests _
0	mvndep	52	Maven dependency ordering for patch
+1	mvninstall	550	the patch passed
+1	compile	413	the patch passed
+1	javac	413	the patch passed
+1	checkstyle	105	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	1	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	693	patch has no errors when building and testing our client artifacts.
+1	javadoc	191	the patch passed
+1	findbugs	642	the patch passed
		_ Other Tests _
+1	unit	299	hadoop-hdds in the patch passed.
-1	unit	189	hadoop-ozone in the patch failed.
+1	asflicense	55	The patch does not generate ASF License warnings.
		6325

Reason	Tests
Failed junit tests	hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/25/artifact/out/Dockerfile
GITHUB PR	#1225
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle shellcheck shelldocs
uname	Linux b61b325f9a52 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `7085512`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/25/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/25/testReport/
Max. process+thread count	1287 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/25/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2019-09-03T02:41:05Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
0	reexec	34	Docker mode activated.
		_ Prechecks _
+1	dupname	2	No case conflicting files found.
0	shelldocs	0	Shelldocs was not available.
+1	@author	0	The patch does not contain any @author tags.
+1	test4tests	0	The patch appears to include 25 new or modified test files.
		_ trunk Compile Tests _
0	mvndep	67	Maven dependency ordering for branch
+1	mvninstall	595	trunk passed
+1	compile	405	trunk passed
+1	checkstyle	81	trunk passed
+1	mvnsite	0	trunk passed
+1	shadedclient	849	branch has no errors when building and testing our client artifacts.
+1	javadoc	177	trunk passed
0	spotbugs	440	Used deprecated FindBugs config; considering switching to SpotBugs.
+1	findbugs	643	trunk passed
		_ Patch Compile Tests _
0	mvndep	41	Maven dependency ordering for patch
+1	mvninstall	549	the patch passed
+1	compile	386	the patch passed
+1	javac	386	the patch passed
+1	checkstyle	87	the patch passed
+1	mvnsite	0	the patch passed
+1	shellcheck	1	There were no new shellcheck issues.
+1	whitespace	0	The patch has no whitespace issues.
+1	shadedclient	740	patch has no errors when building and testing our client artifacts.
+1	javadoc	204	the patch passed
+1	findbugs	726	the patch passed
		_ Other Tests _
+1	unit	298	hadoop-hdds in the patch passed.
-1	unit	1792	hadoop-ozone in the patch failed.
+1	asflicense	47	The patch does not generate ASF License warnings.
		7992

Reason	Tests
Failed junit tests	hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
	hadoop.ozone.TestSecureOzoneCluster
	hadoop.ozone.om.TestSecureOzoneManager
	hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures

Subsystem	Report/Notes
Docker	Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/26/artifact/out/Dockerfile
GITHUB PR	#1225
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle shellcheck shelldocs
uname	Linux 78dae72e3ceb 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `915cbc9`
Default Java	1.8.0_222
unit	https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/26/artifact/out/patch-unit-hadoop-ozone.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/26/testReport/
Max. process+thread count	5295 (vs. ulimit of 5500)
modules	C: hadoop-hdds/common hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager hadoop-ozone/ozone-recon U: .
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1225/26/console
versions	git=2.7.4 maven=3.3.9 shellcheck=0.4.6 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

arp7 · 2019-09-03T20:05:21Z

Thanks @bharatviswa504. The patch looks pretty good to me.

+1

One remaining point was to self-terminate the OM in the RocksDB update failure path. It can be done in a separate jira.

arp7 · 2019-09-03T20:07:21Z

+1 to commit assuming the integration test failures are unrelated.

bharatviswa504 · 2019-09-03T20:24:50Z

Thank You @arp7 for the review.
I have committed this to the trunk.

bharatviswa504 · 2019-09-03T20:28:21Z

Opened jira's HDDS-2078 and HDDS-2079 for secure cluster test failures.

bharatviswa504 added the ozone label Aug 4, 2019

bharatviswa504 force-pushed the HDDS-1909 branch 4 times, most recently from 76af6ab to 6169be1 Compare August 4, 2019 22:43

bharatviswa504 self-assigned this Aug 4, 2019

bharatviswa504 force-pushed the HDDS-1909 branch 2 times, most recently from c18a1e0 to 5ebb33f Compare August 6, 2019 00:09

apache deleted a comment from hadoop-yetus Aug 6, 2019

bharatviswa504 force-pushed the HDDS-1909 branch from 338776d to a425167 Compare August 23, 2019 00:44

apache deleted a comment from hadoop-yetus Aug 23, 2019

bharatviswa504 requested a review from arp7 August 23, 2019 23:05