Skip to content

HDDS-1401. Static ContainerCache in Datanodes can result in overwrite of container db. Contributed by Mukul Kumar Singh. #708

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Apr 10, 2019
Merged

HDDS-1401. Static ContainerCache in Datanodes can result in overwrite of container db. Contributed by Mukul Kumar Singh. #708

merged 4 commits into from
Apr 10, 2019

Conversation

mukul1987
Copy link
Contributor

The datanodes use a static Container cache, and this container cache is key'd using the container ID.
In a MiniOzoneCluster environment, this can lead to multiple container on different datanodes to use the same rocksdb leading to inconsistency.

@mukul1987
HDDS-1401. Static ContainerCache in Datanodes can result in overwrite…
7bb49a1 
… of container db. Contributed by Mukul Kumar Singh.
@mukul1987 mukul1987 added the ozone label Apr 8, 2019
@mukul1987 mukul1987 self-assigned this Apr 8, 2019
@mukul1987 mukul1987 requested review from arp7 and nandakumar131 April 8, 2019 17:56
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 27 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 23 Maven dependency ordering for branch
+1 mvninstall 1024 trunk passed
+1 compile 968 trunk passed
+1 checkstyle 180 trunk passed
+1 mvnsite 121 trunk passed
+1 shadedclient 991 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
+1 findbugs 95 trunk passed
+1 javadoc 83 trunk passed
_ Patch Compile Tests _
0 mvndep 22 Maven dependency ordering for patch
+1 mvninstall 94 the patch passed
+1 compile 906 the patch passed
+1 javac 906 the patch passed
+1 checkstyle 181 the patch passed
+1 mvnsite 103 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 615 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1 findbugs 59 hadoop-hdds/container-service generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 javadoc 88 the patch passed
_ Other Tests _
-1 unit 73 container-service in the patch failed.
+1 unit 109 server-scm in the patch passed.
-1 unit 1382 integration-test in the patch failed.
+1 asflicense 49 The patch does not generate ASF License warnings.
7190
Reason Tests
FindBugs module:hadoop-hdds/container-service
org.apache.hadoop.ozone.container.common.utils.ContainerCache.removeLRU(AbstractLinkedMap$LinkEntry) does not release lock on all exception paths At ContainerCache.java:on all exception paths At ContainerCache.java:[line 110]
Failed junit tests hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.container.common.impl.TestContainerPersistence
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/Dockerfile
GITHUB PR #708
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 6cfaff448837 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 69e3745
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
findbugs v3.1.0-RC1
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/new-findbugs-hadoop-hdds_container-service.html
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/patch-unit-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/testReport/
Max. process+thread count 4009 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-708/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 29 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 96 Maven dependency ordering for branch
+1 mvninstall 1116 trunk passed
+1 compile 975 trunk passed
+1 checkstyle 181 trunk passed
-1 mvnsite 41 container-service in trunk failed.
+1 shadedclient 983 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1 findbugs 26 container-service in trunk failed.
+1 javadoc 80 trunk passed
_ Patch Compile Tests _
0 mvndep 24 Maven dependency ordering for patch
-1 mvninstall 18 container-service in the patch failed.
+1 compile 976 the patch passed
+1 javac 976 the patch passed
+1 checkstyle 181 the patch passed
-1 mvnsite 26 container-service in the patch failed.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 621 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1 findbugs 27 container-service in the patch failed.
+1 javadoc 78 the patch passed
_ Other Tests _
-1 unit 28 container-service in the patch failed.
+1 unit 113 server-scm in the patch passed.
-1 unit 1211 integration-test in the patch failed.
+1 asflicense 47 The patch does not generate ASF License warnings.
7098
Reason Tests
Failed junit tests hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
hadoop.ozone.TestMiniChaosOzoneCluster
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/Dockerfile
GITHUB PR #708
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 81fbf94d3710 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 2d4f6b6
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/branch-mvnsite-hadoop-hdds_container-service.txt
findbugs v3.1.0-RC1
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/patch-mvninstall-hadoop-hdds_container-service.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/patch-mvnsite-hadoop-hdds_container-service.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/patch-findbugs-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/patch-unit-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/testReport/
Max. process+thread count 4818 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-708/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 28 Docker mode activated.
_ Prechecks _
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 3 new or modified test files.
_ trunk Compile Tests _
0 mvndep 22 Maven dependency ordering for branch
-1 mvninstall 1012 root in trunk failed.
-1 compile 926 root in trunk failed.
+1 checkstyle 197 trunk passed
-1 mvnsite 33 container-service in trunk failed.
+1 shadedclient 978 branch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1 findbugs 26 container-service in trunk failed.
+1 javadoc 86 trunk passed
_ Patch Compile Tests _
0 mvndep 21 Maven dependency ordering for patch
-1 mvninstall 18 container-service in the patch failed.
-1 compile 881 root in the patch failed.
-1 javac 881 root in the patch failed.
+1 checkstyle 193 the patch passed
-1 mvnsite 37 container-service in the patch failed.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 621 patch has no errors when building and testing our client artifacts.
0 findbugs 0 Skipped patched modules with no Java source: hadoop-ozone/integration-test
-1 findbugs 29 container-service in the patch failed.
+1 javadoc 81 the patch passed
_ Other Tests _
-1 unit 28 container-service in the patch failed.
+1 unit 101 server-scm in the patch passed.
-1 unit 697 integration-test in the patch failed.
+1 asflicense 47 The patch does not generate ASF License warnings.
6315
Reason Tests
Failed junit tests hadoop.ozone.om.TestOzoneManager
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.web.client.TestOzoneClient
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.web.client.TestBuckets
hadoop.ozone.om.TestOmAcls
hadoop.ozone.om.TestOmInit
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.om.TestMultipleContainerReadWrite
hadoop.ozone.om.TestOmBlockVersioning
hadoop.ozone.scm.node.TestQueryNode
hadoop.ozone.web.TestOzoneVolumes
hadoop.ozone.web.TestOzoneWebAccess
hadoop.ozone.om.TestScmChillMode
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/Dockerfile
GITHUB PR #708
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 1da33cd75fac 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / cfec455
maven version: Apache Maven 3.3.9
Default Java 1.8.0_191
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/branch-mvninstall-root.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/branch-compile-root.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/branch-mvnsite-hadoop-hdds_container-service.txt
findbugs v3.1.0-RC1
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
mvninstall https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-mvninstall-hadoop-hdds_container-service.txt
compile https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-compile-root.txt
javac https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-compile-root.txt
mvnsite https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-mvnsite-hadoop-hdds_container-service.txt
findbugs https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-findbugs-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-unit-hadoop-hdds_container-service.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/artifact/out/patch-unit-hadoop-ozone_integration-test.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/testReport/
Max. process+thread count 4579 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-708/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 0 Docker mode activated.
-1 patch 9 #708 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
Subsystem Report/Notes
GITHUB PR #708
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-708/4/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

nandakumar131
nandakumar131 approved these changes Apr 10, 2019
View reviewed changes
Copy link
Contributor

@nandakumar131 nandakumar131 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@nandakumar131 nandakumar131 merged commit df01469 into apache:trunk Apr 10, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
@prateekm
System and Stream Descriptor API cleanup.
378fd8a 
Major changes:
1. Made withPhysicalName(String) a protected method in StreamDescriptor.
The primary reason to set the physical name is to use stream names with special characters in them, which streamId doesn't support. This change is to make it so that the physical name setter should only be exposed by systems where it means something useful - like HDFS.
2. Renamed some methods in StreamDescriptor for clarity.

Author: Prateek Maheshwari <pmaheshwari@apache.org>

Reviewers: Cameron Lee <calee@linkedin.com>

Closes apache#708 from prateekm/stream-descriptor-cleanup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nandakumar131 nandakumar131 nandakumar131 approved these changes

@arp7 arp7 Awaiting requested review from arp7

Assignees

@mukul1987 mukul1987

Labels
ozone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mukul1987 @hadoop-yetus @nandakumar131