Skip to content

Conversation

@arp7
Copy link
Contributor

@arp7 arp7 commented Apr 11, 2019

…garwal.

Change-Id: I981fbd087baca80cc6b4ff58e91e63dcd9726c13

@arp7 arp7 self-assigned this Apr 11, 2019
@arp7 arp7 added the ozone label Apr 11, 2019
@bharatviswa504
Copy link
Contributor

Hi @arp7
Thanks for the fix.
To understand the fix, we have removed the setting of usage to null, so that even when DN is shut down, and if someone calls volume usage we return old value, instead of throwing an exception.

Is my understanding correct or am I missing something here?

@hadoop-yetus

This comment has been minimized.

@arp7
Copy link
Contributor Author

arp7 commented Apr 11, 2019

Hi @arp7
Thanks for the fix.
To understand the fix, we have removed the setting of usage to null, so that even when DN is shut down, and if someone calls volume usage we return old value, instead of throwing an exception.

Is my understanding correct or am I missing something here?

That is correct @bharatviswa504 .

@bharatviswa504
Copy link
Contributor

Thank You @arp7 for the reply.
Now I want to understand what benefit, this fix will give. If I understand correctly, if during shutdown even if some one is using volume API to get usage they will get some value, instead of exception. Are is there any other thing?

@arp7
Copy link
Contributor Author

arp7 commented Apr 11, 2019

This is to avoid spurious exception messages during DataNode shutdown. They were creating noise in the logs, especially when running MiniOzoneChaosCluster which frequently restarts nodes.

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank You @arp7 for the explanation.
+1 LGTM pending CI.

@arp7
Copy link
Contributor Author

arp7 commented Apr 25, 2019

/retest

@arp7
Copy link
Contributor Author

arp7 commented May 8, 2019

Addressed issues flagged by CI, and rebased to current trunk.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 51 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 2 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 435 trunk passed
+1 compile 212 trunk passed
+1 checkstyle 52 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 841 branch has no errors when building and testing our client artifacts.
+1 javadoc 131 trunk passed
0 spotbugs 325 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 567 trunk passed
_ Patch Compile Tests _
+1 mvninstall 464 the patch passed
+1 compile 251 the patch passed
+1 javac 251 the patch passed
+1 checkstyle 65 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 769 patch has no errors when building and testing our client artifacts.
+1 javadoc 155 the patch passed
+1 findbugs 558 the patch passed
_ Other Tests _
-1 unit 155 hadoop-hdds in the patch failed.
-1 unit 1344 hadoop-ozone in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
6961
Reason Tests
Failed junit tests hadoop.ozone.TestHddsDatanodeService
hadoop.ozone.container.common.volume.TestVolumeSet
hadoop.ozone.container.common.volume.TestVolumeSetDiskChecks
hadoop.ozone.container.keyvalue.TestKeyValueContainer
hadoop.ozone.container.common.TestDatanodeStateMachine
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerCommandHandler
hadoop.ozone.container.keyvalue.TestBlockManagerImpl
hadoop.ozone.container.keyvalue.TestKeyValueBlockIterator
hadoop.ozone.container.ozoneimpl.TestOzoneContainer
hadoop.ozone.container.common.volume.TestHddsVolume
hadoop.ozone.container.keyvalue.TestKeyValueContainerMarkUnhealthy
hadoop.ozone.container.common.volume.TestRoundRobinVolumeChoosingPolicy
hadoop.ozone.container.common.impl.TestHddsDispatcher
hadoop.ozone.container.keyvalue.TestKeyValueContainerCheck
hadoop.ozone.container.keyvalue.TestKeyValueHandler
hadoop.ozone.container.keyvalue.TestChunkManagerImpl
hadoop.ozone.client.rpc.TestBlockOutputStream
hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean
hadoop.ozone.web.TestOzoneWebAccess
hadoop.ozone.client.rpc.TestBCSID
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.scm.node.TestQueryNode
hadoop.ozone.web.client.TestOzoneClient
hadoop.ozone.om.TestOmInit
hadoop.ozone.TestStorageContainerManager
hadoop.hdds.scm.pipeline.TestSCMPipelineManager
hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode
hadoop.ozone.container.common.TestBlockDeletingService
hadoop.ozone.container.TestContainerReplication
hadoop.ozone.TestContainerStateMachineIdempotency
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.container.metrics.TestContainerMetrics
hadoop.ozone.client.rpc.TestContainerStateMachine
hadoop.ozone.scm.node.TestSCMNodeMetrics
hadoop.ozone.om.TestOzoneManager
hadoop.ozone.TestMiniChaosOzoneCluster
hadoop.ozone.om.TestContainerReportWithKeys
hadoop.ozone.om.TestMultipleContainerReadWrite
hadoop.ozone.container.common.impl.TestContainerPersistence
hadoop.hdds.scm.container.TestContainerStateManagerIntegration
hadoop.hdds.scm.pipeline.TestNode2PipelineMap
hadoop.ozone.TestContainerOperations
hadoop.ozone.container.ozoneimpl.TestOzoneContainer
hadoop.ozone.scm.TestXceiverClientMetrics
hadoop.ozone.scm.TestAllocateContainer
hadoop.ozone.container.ozoneimpl.TestSecureOzoneContainer
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.web.client.TestKeys
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerHandler
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.container.common.statemachine.commandhandler.TestDeleteContainerHandler
hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics
hadoop.ozone.ozShell.TestOzoneDatanodeShell
hadoop.ozone.scm.TestGetCommittedBlockLengthAndPutKey
hadoop.ozone.om.TestOmBlockVersioning
hadoop.hdds.scm.pipeline.TestRatisPipelineUtils
hadoop.ozone.om.TestOmAcls
hadoop.ozone.TestMiniOzoneCluster
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
hadoop.ozone.om.TestOmMetrics
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.client.rpc.TestContainerStateMachineFailures
hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.om.TestOMDbCheckpointServlet
hadoop.ozone.ozShell.TestS3Shell
hadoop.ozone.scm.TestContainerSmallFile
hadoop.ozone.scm.TestXceiverClientManager
hadoop.hdds.scm.pipeline.TestPipelineClose
hadoop.ozone.client.rpc.TestWatchForCommit
hadoop.ozone.client.rpc.TestReadRetries
hadoop.ozone.scm.TestSCMNodeManagerMXBean
hadoop.ozone.web.TestOzoneRestWithMiniCluster
hadoop.ozone.scm.TestSCMMXBean
hadoop.ozone.om.TestScmSafeMode
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules
hadoop.ozone.ozShell.TestOzoneShell
hadoop.ozone.om.TestOzoneManagerRestInterface
hadoop.hdds.scm.pipeline.TestNodeFailure
hadoop.ozone.web.TestOzoneVolumes
hadoop.ozone.web.client.TestBuckets
hadoop.ozone.web.client.TestVolume
hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.hdds.scm.pipeline.TestSCMRestart
hadoop.ozone.web.client.TestKeysRatis
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-725/2/artifact/out/Dockerfile
GITHUB PR #725
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 4937e4c93966 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 3418bbb
Default Java 1.8.0_191
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-725/2/artifact/out/patch-unit-hadoop-hdds.txt
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-725/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-725/2/testReport/
Max. process+thread count 5010 (vs. ulimit of 5500)
modules C: hadoop-hdds/container-service U: hadoop-hdds/container-service
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-725/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

…garwal.

Change-Id: I6db6bdd19839a45e5341ed7e745cd38f68af8378
@arp7
Copy link
Contributor Author

arp7 commented May 13, 2019

Fix UT failures caused by previous commit.

@arp7
Copy link
Contributor Author

arp7 commented May 14, 2019

/retest

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM, An extra file is added which is not required can be taken care during the commit.

Copy link
Contributor

@bharatviswa504 bharatviswa504 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM, pending CI.

@anuengineer anuengineer merged commit 732133c into apache:trunk May 18, 2019
@anuengineer
Copy link
Contributor

+1, LGTM. Thanks for the patch, and thanks for the review. @bharatviswa504

ajayydv pushed a commit to ajayydv/hadoop that referenced this pull request May 20, 2019
apache#725)

* HDDS-1422. Exception during DataNode shutdown. Contributed by Arpit Agarwal.

Change-Id: I6db6bdd19839a45e5341ed7e745cd38f68af8378

* Suppress spurious findbugs warning.

* Remove log file that got committed in error
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
1. Made public methods in ApplicationDescriptorImpl non-final for mocking.
2. Updated CachedTableDescriptor public method return types to include type parameters.
3. Changed next release version to 1.0.0
4. Made KafkaSystemAdmin#toKafkaSpec public.

Author: Prateek Maheshwari <pmaheshwari@apache.org>

Reviewers: Shanthoosh Venkatraman <svenkatr@linkedin.com>

Closes apache#725 from prateekm/minor-fixes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants