Skip to content

HDDS-1886. Use ArrayList#clear to address audit failure scenario #1205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

dineshchitlangia
Copy link
Contributor

No description provided.

@dineshchitlangia
Copy link
Contributor Author

/label ozone

@elek elek added the ozone label Aug 1, 2019
@@ -153,7 +153,7 @@ private void verifyLog(String expected) throws IOException {
assertTrue(lines.size() != 0);
assertTrue(expected.equalsIgnoreCase(lines.get(0)));
//empty the file
lines.remove(0);
lines.clear();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dineshchitlangia good idea to avoid unnecessary assumptions.

Here are some further suggestions:

  1. Wrap assertions and cleanup in try and finally, respectively. Otherwise a failed assertion would cause all further independent test cases to also fail due to leftover content in the file. (This one is not specific to multi-line cases.)
  2. Make verifyLog accept String... and check each expected line. Also, retry reading the file until it has enough lines instead of being non-empty.
  3. I think using assertEquals would make it easier to spot differences between expected and actual values. (I'm not sure case-ignorance is really important. Currently the test passes with strict case check, too.)

Copy link
Contributor Author

@dineshchitlangia dineshchitlangia Aug 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai Thanks for review & suggestions.

  1. Wrap assertions and cleanup in try and finally, respectively. Otherwise a failed assertion would cause all further independent test cases to also fail due to leftover content in the file. (This one is not specific to multi-line cases.)

Yes, that was the original approach, however, the review at the time preferred to throw the Exception instead of catching it. This is because we won't accept a few failures, the criteria required is all tests must pass.

  1. Make verifyLog accept String... and check each expected line. Also, retry reading the file until it has enough lines instead of being non-empty.

In this test for the base framework of audit logging, we invoke verifyLog each time a log worthy event has occurred. So at any time, the recent event is the only event in the audit log. Hence we verify only one line at a time as they are invoked by different tests.

  1. I think using assertEquals would make it easier to spot differences between expected and actual values. (I'm not sure case-ignorance is really important. Currently the test passes with strict case check, too.)
    Again, since the test was for base audit logging framework, strict checking was required in the original.

That said, the goal of current jira is only to move away from using ArrayList#remove.
I have filed HDDS-1889 to address the multi-line log scenario that will cover your suggestions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai Could please you let me know if there are any other concerns based on my response above and follow up jira? Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dineshchitlangia Thanks for clarifying the scope and filing the follow-up Jira. I don't have any other concerns.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adoroszlai Thanks for the review. I will commit this now.

@dineshchitlangia
Copy link
Contributor Author

Checkstyle & Integrations failures are unrelated to the patch

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 44 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 582 trunk passed
+1 compile 355 trunk passed
+1 checkstyle 66 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 808 branch has no errors when building and testing our client artifacts.
+1 javadoc 145 trunk passed
0 spotbugs 432 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 616 trunk passed
_ Patch Compile Tests _
+1 mvninstall 567 the patch passed
+1 compile 370 the patch passed
+1 javac 370 the patch passed
+1 checkstyle 67 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 632 patch has no errors when building and testing our client artifacts.
+1 javadoc 160 the patch passed
+1 findbugs 654 the patch passed
_ Other Tests _
+1 unit 281 hadoop-hdds in the patch passed.
-1 unit 153 hadoop-ozone in the patch failed.
+1 asflicense 33 The patch does not generate ASF License warnings.
5686
Reason Tests
Failed junit tests hadoop.ozone.om.ratis.TestOzoneManagerDoubleBufferWithOMResponse
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/1/artifact/out/Dockerfile
GITHUB PR #1205
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux d2102f8a0289 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e111789
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/1/testReport/
Max. process+thread count 506 (vs. ulimit of 5500)
modules C: hadoop-hdds/common U: hadoop-hdds/common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 41 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 635 trunk passed
+1 compile 378 trunk passed
+1 checkstyle 63 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 847 branch has no errors when building and testing our client artifacts.
+1 javadoc 156 trunk passed
0 spotbugs 435 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 625 trunk passed
_ Patch Compile Tests _
+1 mvninstall 574 the patch passed
+1 compile 366 the patch passed
+1 javac 366 the patch passed
+1 checkstyle 72 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 1 The patch has no whitespace issues.
+1 shadedclient 683 patch has no errors when building and testing our client artifacts.
+1 javadoc 154 the patch passed
+1 findbugs 641 the patch passed
_ Other Tests _
+1 unit 282 hadoop-hdds in the patch passed.
-1 unit 1995 hadoop-ozone in the patch failed.
+1 asflicense 43 The patch does not generate ASF License warnings.
7698
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.TestCommitWatcher
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.om.TestKeyManagerImpl
hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.om.TestScmSafeMode
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/2/artifact/out/Dockerfile
GITHUB PR #1205
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux d459ff4e7ee0 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23 09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 70b4617
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/2/testReport/
Max. process+thread count 5402 (vs. ulimit of 5500)
modules C: hadoop-hdds/common U: hadoop-hdds/common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 94 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
+1 mvninstall 595 trunk passed
+1 compile 360 trunk passed
+1 checkstyle 70 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 921 branch has no errors when building and testing our client artifacts.
+1 javadoc 171 trunk passed
0 spotbugs 464 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 687 trunk passed
_ Patch Compile Tests _
+1 mvninstall 629 the patch passed
+1 compile 400 the patch passed
+1 javac 400 the patch passed
+1 checkstyle 87 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 743 patch has no errors when building and testing our client artifacts.
+1 javadoc 181 the patch passed
+1 findbugs 748 the patch passed
_ Other Tests _
+1 unit 340 hadoop-hdds in the patch passed.
-1 unit 2038 hadoop-ozone in the patch failed.
+1 asflicense 42 The patch does not generate ASF License warnings.
8271
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestMultiBlockWritesWithDnFailures
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException
hadoop.ozone.om.TestKeyManagerImpl
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.om.TestScmSafeMode
hadoop.hdds.scm.pipeline.TestRatisPipelineCreateAndDestory
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
Subsystem Report/Notes
Docker Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/3/artifact/out/Dockerfile
GITHUB PR #1205
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 57d888aca484 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 00b5a27
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/3/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/3/testReport/
Max. process+thread count 5315 (vs. ulimit of 5500)
modules C: hadoop-hdds/common U: hadoop-hdds/common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1205/3/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants