Skip to content

[HBASE-22592] : HMaster Construction failure stacktrace to be availab… #311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

virajjasani
Copy link
Contributor

@virajjasani virajjasani commented Jun 16, 2019

…le in .out file - Useful if Logger class is not loaded yet

An example of Exception(NoSuchMethodError) which is not present in .log/.out files:

image-2019-06-16-15-06-40-109

…le in .out file - Useful if Logger class is not loaded yet
@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 172 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 253 master passed
+1 compile 53 master passed
+1 checkstyle 68 master passed
+1 shadedjars 264 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 198 master passed
+1 javadoc 32 master passed
_ Patch Compile Tests _
+1 mvninstall 239 the patch passed
+1 compile 51 the patch passed
+1 javac 51 the patch passed
+1 checkstyle 69 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 269 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 735 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 findbugs 168 the patch passed
+1 javadoc 32 the patch passed
_ Other Tests _
-1 unit 12599 hbase-server in the patch failed.
+1 asflicense 33 The patch does not generate ASF License warnings.
15557
Reason Tests
Failed junit tests hadoop.hbase.client.TestAsyncRegionAdminApi
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/artifact/out/Dockerfile
GITHUB PR #311
JIRA Issue HBASE-22592
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 79f85aa8ee2a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / 1712d2b
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/testReport/
Max. process+thread count 5014 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@virajjasani
Copy link
Contributor Author

retest this please

@HorizonNet
Copy link
Contributor

I kicked off another build in case the test failure is unrelated.

@@ -244,6 +244,7 @@ private int startMaster() {
throw new RuntimeException("HMaster Aborted");
}
} catch (Throwable t) {
t.printStackTrace();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the error be present in the log files with this change? Is there a better way to do this, but still using the logger?

Copy link
Contributor Author

@virajjasani virajjasani Jun 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that but as per the screenshot provided in the description, the Exception provided is not available in log file because Logger is not yet loaded.
Anyways, adding printstacktrace() would ensure that the StackTrace is atleast available in .out file. If the Logger is created and loaded by that time, then it would also be present in .log file based on: LOG.error("Master exiting", t);

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 54 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 239 master passed
+1 compile 53 master passed
+1 checkstyle 67 master passed
+1 shadedjars 268 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 200 master passed
+1 javadoc 33 master passed
_ Patch Compile Tests _
+1 mvninstall 242 the patch passed
+1 compile 51 the patch passed
+1 javac 51 the patch passed
+1 checkstyle 65 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 270 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 744 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 findbugs 200 the patch passed
+1 javadoc 33 the patch passed
_ Other Tests _
-1 unit 13557 hbase-server in the patch failed.
+1 asflicense 31 The patch does not generate ASF License warnings.
16434
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/artifact/out/Dockerfile
GITHUB PR #311
JIRA Issue HBASE-22592
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux a919a19ccce7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / ed30909
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/testReport/
Max. process+thread count 5273 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@virajjasani
Copy link
Contributor Author

Error not relevant.
retest this please

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 45 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 240 master passed
+1 compile 53 master passed
+1 checkstyle 65 master passed
+1 shadedjars 276 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 179 master passed
+1 javadoc 32 master passed
_ Patch Compile Tests _
+1 mvninstall 262 the patch passed
+1 compile 56 the patch passed
+1 javac 56 the patch passed
+1 checkstyle 69 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 262 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 740 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 findbugs 176 the patch passed
+1 javadoc 34 the patch passed
_ Other Tests _
-1 unit 16854 hbase-server in the patch failed.
+1 asflicense 25 The patch does not generate ASF License warnings.
19679
Reason Tests
Failed junit tests hadoop.hbase.client.TestFromClientSide
hadoop.hbase.client.TestFromClientSide3
hadoop.hbase.quotas.TestSpaceQuotas
hadoop.hbase.master.procedure.TestSCPWithReplicas
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/artifact/out/Dockerfile
GITHUB PR #311
JIRA Issue HBASE-22592
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 64bd6c9825d6 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / 9b413cf
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/testReport/
Max. process+thread count 5263 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 50 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 240 master passed
+1 compile 54 master passed
+1 checkstyle 71 master passed
+1 shadedjars 266 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 155 master passed
+1 javadoc 34 master passed
_ Patch Compile Tests _
+1 mvninstall 241 the patch passed
+1 compile 51 the patch passed
+1 javac 51 the patch passed
+1 checkstyle 64 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 263 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 742 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 findbugs 199 the patch passed
+1 javadoc 34 the patch passed
_ Other Tests _
-1 unit 16073 hbase-server in the patch failed.
+1 asflicense 26 The patch does not generate ASF License warnings.
18878
Reason Tests
Failed junit tests hadoop.hbase.regionserver.TestSplitTransactionOnCluster
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/artifact/out/Dockerfile
GITHUB PR #311
JIRA Issue HBASE-22592
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux af0a84ca8d36 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / 5f2699e
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
unit https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/artifact/out/patch-unit-hbase-server.txt
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/testReport/
Max. process+thread count 4910 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
0 reexec 27 Docker mode activated.
_ Prechecks _
+1 hbaseanti 0 Patch does not have any anti-patterns.
+1 @author 0 The patch does not contain any @author tags.
-0 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 mvninstall 297 master passed
+1 compile 61 master passed
+1 checkstyle 81 master passed
+1 shadedjars 260 branch has no errors when building our shaded downstream artifacts.
+1 findbugs 167 master passed
+1 javadoc 31 master passed
_ Patch Compile Tests _
+1 mvninstall 238 the patch passed
+1 compile 50 the patch passed
+1 javac 50 the patch passed
+1 checkstyle 65 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedjars 250 patch has no errors when building our shaded downstream artifacts.
+1 hadoopcheck 717 Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 findbugs 225 the patch passed
+1 javadoc 33 the patch passed
_ Other Tests _
+1 unit 8461 hbase-server in the patch passed.
+1 asflicense 24 The patch does not generate ASF License warnings.
11300
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/5/artifact/out/Dockerfile
GITHUB PR #311
JIRA Issue HBASE-22592
Optional Tests dupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
uname Linux 97845c247046 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build tool maven
Personality /testptch/patchprocess/precommit/personality/provided.sh
git revision master / 214553d
maven version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java 1.8.0_181
findbugs v3.1.11
Test Results https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/5/testReport/
Max. process+thread count 4827 (vs. ulimit of 10000)
modules C: hbase-server U: hbase-server
Console output https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/5/console
Powered by Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@virajjasani
Copy link
Contributor Author

@apurtell please review

Copy link
Contributor

@busbey busbey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't print stuff to stdout if there's a logger available. Please refactor to only print if the logging system isn't present. (which I'm not sure is possible)

@virajjasani
Copy link
Contributor Author

we don't print stuff to stdout if there's a logger available. Please refactor to only print if the logging system isn't present. (which I'm not sure is possible)

@busbey I agree we shouldn't print everything to stdout when logger is available. However, the error present in the screenshot was encountered due to a library that contained some incompatible guava method and HBase had classpath setup to use that library. Hence, when HMaster was started, no error was printed to .log or .out file. We had to attach JDB to figure out this Exception.
If HMaster fails due to some valid Exception, do you think it is fine to print the StackTrace on stdout also. Not that it will harm anything but it will be redundant if Logger is already loaded in JVM, else stdout will be the only source of truth to get the details of Exception.

@busbey
Copy link
Contributor

busbey commented Jun 19, 2019

the current patch prints a stack trace to stdout for any throwable. we know there are lots of ways to bring down the master with a throwable that will have a logger in place. Don't print those to stdout.

at a minimum you should be adding a catch block for things that are likely to go wrong in a way that can predate a logger being available in the runtime.

The stacktrace in your screen shot shows a problem that should not be possible before the logging system is available (it happens in a catch block in an instance method and the logger is a final static variable that should be initialized already before any instances can be created). So please include a reproduction that ensures proper logging configs.

@virajjasani
Copy link
Contributor Author

After updating the dependent library, I am not able to repro this Exception anymore. Tried certain ways but didn't get much progress.
Since it is really difficult to believe that Logger might have some issues as it is already a public static final, closing this PR

@virajjasani virajjasani deleted the HBASE-22592-master branch June 19, 2019 15:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants