Skip to content

[HBASE-22592] : HMaster Construction failure stacktrace to be availab… - #311

Closed
virajjasani wants to merge 4 commits into
apache:masterfrom
virajjasani:HBASE-22592-master
Closed

[HBASE-22592] : HMaster Construction failure stacktrace to be availab…#311
virajjasani wants to merge 4 commits into
apache:masterfrom
virajjasani:HBASE-22592-master

Conversation

@virajjasani

@virajjasanivirajjasani commented Jun 16, 2019

Copy link
Copy Markdown
Contributor

…le in .out file - Useful if Logger class is not loaded yet

An example of Exception(NoSuchMethodError) which is not present in .log/.out files:

image-2019-06-16-15-06-40-109

…le in .out file - Useful if Logger class is not loaded yet
@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec172Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
-0test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1mvninstall253master passed
+1compile53master passed
+1checkstyle68master passed
+1shadedjars264branch has no errors when building our shaded downstream artifacts.
+1findbugs198master passed
+1javadoc32master passed
_ Patch Compile Tests _
+1mvninstall239the patch passed
+1compile51the patch passed
+1javac51the patch passed
+1checkstyle69the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedjars269patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck735Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs168the patch passed
+1javadoc32the patch passed
_ Other Tests _
-1unit12599hbase-server in the patch failed.
+1asflicense33The patch does not generate ASF License warnings.
15557
ReasonTests
Failed junit testshadoop.hbase.client.TestAsyncRegionAdminApi
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/artifact/out/Dockerfile
GITHUB PR#311
JIRA IssueHBASE-22592
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 79f85aa8ee2a 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 1712d2b
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/testReport/
Max. process+thread count5014 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/1/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

retest this please

@HorizonNet

Copy link
Copy Markdown
Contributor

I kicked off another build in case the test failure is unrelated.

throw new RuntimeException("HMaster Aborted");
}
} catch (Throwable t) {
t.printStackTrace();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will the error be present in the log files with this change? Is there a better way to do this, but still using the logger?

@virajjasanivirajjasaniJun 16, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that but as per the screenshot provided in the description, the Exception provided is not available in log file because Logger is not yet loaded.
Anyways, adding printstacktrace() would ensure that the StackTrace is atleast available in .out file. If the Logger is created and loaded by that time, then it would also be present in .log file based on: LOG.error("Master exiting", t);

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec54Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
-0test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1mvninstall239master passed
+1compile53master passed
+1checkstyle67master passed
+1shadedjars268branch has no errors when building our shaded downstream artifacts.
+1findbugs200master passed
+1javadoc33master passed
_ Patch Compile Tests _
+1mvninstall242the patch passed
+1compile51the patch passed
+1javac51the patch passed
+1checkstyle65the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedjars270patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck744Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs200the patch passed
+1javadoc33the patch passed
_ Other Tests _
-1unit13557hbase-server in the patch failed.
+1asflicense31The patch does not generate ASF License warnings.
16434
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/artifact/out/Dockerfile
GITHUB PR#311
JIRA IssueHBASE-22592
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux a919a19ccce7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / ed30909
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/testReport/
Max. process+thread count5273 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/2/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

Error not relevant.
retest this please

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec45Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
-0test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1mvninstall240master passed
+1compile53master passed
+1checkstyle65master passed
+1shadedjars276branch has no errors when building our shaded downstream artifacts.
+1findbugs179master passed
+1javadoc32master passed
_ Patch Compile Tests _
+1mvninstall262the patch passed
+1compile56the patch passed
+1javac56the patch passed
+1checkstyle69the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedjars262patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck740Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs176the patch passed
+1javadoc34the patch passed
_ Other Tests _
-1unit16854hbase-server in the patch failed.
+1asflicense25The patch does not generate ASF License warnings.
19679
ReasonTests
Failed junit testshadoop.hbase.client.TestFromClientSide
hadoop.hbase.client.TestFromClientSide3
hadoop.hbase.quotas.TestSpaceQuotas
hadoop.hbase.master.procedure.TestSCPWithReplicas
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/artifact/out/Dockerfile
GITHUB PR#311
JIRA IssueHBASE-22592
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 64bd6c9825d6 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 9b413cf
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/testReport/
Max. process+thread count5263 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/3/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
0reexec50Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
-0test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1mvninstall240master passed
+1compile54master passed
+1checkstyle71master passed
+1shadedjars266branch has no errors when building our shaded downstream artifacts.
+1findbugs155master passed
+1javadoc34master passed
_ Patch Compile Tests _
+1mvninstall241the patch passed
+1compile51the patch passed
+1javac51the patch passed
+1checkstyle64the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedjars263patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck742Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs199the patch passed
+1javadoc34the patch passed
_ Other Tests _
-1unit16073hbase-server in the patch failed.
+1asflicense26The patch does not generate ASF License warnings.
18878
ReasonTests
Failed junit testshadoop.hbase.regionserver.TestSplitTransactionOnCluster
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/artifact/out/Dockerfile
GITHUB PR#311
JIRA IssueHBASE-22592
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux af0a84ca8d36 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 5f2699e
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/testReport/
Max. process+thread count4910 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/4/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
0reexec27Docker mode activated.
_ Prechecks _
+1hbaseanti0Patch does not have any anti-patterns.
+1@author0The patch does not contain any @author tags.
-0test4tests0The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1mvninstall297master passed
+1compile61master passed
+1checkstyle81master passed
+1shadedjars260branch has no errors when building our shaded downstream artifacts.
+1findbugs167master passed
+1javadoc31master passed
_ Patch Compile Tests _
+1mvninstall238the patch passed
+1compile50the patch passed
+1javac50the patch passed
+1checkstyle65the patch passed
+1whitespace0The patch has no whitespace issues.
+1shadedjars250patch has no errors when building our shaded downstream artifacts.
+1hadoopcheck717Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1findbugs225the patch passed
+1javadoc33the patch passed
_ Other Tests _
+1unit8461hbase-server in the patch passed.
+1asflicense24The patch does not generate ASF License warnings.
11300
SubsystemReport/Notes
DockerClient=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/5/artifact/out/Dockerfile
GITHUB PR#311
JIRA IssueHBASE-22592
Optional Testsdupname asflicense javac javadoc unit findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 97845c247046 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 GNU/Linux
Build toolmaven
Personality/testptch/patchprocess/precommit/personality/provided.sh
git revisionmaster / 214553d
mavenversion: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)
Default Java1.8.0_181
findbugsv3.1.11
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/5/testReport/
Max. process+thread count4827 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-311/5/console
Powered byApache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

@apurtell please review

@busbeybusbey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't print stuff to stdout if there's a logger available. Please refactor to only print if the logging system isn't present. (which I'm not sure is possible)

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

we don't print stuff to stdout if there's a logger available. Please refactor to only print if the logging system isn't present. (which I'm not sure is possible)

@busbey I agree we shouldn't print everything to stdout when logger is available. However, the error present in the screenshot was encountered due to a library that contained some incompatible guava method and HBase had classpath setup to use that library. Hence, when HMaster was started, no error was printed to .log or .out file. We had to attach JDB to figure out this Exception.
If HMaster fails due to some valid Exception, do you think it is fine to print the StackTrace on stdout also. Not that it will harm anything but it will be redundant if Logger is already loaded in JVM, else stdout will be the only source of truth to get the details of Exception.

@busbey

Copy link
Copy Markdown
Contributor

the current patch prints a stack trace to stdout for any throwable. we know there are lots of ways to bring down the master with a throwable that will have a logger in place. Don't print those to stdout.

at a minimum you should be adding a catch block for things that are likely to go wrong in a way that can predate a logger being available in the runtime.

The stacktrace in your screen shot shows a problem that should not be possible before the logging system is available (it happens in a catch block in an instance method and the logger is a final static variable that should be initialized already before any instances can be created). So please include a reproduction that ensures proper logging configs.

@virajjasani

Copy link
Copy Markdown
ContributorAuthor

After updating the dependent library, I am not able to repro this Exception anymore. Tried certain ways but didn't get much progress.
Since it is really difficult to believe that Logger might have some issues as it is already a public static final, closing this PR

@virajjasani
virajjasani deleted the HBASE-22592-master branch June 19, 2019 15:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@virajjasani@Apache-HBase@HorizonNet@busbey