Skip to content

HBASE-23333 Provide call context around timeouts and other failure scenarios - #872

Merged
ndimiduk merged 1 commit into
apache:masterfrom
ndimiduk:23333-call-context
Jan 3, 2020
Merged

HBASE-23333 Provide call context around timeouts and other failure scenarios#872
ndimiduk merged 1 commit into
apache:masterfrom
ndimiduk:23333-call-context

Conversation

@ndimiduk

@ndimidukndimiduk commented Nov 23, 2019

Copy link
Copy Markdown
Member

No description provided.

@ndimiduk

ndimiduk commented Nov 23, 2019

Copy link
Copy Markdown
MemberAuthor

Initial commit simply include the output of Call#toString where before we only have call.id. Looking to see if there's a sensible way to include a call stack that tracks up into the application code. Will be back.

@Apache-HBase

Apache-HBase commented Nov 23, 2019

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 33sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
-0 ⚠️test4tests0m 0sThe patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ master Compile Tests _
+1 💚mvninstall5m 25smaster passed
+1 💚compile0m 26smaster passed
+1 💚checkstyle0m 32smaster passed
+1 💚shadedjars4m 34sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
+0 🆗spotbugs1m 7sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 6smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 50sthe patch passed
+1 💚compile0m 25sthe patch passed
+1 💚javac0m 25sthe patch passed
+1 💚checkstyle0m 32shbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars4m 33spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck15m 45sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚javadoc0m 23sthe patch passed
+1 💚findbugs1m 12sthe patch passed
_ Other Tests _
+1 💚unit1m 53shbase-client in the patch passed.
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
49m 8s
SubsystemReport/Notes
DockerClient=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/1/artifact/out/Dockerfile
GITHUB PR#872
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 435e5cb8da54 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revisionmaster / ee730c8
Default Java1.8.0_181
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/1/testReport/
Max. process+thread count294 (vs. ulimit of 10000)
modulesC: hbase-client U: hbase-client
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/1/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

bharathv
bharathv approved these changes Nov 27, 2019
View reviewed changes

@bharathvbharathv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice cleanup. Couple of optional nits.

Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Outdated
Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Outdated
Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java
public String toString() {
return "callId: " + this.id + " methodName: " + this.md.getName() + " param {"
+ (this.param != null ? ProtobufUtil.getShortTextFormat(this.param) : "") + "}";
return new ToStringBuilder(this, ToStringStyle.SHORT_PREFIX_STYLE)

@bharathvbharathvNov 27, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice..

@ndimiduk
ndimidukforce-pushed the 23333-call-context branch 2 times, most recently from a56df5f to c7953ddCompareDecember 4, 2019 23:18
@ndimiduk

ndimiduk commented Dec 4, 2019

Copy link
Copy Markdown
MemberAuthor

I found more-or-less what I was looking for over on HBASE-22316. Somehow the piece of stack trace i found in my logs was omitted. Settling for this for now.

@Apache-HBase

Apache-HBase commented Dec 5, 2019

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 55sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 46sMaven dependency ordering for branch
+1 💚mvninstall5m 15smaster passed
+1 💚compile0m 49smaster passed
+1 💚checkstyle0m 59smaster passed
+1 💚shadedjars4m 35sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45smaster passed
+0 🆗spotbugs1m 4sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 49smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall4m 56sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
+1 💚checkstyle0m 26sThe patch passed checkstyle in hbase-common
+1 💚checkstyle0m 31shbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars4m 35spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck15m 38sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚javadoc0m 45sthe patch passed
+1 💚findbugs2m 15sthe patch passed
_ Other Tests _
+1 💚unit3m 15shbase-common in the patch passed.
+1 💚unit1m 55shbase-client in the patch passed.
+1 💚asflicense0m 26sThe patch does not generate ASF License warnings.
59m 42s
SubsystemReport/Notes
DockerClient=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/2/artifact/out/Dockerfile
GITHUB PR#872
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux aa33225b15ec 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revisionmaster / 5e34a59
Default Java1.8.0_181
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/2/testReport/
Max. process+thread count292 (vs. ulimit of 10000)
modulesC: hbase-common hbase-client U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/2/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

Apache9
Apache9 requested changes Dec 5, 2019
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the new error message look like? Will it be too long as we log the whole call?

Comment threadhbase-common/src/test/java/org/apache/hadoop/hbase/util/TestFutureUtils.java
assertTrue(Stream.of(elements)
.anyMatch(element -> element.toString().contains("--------Future.get--------")));
} catch (Throwable t) {
throw new AssertionError("Caught unexpected Throwable", t);

@Apache9Apache9Dec 5, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's this...

@ndimidukndimidukDec 5, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to ensure that the test fails when the exception we expect isn't thrown. It would be simply fail() except that api doesn't let me retain the context provided by t.

@ndimiduk

ndimiduk commented Dec 5, 2019

Copy link
Copy Markdown
MemberAuthor

Maybe this is too verbose?

org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=11, exceptions:
Thu Dec 05 23:11:14 UTC 2019, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60160: Call to redacted:16020 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call[id=4,methodName=Scan,param=region { type: REGION_NAME value: "hbase:meta,,1" } scan { column { family: "info" } start_row: “my_table,,” stop_row: “my_table ,," time_range { from: 0 to: 9223372036854775807 } max_versions: 1 cache_blocks: true max_result_size: 2097152 caching: 100 include_stop_row: false readType: PREAD } number_of_rows: 100 close_scanner: false client_handles_partials: true client_handles_heartbeats: true track_scan_metrics: false limit_of_rows: 2147483647], waitTime=60021, rpcTimeout=59999 row ‘my_table,,’ on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=redacted,16020,1575517640564, seqNum=-1

@ndimiduk

ndimiduk commented Dec 6, 2019

Copy link
Copy Markdown
MemberAuthor

Rebased onto master and reduced the verbosity to just call.id and call.methodName. It would be nice to selectively include parameters, such as all non-default values. Could update the log sites to log less when the redundant values are already in the params...

What do you folks think?

@Apache-HBase

Apache-HBase commented Dec 6, 2019

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 35sMaven dependency ordering for branch
+1 💚mvninstall5m 16smaster passed
+1 💚compile0m 50smaster passed
+1 💚checkstyle1m 0smaster passed
+1 💚shadedjars4m 36sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45smaster passed
+0 🆗spotbugs1m 7sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 53smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall4m 56sthe patch passed
+1 💚compile0m 50sthe patch passed
+1 💚javac0m 50sthe patch passed
+1 💚checkstyle0m 25sThe patch passed checkstyle in hbase-common
+1 💚checkstyle0m 31shbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars4m 34spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck15m 49sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚javadoc0m 44sthe patch passed
+1 💚findbugs2m 9sthe patch passed
_ Other Tests _
+1 💚unit3m 9shbase-common in the patch passed.
+1 💚unit1m 52shbase-client in the patch passed.
+1 💚asflicense0m 26sThe patch does not generate ASF License warnings.
57m 51s
SubsystemReport/Notes
DockerClient=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/3/artifact/out/Dockerfile
GITHUB PR#872
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 5620002d1c93 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revisionmaster / 6d251ef
Default Java1.8.0_181
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/3/testReport/
Max. process+thread count292 (vs. ulimit of 10000)
modulesC: hbase-common hbase-client U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/3/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@ndimiduk

ndimiduk commented Dec 9, 2019

Copy link
Copy Markdown
MemberAuthor

Still have objections @Apache9 ?

bharathv
bharathv approved these changes Dec 11, 2019
View reviewed changes

@bharathvbharathv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patch seems fine to me, fwiw.

Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/ipc/Call.java Outdated
@Apache-HBase

Apache-HBase commented Jan 3, 2020

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
+0 🆗mvndep0m 35sMaven dependency ordering for branch
+1 💚mvninstall5m 21smaster passed
+1 💚compile0m 49smaster passed
+1 💚checkstyle0m 58smaster passed
+1 💚shadedjars4m 37sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 46smaster passed
+0 🆗spotbugs1m 5sUsed deprecated FindBugs config; considering switching to SpotBugs.
+1 💚findbugs1m 50smaster passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall4m 54sthe patch passed
+1 💚compile0m 49sthe patch passed
+1 💚javac0m 49sthe patch passed
+1 💚checkstyle0m 26sThe patch passed checkstyle in hbase-common
+1 💚checkstyle0m 31shbase-client: The patch generated 0 new + 7 unchanged - 3 fixed = 7 total (was 10)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚shadedjars4m 31spatch has no errors when building our shaded downstream artifacts.
+1 💚hadoopcheck15m 38sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
+1 💚javadoc0m 44sthe patch passed
+1 💚findbugs2m 15sthe patch passed
_ Other Tests _
+1 💚unit3m 15shbase-common in the patch passed.
+1 💚unit1m 58shbase-client in the patch passed.
+1 💚asflicense0m 24sThe patch does not generate ASF License warnings.
58m 7s
SubsystemReport/Notes
DockerClient=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/4/artifact/out/Dockerfile
GITHUB PR#872
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 79f2b61f1698 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-872/out/precommit/personality/provided.sh
git revisionmaster / 85ecdd1
Default Java1.8.0_181
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/4/testReport/
Max. process+thread count286 (vs. ulimit of 10000)
modulesC: hbase-common hbase-client U: .
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-872/4/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

@ndimiduk
ndimiduk merged commit af3bb1f into apache:masterJan 3, 2020
@ndimiduk
ndimiduk deleted the 23333-call-context branch January 3, 2020 19:09
This was referenced Jan 3, 2020
Merged
Merged
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@saintstacksaintstacksaintstack left review comments
@Apache9Apache9Apache9 requested changes
@bharathvbharathvbharathv approved these changes
@virajjasanivirajjasanivirajjasani approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ndimiduk@Apache-HBase@saintstack@bharathv@Apache9@virajjasani