Skip to content

HBASE-27398 Remove dumping of EOFException while reading WAL with ProtobufLogReader - #4806

Closed
virajjasani wants to merge 2 commits into
apache:masterfrom
virajjasani:HBASE-27398-master
Closed

HBASE-27398 Remove dumping of EOFException while reading WAL with ProtobufLogReader#4806
virajjasani wants to merge 2 commits into
apache:masterfrom
virajjasani:HBASE-27398-master

Conversation

@virajjasani

@virajjasanivirajjasani commented Sep 28, 2022

Copy link
Copy Markdown
Contributor

No description provided.

@Apache-HBase

Apache-HBase commented Sep 28, 2022

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 6sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 32smaster passed
+1 💚compile2m 16smaster passed
+1 💚checkstyle0m 32smaster passed
+1 💚spotless0m 49sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 21smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 12sthe patch passed
+1 💚compile2m 15sthe patch passed
+1 💚javac2m 15sthe patch passed
+1 💚checkstyle0m 31sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck7m 59sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
-1 ❌spotless0m 33spatch has 26 errors when running spotless:check, run spotless:apply to fix.
+1 💚spotbugs1m 26sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
29m 59s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4806
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux dd6e7acc0123 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 23a5633
Default JavaTemurin-1.8.0_345-b01
spotlesshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count64 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.7.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 28, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 8sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 24smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars3m 49sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 18sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars3m 48spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sthe patch passed
_ Other Tests _
+1 💚unit218m 23shbase-server in the patch passed.
236m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4806
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b634435e81c8 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 23a5633
Default JavaTemurin-1.8.0_345-b01
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/testReport/
Max. process+thread count2557 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 28, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 1sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 12smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars3m 54sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 29smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 38sthe patch passed
+1 💚compile0m 46sthe patch passed
+1 💚javac0m 46sthe patch passed
+1 💚shadedjars3m 46spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit232m 5shbase-server in the patch passed.
250m 46s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4806
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 9f9211d18ece 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 23a5633
Default JavaEclipse Adoptium-11.0.16.1+1
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/testReport/
Max. process+thread count2682 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 28, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 28sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 26smaster passed
+1 💚compile2m 16smaster passed
+1 💚checkstyle0m 32smaster passed
+1 💚spotless0m 42sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 19smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 7sthe patch passed
+1 💚compile2m 12sthe patch passed
+1 💚javac2m 12sthe patch passed
+1 💚checkstyle0m 31sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck7m 55sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚spotless0m 37spatch has no errors when running spotless:check.
+1 💚spotbugs1m 23sthe patch passed
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
29m 29s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4806
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux d55a1526cb76 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 23a5633
Default JavaTemurin-1.8.0_345-b01
Max. process+thread count60 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.7.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache9

Apache9 commented Sep 29, 2022

Copy link
Copy Markdown
Contributor

Please wait a bit more time before merging. @ddupg is still trying to fix the EOF related problem in #4756, I'm not sure if removing the exception stacktrace will have negative impact there, especially that this is just a debug log, not info...

Thanks.

Apache9
Apache9 requested changes Sep 29, 2022
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, after reading the related code, I do not think we should remove the dumping here.

We can throw EOFException at many places in the try block, and they all have a fat message to describe what is going on, and if we remove the dumping of EOFException below, there is no way for us to print out these messages.

IF this is too excessive, I think we should try to avoid throwing too many EOFExceptions for normal cases, for example, if we are tailing a WAL which is currently being written, then at the last position we will get an EOF, and if we want to catch up fast, we will get a lot EOF, which is expected, so we should try to reduce these noises, instead of purging the output for all the EOFException.

Thanks.

@Apache-HBase

Apache-HBase commented Sep 29, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 48sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 50smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars3m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 29smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 35sthe patch passed
+1 💚compile0m 47sthe patch passed
+1 💚javac0m 47sthe patch passed
+1 💚shadedjars3m 45spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit201m 41shbase-server in the patch passed.
219m 34s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4806
Optional Testsjavac javadoc unit shadedjars compile
unameLinux cbe7262f8d17 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 23a5633
Default JavaEclipse Adoptium-11.0.16.1+1
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/testReport/
Max. process+thread count2567 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 29, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 7sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 14smaster passed
+1 💚compile0m 39smaster passed
+1 💚shadedjars3m 52sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 13sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars3m 48spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25sthe patch passed
_ Other Tests _
+1 💚unit215m 56shbase-server in the patch passed.
232m 47s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#4806
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 49f9e8147af7 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 23a5633
Default JavaTemurin-1.8.0_345-b01
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/testReport/
Max. process+thread count2465 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4806/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani
virajjasani commented Sep 29, 2022
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java
// read successfully.
LOG.debug("Encountered a malformed edit, seeking back to last good position in file, "
+ "from {} to {}", inputStream.getPos(), originalPosition, eof);
+ "from {} to {}", inputStream.getPos(), originalPosition);

@virajjasanivirajjasaniSep 29, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Apache9 how about this change (line 447)? This change can be done if not the one above where we are throwing EOF (line 430-431)?

@apurtell

apurtell commented Oct 5, 2022

Copy link
Copy Markdown
Contributor

This was -1ed by @Apache9 , so closing it

@apurtellapurtell closed this Oct 5, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 requested changes
@apurtellapurtellapurtell approved these changes
@wchevreuilwchevreuilAwaiting requested review from wchevreuil

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@virajjasani@Apache-HBase@Apache9@apurtell