Uh oh!
There was an error while loading. Please reload this page.
HBASE-25556 Frequent replication "Encountered a malformed edit" warnings - #2965
Merged
Conversation
apurtellforce-pushed
the
HBASE-25556
branch
2 times, most recently
from
February 19, 2021 22:55
c226007 to
5195981CompareApache-HBase
commented
Feb 19, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 20, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 20, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 20, 2021
🎊 +1 overall
This message was automatically generated. |
busbey
reviewed
Feb 20, 2021
| initCause(realEofEx != null ? realEofEx : ex); | ||
| } | ||
| if (trailerPresent && this.inputStream.getPos() > this.walEditsStopOffset) { | ||
| LOG.error( |
Contributor
There was a problem hiding this comment.
This particular message should only happen if the read was expected to succeed because the trailer presence means the wal was closed. Is this specific message showing up in the false alarm?
Apache-HBase
commented
Feb 20, 2021
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Feb 20, 2021
🎊 +1 overall
This message was automatically generated. |
apurtell
commented
Feb 20, 2021
| // Else restore our position to original location in hope that next time through we will | ||
| // read successfully. | ||
| LOG.warn("Encountered a malformed edit, seeking back to last good position in file, " | ||
| LOG.debug("Encountered a malformed edit, seeking back to last good position in file, " |
ContributorAuthor
There was a problem hiding this comment.
@busbey This is the frequent message. The particular EOFexception it prints originates from
Contributor
There was a problem hiding this comment.
Yeah I agree that's not useful.
busbey
commented
Feb 20, 2021
Contributor
I'm +1 with the one log statement I commented on kept at error and the rest moved to debug. |
ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below.
Apache-HBase
commented
Feb 23, 2021
🎊 +1 overall
This message was automatically generated. |
apurtell
commented
Feb 23, 2021
ContributorAuthor
If I am reading Sean's comment correctly with the change I pushed we are good to go. Will merge in a few hours unless objection. |
Apache-HBase
commented
Feb 23, 2021
💔 -1 overall
This message was automatically generated. |
asfgit pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
asfgit pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
asfgit pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
asfgit pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
Apache-HBase
commented
Feb 23, 2021
🎊 +1 overall
This message was automatically generated. |
wchevreuil pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Feb 24, 2021
…ngs (apache#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
wchevreuil pushed a commit
to wchevreuil/hbase
that referenced
this pull request
May 24, 2021
…ngs (apache#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org> (cherry picked from commit ecdc942) Change-Id: I5181c60fc8d97586e2be77909cd923df37c94325
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as a few times per minute, and this will cause false positive assessment from operators that they are experiencing a bug.
Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below.