Uh oh!
There was an error while loading. Please reload this page.
HBASE-26947 Implement a special TestAppender to limit the size of tes… - #4340
Merged
Conversation
Apache9
commented
Apr 12, 2022
ContributorAuthor
It is not easy to write a UT for the log4j2 appender so I just test it manually. Change the 'appender.console.maxSize' property in log4j2.properties to a very small value, for example, 1K, you will soon see this output when running UT in IDE So I think this approach works. And I set default value to 1G, which should be enough for most cases... @busbey@ndimiduk FYI. I think this could solve the recent space problem of our ci system. |
Apache-HBase
commented
Apr 12, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 12, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 12, 2022
💔 -1 overall
This message was automatically generated. |
busbey
reviewed
Apr 12, 2022
Comment on lines
+145
to
+166
| public void append(LogEvent event) { | ||
| if (stop.get()) { | ||
| return; | ||
| } | ||
| // for accounting, here we always convert the event to byte array first | ||
| // this will effect performance a bit but it is OK since this is for UT only | ||
| byte[] bytes = getLayout().toByteArray(event); | ||
| if (bytes == null || bytes.length == 0) { | ||
| return; | ||
| } | ||
| long sizeAfterAppend = size.addAndGet(bytes.length); | ||
| if (sizeAfterAppend >= maxSize) { | ||
| // stop logging if the log size exceeded the limit | ||
| if (stop.compareAndSet(false, true)) { | ||
| LOGGER.error("Log size exceeded the limit {}, will stop logging to prevent eating" | ||
| + " too much disk space", maxSize); | ||
| } | ||
| return; | ||
| } | ||
| super.append(event); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Can we abort the test in this case rather than having the test continue without logs?
ContributorAuthor
There was a problem hiding this comment.
Throw an Error out maybe? In general I'm not sure if this could work, because it could be a thread other than the main thread which enters here and exceeds the limit, fail this thread can not abort the test?
Member
There was a problem hiding this comment.
I've done this in another logging framework but not log4j2. This is appending events to a single log line, building it up, is that correct? I think it's fine if we truncate log lines at a certain length, no need to error or whatever, just terminate the message. I'm a little surprised that this not available to use as a configuration option.
In fact, the pattern layout does support it. https://stackoverflow.com/questions/27502536/limit-max-message-size-in-log4j2-pattern
Member
There was a problem hiding this comment.
Or maybe it's one log line per Event object. I see.
I'm fine with this. I think it's better to let the test finish than to kill the build when the log limit is exceeded. In my experience, it's relatively rare that logs from CI are used by anyone. This strategy for logging once max log volume for a test is exceeded is fine by me.
What happens when the appender is for some class that's used over and over in test? There's no way to reset this counter to be a per-test-class accumulator, right?
ContributorAuthor
There was a problem hiding this comment.
We do not use reuseForks so this is not a problem. Every test will have its own process.
Member
There was a problem hiding this comment.
We do not use reuseForks so this is not a problem. Every test will have its own process.
In that case, it would be better to add an assertion that this invariant is held. With that, I'm +1.
Member
There was a problem hiding this comment.
If asserting on the invariant is impossible, I'm okay with adding a comment here so that at least a future change to test execution will have a breadcrumb to follow.
ContributorAuthor
There was a problem hiding this comment.
I do not know how to add the assertion in code to test a pom config :(
But anyway, we could add a comment about this, both in the appender code and also in pom.
ContributorAuthor
There was a problem hiding this comment.
I've tried to use RunListener to reset the counter when starting a new test but failed... The behavior is really confusing, I mean when will the methods in the RunListener be called, and when will the surefire plugin rotate the output tests when reuseForks is true...
So I just add some comments to pom.xml to say that why we can not set reuseForks to true.
PTAL. @ndimiduk
Apache-HBase
commented
Apr 12, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 12, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 12, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 13, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 13, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 13, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 14, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 14, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 14, 2022
💔 -1 overall
This message was automatically generated. |
Apache9
commented
Apr 15, 2022
ContributorAuthor
Apache-HBase
commented
Apr 19, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 19, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 19, 2022
💔 -1 overall
This message was automatically generated. |
Apache9
commented
Apr 20, 2022
ContributorAuthor
Ping @ndimiduk |
ndimiduk
approved these changes
Apr 21, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…t output