Reduce MockedCANCoderTest Fail Rate - #82
Closed
CoolSpy3 wants to merge 1 commit into
Closed
Conversation
brettle
requested changes
Apr 29, 2024
brettle
left a comment
Member
There was a problem hiding this comment.
Thanks for pointing out this issue! My preference would be the fix I just submitted as PR #83 because it more accurately conveys the test's intent. If you are ok with that, please confirm that PR works for you with NUM_REPS = 10000 (it does for me on Linux) and then approve/merge it (with NUM_REPS=1) and close this PR. If not, I'm happy to discuss.
Member
Author
|
Agreed 👍 Closing as resolved by #83. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I was running into a bug where
MockedCANCoderTestwas failing semi-regularly. This PR introduces three additional waits to reduce this occurrence. I tried a couple configurations.Adding a check after setting the position to
0and performing two waits before each check seemed to be the best combination of functionality and clarity.(Note: From my testing, it was not clear that increasing the timeout parameter or changing the location of the wait calls had a significant effect on the failure rate.)
Original Failure Rate:
18/100(The denominator is not a typo. This number is from a test 10x smaller than the others.)Failure rate after adding wait on line 34:
38/1000Failure after adding waits on lines 34 and 44:
1/1000Failure after adding waits on lines 34, 44, and 35:
2/1000These last two results both have a
<1%failure rate, so they are basically identical. I kept the extra wait in because it's clearer just to perform two waits before reading any data. I tried to run some more detailed tests on them, but they didn't return reasonable results.