Uh oh!
There was an error while loading. Please reload this page.
[SPARK-20407][TESTS] ParquetQuerySuite 'Enabling/disabling ignoreCorruptFiles' flaky test - #17701
[SPARK-20407][TESTS] ParquetQuerySuite 'Enabling/disabling ignoreCorruptFiles' flaky test#17701bogdanrdc wants to merge 4 commits into
Conversation
SparkQA
commented
Apr 20, 2017
Test build #75983 has started for PR 17701 at commit |
HyukjinKwon
commented
Apr 20, 2017
Do you know any Jenkins build where this test was failed? |
bogdanrdc
commented
Apr 20, 2017
Fails rarely in our own jenkins. The test I added is just part of an existing test that is run 100 times, that one fails reliably. |
| * this is part of test 'Enabling/disabling ignoreCorruptFiles' but run in a loop | ||
| * to increase the chance of failure | ||
| */ | ||
| test("SPARK-20407 ParquetQuerySuite 'Enabling/disabling ignoreCorruptFiles' flaky test") { |
There was a problem hiding this comment.
This might take a while to complete. Shall we ignore this? It is still useful to have for reference though.
| DebugFilesystem.assertNoOpenStreams() | ||
| // files can be closed from other threads, so wait a bit | ||
| // normally this doesn't take more than 1s | ||
| eventually(timeout(10.seconds)) { |
There was a problem hiding this comment.
This is a catch all for when we are not using withTempDir right?
There was a problem hiding this comment.
this is so that DebugFilesystem.assertNoOpenStreams has a better chance of working. And in general, whatever else runs at the end of the test. For now it's just assertNoOpenStreams
hvanhovell
commented
Apr 20, 2017
A few minor comments, otherwise LGTM. |
SparkQA
commented
Apr 20, 2017
Test build #75991 has finished for PR 17701 at commit
|
…uptFiles' flaky test ## What changes were proposed in this pull request? SharedSQLContext.afterEach now calls DebugFilesystem.assertNoOpenStreams inside eventually. SQLTestUtils withTempDir calls waitForTasksToFinish before deleting the directory. ## How was this patch tested? Added new test in ParquetQuerySuite based on the flaky test Author: Bogdan Raducanu <bogdan@databricks.com> Closes#17701 from bogdanrdc/SPARK-20407. (cherry picked from commit c5a31d1) Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>
hvanhovell
commented
Apr 20, 2017
Merging to master/branch-2.2. Thanks! |
…uptFiles' flaky test ## What changes were proposed in this pull request? SharedSQLContext.afterEach now calls DebugFilesystem.assertNoOpenStreams inside eventually. SQLTestUtils withTempDir calls waitForTasksToFinish before deleting the directory. ## How was this patch tested? Added new test in ParquetQuerySuite based on the flaky test Author: Bogdan Raducanu <bogdan@databricks.com> Closesapache#17701 from bogdanrdc/SPARK-20407.
What changes were proposed in this pull request?
SharedSQLContext.afterEach now calls DebugFilesystem.assertNoOpenStreams inside eventually.
SQLTestUtils withTempDir calls waitForTasksToFinish before deleting the directory.
How was this patch tested?
Added new test in ParquetQuerySuite based on the flaky test