Uh oh!
There was an error while loading. Please reload this page.
[SPARK-17235][SQL] Support purging of old logs in MetadataLog - #14802
[SPARK-17235][SQL] Support purging of old logs in MetadataLog#14802petermaxlee wants to merge 1 commit into
Conversation
petermaxlee
commented
Aug 25, 2016
| def testManager(basePath: Path, fm: FileManager): Unit = { | ||
| /** Basic test case for [[FileManager]] implementation. */ | ||
| private def testFileManager(basePath: Path, fm: FileManager): Unit = { |
There was a problem hiding this comment.
I renamed this because initially I thought it's a noun meaning "manager for testing", rather than "to test the file manager".
jerryshao
commented
Aug 25, 2016
Looks like this is a little similar to this one #13513 . |
SparkQA
commented
Aug 25, 2016
Test build #64403 has finished for PR 14802 at commit
|
zsxwing
commented
Aug 26, 2016
It would be great if we can reuse codes in |
petermaxlee
commented
Aug 26, 2016
@zsxwing yup I plan to consolidate them. |
zsxwing
commented
Aug 26, 2016
@petermaxlee mind to submit the consolidation PR instead when you finish? |
petermaxlee
commented
Aug 26, 2016
I can but I'm doing a lot of work in this area and it is a lot more difficult since they have dependencies. It would be better to merge the logically atomic pull requests. |
frreiss
commented
Aug 26, 2016
LGTM. I have written nearly the exact same thing as part of [https://github.com//pull/14553], but can use this version of the method instead. |
rxin
commented
Aug 26, 2016
Alright I'm going to merge this in master/2.0. @petermaxlee and @frreiss can you guys work together? |
## What changes were proposed in this pull request? This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge function is currently unused, but I will use it to purge old execution and file source logs in follow-up patches. These changes are required in a production structured streaming job that runs for a long period of time. ## How was this patch tested? Added a unit test case in HDFSMetadataLogSuite. Author: petermaxlee <petermaxlee@gmail.com> Closes#14802 from petermaxlee/SPARK-17235. (cherry picked from commit f64a1dd) Signed-off-by: Reynold Xin <rxin@databricks.com>
What changes were proposed in this pull request?
This patch adds a purge interface to MetadataLog, and an implementation in HDFSMetadataLog. The purge function is currently unused, but I will use it to purge old execution and file source logs in follow-up patches. These changes are required in a production structured streaming job that runs for a long period of time.
How was this patch tested?
Added a unit test case in HDFSMetadataLogSuite.