Uh oh!
There was an error while loading. Please reload this page.
[SPARK-25818][CORE] WorkDirCleanup should only remove the directory at the beginning of the app - #22813
[SPARK-25818][CORE] WorkDirCleanup should only remove the directory at the beginning of the app#22813ouyangxiaochen wants to merge 1 commit into
Conversation
AmplabJenkins
commented
Oct 24, 2018
Can one of the admins verify this patch? |
IIUC it's not expected to share the SPARK_WORK_DIR with any other usage. Does this happen in real environment and hard to bypass the limitation? |
@jiangxb1987 Yes, it happened in our real environment. |
srowen
commented
Oct 27, 2018
I don't think that's a reasonable usage scenario. That said is there any harm to this change? would it ever miss cleaning something up that it should? |
dongjoon-hyun
commented
Oct 29, 2018
@ouyangxiaochen . Sorry, but the use case sounds like a misconfiguration. |
ouyangxiaochen
commented
Oct 29, 2018
As far as I know, when a spark program is submitted to the cluster, a directory will be created under |
ouyangxiaochen
commented
Oct 29, 2018
@dongjoon-hyun Yes, you can think so. So I want to solve this problem on the spark platform to reduce the risk of some misoperations of operation and maintenance engineer. |
srowen
commented
Oct 29, 2018
I don't know what else goes in the work dir. It isn't valid to reuse it for anything else. Can you simply avoid using a work dir that is or has been used by something else? The argument for making this change anyway is just that the code should delete just what it writes. But I am just not sure it can be something you rely on for correct behavior |
ouyangxiaochen
commented
Nov 1, 2018
@srowen Your suggestion is very good, but sometimes the maintenance engineers have limited skills in this area. If they configures the operating system root directory as |
srowen
commented
Nov 1, 2018
I just think that if you have engineers randomly writing and reading stuff in this dir, a bunch of other stuff goes wrong. This is not a problem that Spark can reasonably solve. Certainly, you have much bigger production problems if this level of discipline can't be enforced. |
Closesapache#21766Closesapache#21679Closesapache#21161Closesapache#20846Closesapache#19434Closesapache#18080Closesapache#17648Closesapache#17169 Add: Closesapache#22813Closesapache#21994Closesapache#22005Closesapache#22463 Add: Closesapache#15899 Add: Closesapache#22539Closesapache#21868Closesapache#21514Closesapache#21402Closesapache#21322Closesapache#21257Closesapache#20163Closesapache#19691Closesapache#18697Closesapache#18636Closesapache#17176Closesapache#23001 from wangyum/CloseStalePRs. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: hyukjinkwon <gurwls223@apache.org>
What changes were proposed in this pull request?
The cleanup mechanism will clear all the eligible directories under SPARK_WORK_DIR. If the other configured paths are the same as the SPARK_WORK_DIR configuration, this will cause the file directories of other configuration items to be deleted by mistake. For example, the SPARK_LOCAL_DIRS and SPARK_WORK_DIR settings are the same.
We should add an another condition which start with 'app-' when removing the expired app-* directories in SPARK_WORK_DIR
How was this patch tested?
manual test