Uh oh!
There was an error while loading. Please reload this page.
[SPARK] [Scheduler] Configurable default scheduling mode - #19456
[SPARK] [Scheduler] Configurable default scheduling mode#19456blyncsy-david-lewis wants to merge 1 commit into
Conversation
jiangxb1987
commented
Oct 11, 2017
Could you elaborate on the scenario that you should need to make these settings configurable? |
blyncsy-david-lewis
commented
Oct 11, 2017
I have a multiuser application where I use the userId as the name of the scheduling pool so that users are balanced equally by spark and within a user's workload I can set the scheduling mode to FAIR (or whatever I want). It is unreasonable to specify xml for each user and restart spark, so this patch allows me to specify the default configuration used by undefined pools. |
varunvish910
left a comment
There was a problem hiding this comment.
Can you guarantee that your users share an even workload? With this setup you could potentially have a user with a high workload using 1/#users resources. Wouldn't using the queue feature be better?
AmplabJenkins
commented
Jun 9, 2018
Can one of the admins verify this patch? |
HyukjinKwon
commented
Jul 16, 2018
ping @blyncsy-david-lewis for @varuvish 's comment. Otherwise let me propose to close it for now. |
Closesapache#17422Closesapache#17619Closesapache#18034Closesapache#18229Closesapache#18268Closesapache#17973Closesapache#18125Closesapache#18918Closesapache#19274Closesapache#19456Closesapache#19510Closesapache#19420Closesapache#20090Closesapache#20177Closesapache#20304Closesapache#20319Closesapache#20543Closesapache#20437Closesapache#21261Closesapache#21726Closesapache#14653Closesapache#13143Closesapache#17894Closesapache#19758Closesapache#12951Closesapache#17092Closesapache#21240Closesapache#16910Closesapache#12904Closesapache#21731Closesapache#21095 Added: Closesapache#19233Closesapache#20100Closesapache#21453Closesapache#21455Closesapache#18477 Added: Closesapache#21812Closesapache#21787 Author: hyukjinkwon <gurwls223@apache.org> Closesapache#21781 from HyukjinKwon/closing-prs.
Pulling default values for scheduling mode from spark conf.