Uh oh!
There was an error while loading. Please reload this page.
Add the validation of spark.cores.max under Streaming - #18948
Conversation
AmplabJenkins
commented
Aug 15, 2017
Can one of the admins verify this patch? |
srowen
commented
Aug 15, 2017
I don't think that's valid. It doesn't catch all cases where you need > 1 core, and isn't necessary for cases that don't involve receivers, I think. |
SOmeONee
commented
Aug 15, 2017
To example NetworkWordCount, if run with '--total-executor-cores 1', it can' be success. |
srowen
commented
Aug 15, 2017
That's not the problem -- I'm saying there are cases where 1 is valid. There are obviously cases where it isn't. But your change makes them all fail. |
SOmeONee
commented
Aug 15, 2017
Ok, sorry, I'ill review it, thanks. |
| } | ||
| if (sc.conf.contains("spark.cores.max")) { | ||
| val totalCores = sc.conf.getInt("spark.cores.max", 1) |
There was a problem hiding this comment.
AFAIK, "spark.cores.max" is not set by default, with your change it will always return 1 if not set and throws exception as you wrote.
Besides if there're several receivers in streaming application, even > 1 is not sufficient. So simply checking "spark.cores.max" is not a feasible way to address the issue, and there may not be a good way to handle this problem properly, so I'd suggest to leave the current code as it is.
There was a problem hiding this comment.
The config spark.cores.max is used to limit the max number of cores that a single executor can require, and as @jerryshao pointed out, it's not set by default. I'm not convinced that validating the config can benefit your issue.
There was a problem hiding this comment.
@jerryshao spark.cores.max will be returned only when conf contain it.
Here is the biggest possibility to make a judgment.
There was a problem hiding this comment.
@SOmeONee I'm not quite following your comment here. You assume that default value is "1" which changes the semantics of this configuration. What's more, checking whether "<=1" is not so sufficient as I mentioned above.
There was a problem hiding this comment.
@jiangxb1987 "spark.cores.max" is per application configuration to limit the numbers of cores can be requested for this application, it is not a per executor limitation.
The config spark.cores.max is used to limit the max number of cores that a single executor can require
So still if we have 2 receivers in one streaming application, the minimum number should > 2, checking "1" here is still not feasible.
Since receiver number can only be gotten in run-time, checking configuration will not be worked as expected.
There was a problem hiding this comment.
Yea, seems the validation here is not very useful.
SOmeONee
commented
Aug 17, 2017
@srowen I reviewed again and run the tests, there are no test cases with spark.cores.max=1 in streaming |
jerryshao
commented
Sep 8, 2017
The patch here is not solid, we will not merge it unless you have better solution. |
Closesapache#18522Closesapache#17722Closesapache#18879Closesapache#18891Closesapache#18806Closesapache#18948Closesapache#18949Closesapache#19070Closesapache#19039Closesapache#19142Closesapache#18515Closesapache#19154Closesapache#19162Closesapache#19187
Closesapache#18522Closesapache#17722Closesapache#18879Closesapache#18891Closesapache#18806Closesapache#18948Closesapache#18949Closesapache#19070Closesapache#19039Closesapache#19142Closesapache#18515Closesapache#19154Closesapache#19162Closesapache#19187Closesapache#19091 Author: Sean Owen <sowen@cloudera.com> Closesapache#19203 from srowen/CloseStalePRs3.
What changes were proposed in this pull request?
By using spark streaming, --total-executor-cores must greater than 1.
This pull requst add the validation of spark.cores.max.
How was this patch tested?
manual tests:
--total-executor-cores 1
--total-executor-cores 2
Please review http://spark.apache.org/contributing.html before opening a pull request.