[SPARK-22900] [Spark-Streaming] Remove unnecessary restrict for streaming dynamic allocation

[sharkdtu]

What changes were proposed in this pull request?

When i set the conf spark.streaming.dynamicAllocation.enabled=true, the conf num-executors can not be set. As a result, it will allocate default 2 executors and all receivers will be run on this 2 executors, there may not be redundant cpu cores for tasks. it will stuck all the time.

in my opinion, we should remove unnecessary restrict for streaming dynamic allocation. we can set num-executors and spark.streaming.dynamicAllocation.enabled=true together. when application starts, each receiver will be run on an executor.

How was this patch tested?

Manual test.

[felixcheung]

shouldn't you use spark.streaming.dynamicAllocation.minExecutors?

[sharkdtu]

@felixcheung
At the beginning, if numReceivers > totleExecutorCores, there is not cpu cores for batch processing, and ExecutorAllocationManager can't listen metrics of any batches. As a result, it doesn't work.

[felixcheung]

hmm, that sounds like a different problem, why is numReceivers set to > spark.cores.max?

[sharkdtu]

@felixcheung
if you submit spark on yarn with spark.streaming.dynamicAllocation.enabled=true, the num-executors can not be set. So, at the begining, there are only 2(default value) executors.

[jerryshao]

Sorry to chime in. This feature (streaming dynamic allocation) is obsolete and has bugs, users seldom enabled this feature, does it still worth to fix?

[sharkdtu]

@jerryshao
if this PR can fix bugs as you said. why not fix it. Or, it should be marked as deprecated.

[jerryshao]

I'm not against the fix. My concern is that we've shifted to structured streaming, also this feature (streaming dynamic allocation) is seldom used/tested, this might not be the only issue regarding to it (in dynamic allocation we updated a lot), do we still need to put effort on it?

Just my concern.

[felixcheung]

not saying about this change, but I've used streaming dynamic allocation quite a bit back in the day.

but in this case I think simply is to set spark.streaming.dynamicAllocation.minExecutors instead of num-executors as pointed out above.

[jerryshao]

Originally in Spark dynamic allocation, "spark.executor.instances" and dynamic allocation conf cannot be co-existed, if "spark.executor.instances" is set, dynamic allocation will not be enabled. But this behavior is changed after 2.0.

I think here for streaming dynamic allocation, we'd better keep it consistent with Spark dynamic allocation.

[felixcheung]

hmm, I didn't know that was changed actually (SPARK-13723)
But it seems to me spark.streaming.dynamicAllocation.minExecutors is still a valid approach. To match the non-streaming behavior would be a bigger change that I'd agree not sure we should take at this point.

[sharkdtu]

@felixcheung
Have you ever thought about initial num-executors? Actually, it is default 2 executors when you run spark on yarn. How can you make sure that this 2 executors have enougth cores for receivers at the begining?

[AmplabJenkins]

Can one of the admins verify this patch?

[vc60er]

by set spark.streaming.dynamicAllocation.minExecutors also has same issue .https://issues.apache.org/jira/browse/SPARK-14788

@felixcheung