[SPARK-26588][CORE] Idle executor should properly be removed when no job is su…

[TopGunViper]

What changes were proposed in this pull request?

I enable dynamic allocation feature with spark-shell and do not submit any task. After spark.dynamicAllocation.executorIdleTimeout seconds(default 60s), there is still one active executor, which is abnormal. All idle executors are timeout and should be removed.(default spark.dynamicAllocation.minExecutors=0). The spark-shell command show below:

spark-shell --master=yarn --conf spark.ui.port=8040 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.maxExecutors=8 --conf spark.dynamicAllocation.initialExecutors=4 --conf spark.shuffle.service.enabled=true

How was this patch tested?

unit tests

[srowen]

CC @jerryshao

[SparkQA]

Test build #4502 has finished for PR 23509 at commit bd300d6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Can you please describe how to reproduce your problem?

[TopGunViper]

Can you please describe how to reproduce your problem?

Execute the following command and waiting for 60 seconds. The problem can be observed from Spark Web UI under the Executors Tab.

spark-shell --master=yarn --conf spark.ui.port=8040 --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.maxExecutors=8 --conf spark.dynamicAllocation.initialExecutors=4 --conf spark.shuffle.service.enabled=true

[jerryshao]

1. please fix the PR title to follow Spark's way. 2) Can you please figure out which patch introduce this regression? I did remember that this should be OK in my previous test.

[TopGunViper]

1. please fix the PR title to follow Spark's way. 2) Can you please figure out which patch introduce this regression? I did remember that this should be OK in my previous test.

Hi @jerryshao I guess this problem was introduced by the patch SPARK-21656.

[TopGunViper]

CC @tgravescs

[jerryshao]

Jenkins, retest this please.

[jerryshao]

I can reproduce this issue, will spend time on it. Thanks.

[SparkQA]

Test build #101223 has finished for PR 23509 at commit bd300d6.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[jerryshao]

Jenkins, retest this please.

[jerryshao]

The change itself looks fine to me. @tgravescs@vanzin would you please also take a look.

[SparkQA]

Test build #101284 has finished for PR 23509 at commit bd300d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Already fixed by SPARK-26758.