[SPARK-25516][k8s]Utilities needed for spark-history and spark-shuffle-service

[BYondRAK]

Summary

When attempting to run ./sbin/start-shuffle-service.sh the image errors with the following output:
ps: unrecognized option: p
Adding the procps package fixes this issue.
After correcting the issue with ps, the code will then error with:
nohup: can't execute '--': No such file or directory
This is remedied with the inclusion of coreutils to the apk add statement.

The ./sbin/start-history-server.sh also requires the added packages to successfully start.

What changes were proposed in this pull request?

Adding procps and coreutils to enable this image to execute the spark-history and spark-shuffle-service shell scripts.

How was this patch tested?

Tested via docker manually

[AmplabJenkins]

Can one of the admins verify this patch?

[dongjoon-hyun]

Thank you for your first contribution, @BYondRAK .

• Could you file an Apache Spark JIRA issue for this?
• Could you update the PR title like [SPARK-?????][K8S] Add procps and coreutils for spark-history and spark-shuffle-service with your JIRA ID (instead of ?????)?

[dongjoon-hyun]

cc @liyinan926

[BYondRAK]

Created an issue and updated as requested.

[liyinan926]

The Kubernetes mode doesn't yet have a shuffle service nor support dynamic resource allocation. I don't know what the script you referred to is for, but it's not for starting a shuffle service to be used with the Kubernetes scheduler backend.

[BYondRAK]

Shuffle Service - Attempting to establish a standalone shuffle service via daemonset to allow for dynamic allocation when using the kubernetes master. When I try to run the shuffle service via that script as entrypoint, the errors above prevent that.

Spark History Server - Attempting to collect in a common location the spark events that will be written after passing the pod information to spark using kubernetes master.

Trying this is what led me to the issue. I'd like to use the image generated via spark documentation if possible.

[liyinan926]

Shuffle Service - Attempting to establish a standalone shuffle service via daemonset to allow for dynamic allocation when using the kubernetes master. When I try to run the shuffle service via that script as entrypoint, the errors above prevent that.

OK, this image is for the kubernetes scheduler backend that runs executors natively in k8s pods. The image is not supposed to be used in the way as described above. I assume the script your referred to is for starting the shuffle service for a standalone cluster. Yes, you can deploy a standalone Spark cluster on k8s. However, this is not the image you should be using for doing that.

[BYondRAK]

OK, thanks for the explanation.

[dongjoon-hyun]

Given the explanation, can we close this PR?

However, this is not the image you should be using for doing that.

[BYondRAK]

sure thing

[dongjoon-hyun]

Thanks, @BYondRAK .