[SPARK-23529][K8s] Support mounting hostPath volumes by madanadit · Pull Request #21095 · apache/spark

madanadit · 2018-04-18T07:32:15Z

What changes were proposed in this pull request?

This PR introduces a new config spark.kubernetes.driver/executor.volumes taking a values of the format documented here

The use case is to enable short-circuit writes to distributed storage on k8s. The Alluxio File System uses domain sockets to enable short-circuit writes from the client to worker memory when co-located on the same host machine. A directory, lets say /tmp/domain on the host, is mounted on the Alluxio worker container as well as the Alluxio client ( = Spark executor) container. The worker creates a domain socket /tmp/domain/d and if the client container mounts the same directory, it can write directly to the Alluxio worker w/o passing through network stack. The end result is faster data access when data is local.

How was this patch tested?

Manual testing on a k8s v1.8 cluster. Unit tests added to Driver/ExecutorPodFactorySuite.

This PR replaces #21032.

madanadit · 2018-04-18T07:33:31Z

@liyinan926 @foxish Please take a look

foxish · 2018-04-18T19:17:26Z

jenkins, ok to test

foxish · 2018-04-18T19:19:12Z

@madanadit, thanks for following up with the PR and for your design doc. I had a few comments on the doc for further discussion, but in general, I think this is a good direction.

SparkQA · 2018-04-18T19:33:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2398/

SparkQA · 2018-04-18T19:38:11Z

Test build #89529 has finished for PR 21095 at commit 7c1be8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-18T19:39:12Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2398/

madanadit · 2018-04-18T21:03:50Z

Thanks @foxish for your feedback. As a first time contributor to Spark, I would like to limit the scope of the changes in this PR. Let me know when you're ready to review again.

liyinan926 · 2018-04-19T19:36:24Z

My general comment is the code is centered around parsing and setting up hostPath volumes even though the proposal is generic. I would suggest making the code generic as well. Particularly I would like to see an implementation that is able to parse all volume related properties and add all discovered volumes in one shot.

madanadit · 2018-04-20T00:01:00Z

@liyinan926 Thanks for the comment. I refactored it to address your concern. The code is simple enough to be easily refactored if the need be by whoever implements the next volume type.

Note that the goal of this PR is limited to hostPath volumes. Even though the implementation here is general enough to be extended, I will not attempt to add other volume types in this PR.

SparkQA · 2018-04-20T00:11:28Z

Test build #89601 has finished for PR 21095 at commit f482dfc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-04-20T00:13:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2450/

SparkQA · 2018-04-20T00:19:06Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/2450/

liyinan926 · 2018-04-20T17:38:14Z

Thanks @madanadit! I will take a look over the weekend or early net week.

madanadit · 2018-04-26T20:35:35Z

Hey @liyinan926, do you think you'll have time to take a look this week?

foxish · 2018-04-26T20:44:20Z

@madanadit, sorry about the long turnaround time here. I made a cursory pass over it - and it looks good. Before getting deeper into the code, I think we need more exhaustive unit testing for non-hostpath type volumes, and a couple of e2e tests. Once those are done, I'm also happy to do a more detailed review here.

liyinan926 · 2018-04-26T21:23:12Z

@madanadit sorry I didn't get a chance to look into this. I will take a detailed look once @foxish's comments on testing are addressed.

madanadit · 2018-04-30T23:49:20Z

Hi @foxish, I don't see why the 2 testing concerns should block reviewing this PR.

This PR does not attempt to address non-hostpath volumes (both implementation and unit tests are hence not included).
I agree e2e tests do make sense. However, e2e tests should not block this PR in any way. e2e tests can also be added once the PR is reviewed?

foxish · 2018-05-01T00:24:40Z

The testing is not a blocker for the review. When I said "tests for non-hostpath type volumes", I meant to say that we want to cover more than just hostpath mounts with the initial PR - because we might end up with something too specific. Sorry if that wasn't clear from my comment. I think doing EmptyDir mounts alongside hostpath (with as much code sharing as possible) would be a good idea since it's the most common volume type. With those two, I think we can push this forward and have the e2e come along side by side.

liyinan926 · 2018-05-03T23:09:31Z

+ * @param prefix the prefix for volume configuration
+ * @return a tuple of (pod with the volume(s) added, container with mount(s) added)
+ */
+ def addVolumes(


The Scaladoc should not mention hostPath as this function is not hostPath exclusively.

liyinan926 · 2018-05-03T23:13:08Z

+ }
+ // Populate spec
+ volumes.foreach {
+ case (name, spec) =>


The case line should be merged into the previous line according to the Spark code convention, e.g., volumes.foreach { case (name, spec) =>.

liyinan926 · 2018-05-03T23:13:16Z

+ volumes.foreach {
+ case (name, spec) =>
+ properties.foreach {
+ k =>


liyinan926 · 2018-05-03T23:14:01Z

+ val podBuilder = new PodBuilder(pod).editOrNewSpec()
+ val containerBuilder = new ContainerBuilder(container)
+ volumes foreach {
+ case (name, spec) =>


liyinan926 · 2018-05-03T23:16:36Z

+ if (spec.optionsSpec.contains(KUBERNETES_VOLUMES_PATH_KEY)) {
+ hostPath = Some(spec.optionsSpec(KUBERNETES_VOLUMES_PATH_KEY))
+ }
+ if (hostPath.isDefined && spec.mountPath.isDefined) {


Looks like this if block can be combined with the previous if block, e.g., if (spec.mountPath.isDefined && spec.optionsSpec.contains(KUBERNETES_VOLUMES_PATH_KEY)) {...}.

liyinan926 · 2018-05-03T23:19:00Z

+ .withHostPath(new HostPathVolumeSource(hostPath.get))
+ .withName(name)
+ .build())
+ val volumeBuilder = new VolumeMountBuilder()


s/volumeBuilder/mountBuilder.

liyinan926 · 2018-05-04T02:20:14Z

+ * @param prefix the given property name prefix
+ * @return a Map storing with volume name as key and spec as value
+ */
+ def parseHostPathVolumesWithPrefix(


Looks like you don't really need this function as it's just a wrapper of parseVolumesWithPrefix .

liyinan926 · 2018-05-04T02:25:38Z

 .build()
- SparkPod(driverPod, driverContainer)
+
+ val (driverPodWithVolumes, driverContainerVolumes) =


Instead of putting the logic of volume mounting in BasicDriverFeatureStep and BasicExecutorFeatureStep, we should add a new step for mounting volumes, similarly to how we handle secrets, e.g., MountVolumesFeatureStep where the logic of addVolumes should be. This feature step can be used for both the driver and executors.

andrusha · 2018-05-07T17:17:18Z

@madanadit @liyinan926 @foxish I addressed comments here and added PVC support #21260. However I'm unsure if this is the right way to go, please check the PR.

liyinan926 · 2018-05-07T18:26:52Z

@andrusha Is the purpose of #21260 to replace both this PR and #21238?

andrusha · 2018-05-08T09:31:49Z

@liyinan926 that's the idea. We should have a single step to mount all kinds of volumes including hostPath and emptyDir. The only problem is the configuration awkwardness.

madanadit · 2018-05-08T17:31:52Z

Thanks @liyinan926 for the review and @andrusha for addressing the comments and moving the work along. I'll close this PR once #21260 merges.

AmplabJenkins · 2018-06-09T00:10:49Z

Can one of the admins verify this patch?

This PR continues #21095 and intersects with #21238. I've added volume mounts as a separate step and added PersistantVolumeClaim support. There is a fundamental problem with how we pass the options through spark conf to fabric8. For each volume type and all possible volume options we would have to implement some custom code to map config values to fabric8 calls. This will result in big body of code we would have to support and means that Spark will always be somehow out of sync with k8s. I think there needs to be a discussion on how to proceed correctly (eg use PodPreset instead) ---- Due to the complications of provisioning and managing actual resources this PR addresses only volume mounting of already present resources. ---- - [x] emptyDir support - [x] Testing - [x] Documentation - [x] KubernetesVolumeUtils tests Author: Andrew Korzhuev <andrew.korzhuev@klarna.com> Author: madanadit <adit@alluxio.com> Closes#21260 from andrusha/k8s-vol.

liyinan926 · 2018-07-13T20:52:53Z

@madanadit can you close this as #21260 has been merged?

Closesapache#17422Closesapache#17619Closesapache#18034Closesapache#18229Closesapache#18268Closesapache#17973Closesapache#18125Closesapache#18918Closesapache#19274Closesapache#19456Closesapache#19510Closesapache#19420Closesapache#20090Closesapache#20177Closesapache#20304Closesapache#20319Closesapache#20543Closesapache#20437Closesapache#21261Closesapache#21726Closesapache#14653Closesapache#13143Closesapache#17894Closesapache#19758Closesapache#12951Closesapache#17092Closesapache#21240Closesapache#16910Closesapache#12904Closesapache#21731Closesapache#21095 Added: Closesapache#19233Closesapache#20100Closesapache#21453Closesapache#21455Closesapache#18477 Added: Closesapache#21812Closesapache#21787 Author: hyukjinkwon <gurwls223@apache.org> Closesapache#21781 from HyukjinKwon/closing-prs.

madanadit added 17 commits April 16, 2018 15:39

Support mounting hostPath volumes for executors
51a877a

Read mode for mounted volumes
4ada724

Refactor
17258a3

Fix style
6fff716

Add unit tests
f961b33

Update comment
af4d9ba

Fix unit tests
13e1049

Fix typo
7a25d76

Change configuration format
a5a277a

Fix build
772128e

Fix test
a393b92

Fetch properties correctly
b9b3dcb

Fix test cases
81a7811

Abstract tests
95d1b0d

Add readOnly option
facde97

Fix test
ccdc799

Driver hostPath volumes with tests
7c1be8a

Refactor
f482dfc

liyinan926 mentioned this pull request Apr 20, 2018
Sync with apache/spark/master kubeflow/spark-operator#129
Closed
8 tasks

liyinan926 suggested changes May 4, 2018
View reviewed changes

This was referenced May 7, 2018
[SPARK-24137][K8s] Mount local directories as empty dir volumes. #21238
Closed
[SPARK-23529][K8s] Support mounting volumes #21260
Closed

HyukjinKwon mentioned this pull request Jul 16, 2018
[INFRA] Close stale PR #21781
Closed

asfgit closed this in 1a4fda8Jul 19, 2018

Uh oh!

Conversation

madanadit commented Apr 18, 2018

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

madanadit commented Apr 18, 2018

Uh oh!

foxish commented Apr 18, 2018

Uh oh!

foxish commented Apr 18, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Apr 18, 2018

Uh oh!

SparkQA commented Apr 18, 2018

Uh oh!

SparkQA commented Apr 18, 2018

Uh oh!

madanadit commented Apr 18, 2018

Uh oh!

liyinan926 commented Apr 19, 2018

Uh oh!

madanadit commented Apr 20, 2018

Uh oh!

SparkQA commented Apr 20, 2018

Uh oh!

SparkQA commented Apr 20, 2018

Uh oh!

SparkQA commented Apr 20, 2018

Uh oh!

liyinan926 commented Apr 20, 2018

Uh oh!

madanadit commented Apr 26, 2018

Uh oh!

foxish commented Apr 26, 2018

Uh oh!

liyinan926 commented Apr 26, 2018

Uh oh!

madanadit commented Apr 30, 2018

Uh oh!

foxish commented May 1, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrusha commented May 7, 2018

Uh oh!

liyinan926 commented May 7, 2018

Uh oh!

andrusha commented May 8, 2018

Uh oh!

madanadit commented May 8, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented Jun 9, 2018

Uh oh!

liyinan926 commented Jul 13, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

foxish commented Apr 18, 2018•
edited
Loading

madanadit commented May 8, 2018•
edited
Loading