Skip to content

[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work#969

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
witgo wants to merge 19 commits into from
Closed

[SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work #969

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8362489
yarn.ClientBase spark.yarn.dist.* do not work
witgo Jun 4, 2014
ce609fc
Merge branch 'master' into yarn_ClientBase
witgo Jun 9, 2014
0961151
merge master
witgo Jun 11, 2014
20e667c
Merge branch 'master' into yarn_ClientBase
witgo Jun 13, 2014
8bc2f4b
review commit
witgo Jun 13, 2014
9cdff16
review commit
witgo Jun 13, 2014
55d72fc
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 13, 2014
35d6fa0
move to ClientArguments
witgo Jun 13, 2014
41bce59
review commit
witgo Jun 13, 2014
871f1db
add spark.yarn.dist.* documentation
witgo Jun 13, 2014
1048549
remove Utils.resolveURIs
witgo Jun 15, 2014
8d7b82f
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 15, 2014
2f48789
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 16, 2014
c8b4554
review commit
witgo Jun 16, 2014
b6a9aa1
merge master
witgo Jun 17, 2014
e3c1107
update docs
witgo Jun 17, 2014
5261b6c
fix sys.props.get("SPARK_YARN_DIST_FILES")
witgo Jun 17, 2014
3bdbc52
Merge branch 'master' of https://github.com/apache/spark into yarn_Cl…
witgo Jun 17, 2014
8117765
review commit
witgo Jun 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 17 additions & 3 deletions docs/running-on-yarn.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -68,15 +68,29 @@ Most of the configs are the same for Spark on YARN as for other deployment modes
</td>
</tr>
<tr>
<td><code>spark.yarn.executor.memoryOverhead</code></td>
<td>384</code></td>
<td><code>spark.yarn.dist.archives</code></td>
<td>(none)</td>
<td>
Comma separated list of archives to be extracted into the working directory of each executor.
</td>
</tr>
<tr>
<td><code>spark.yarn.dist.files</code></td>
<td>(none)</td>
<td>
Comma-separated list of files to be placed in the working directory of each executor.
<td>
</tr>
<tr>
<td><code>spark.yarn.executor.memoryOverhead</code></td>
<td>384</td>
<td>
The amount of off heap memory (in megabytes) to be allocated per executor. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc.
</td>
</tr>
<tr>
<td><code>spark.yarn.driver.memoryOverhead</code></td>
<td>384</code></td>
<td>384</td>
<td>
The amount of off heap memory (in megabytes) to be allocated per driver. This is memory that accounts for things like VM overheads, interned strings, other native overheads, etc.
</td>
Expand Down
15 changes: 13 additions & 2 deletions yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -21,8 +21,7 @@ import scala.collection.mutable.{ArrayBuffer, HashMap}

import org.apache.spark.SparkConf
import org.apache.spark.scheduler.InputFormatInfo
import org.apache.spark.util.IntParam
import org.apache.spark.util.MemoryParam
import org.apache.spark.util.{Utils, IntParam, MemoryParam}


// TODO: Add code and support for ensuring that yarn resource 'tasks' are location aware !
Expand All@@ -45,6 +44,18 @@ class ClientArguments(val args: Array[String], val sparkConf: SparkConf) {

parseArgs(args.toList)

// env variable SPARK_YARN_DIST_ARCHIVES/SPARK_YARN_DIST_FILES set in yarn-client then
// it should default to hdfs://
files = Option(files).getOrElse(sys.env.get("SPARK_YARN_DIST_FILES").orNull)

@tgravescstgravescsJun 13, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to add support for the env variables for yarn-cluster mode. We only support them on yarn-client mode for backwards compatibility. Can you remove this.

@tgravescstgravescsJun 17, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again this is already handled in the YarnClientSchedulerBackend. It reads the env variables and passes in the --files/--archives without being resolveURI extended. The issue with that code is that it also looks at spark.yarn.dist.archives and spark.yarn.dist.files and doesn't resolveURI extend them.

archives = Option(archives).getOrElse(sys.env.get("SPARK_YARN_DIST_ARCHIVES").orNull)

@tgravescstgravescsJun 13, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above comment.


// spark.yarn.dist.archives/spark.yarn.dist.files defaults to use file:// if not specified,
// for both yarn-client and yarn-cluster
files = Option(files).getOrElse(sparkConf.getOption("spark.yarn.dist.files").
map(p => Utils.resolveURIs(p)).orNull)
archives = Option(archives).getOrElse(sparkConf.getOption("spark.yarn.dist.archives").
map(p => Utils.resolveURIs(p)).orNull)

private def parseArgs(inputArgs: List[String]): Unit = {
val userArgsBuffer: ArrayBuffer[String] = new ArrayBuffer[String]()
val inputFormatMap: HashMap[String, InputFormatInfo] = new HashMap[String, InputFormatInfo]()
Expand Down
3 changes: 2 additions & 1 deletion yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -162,7 +162,7 @@ trait ClientBase extends Logging {
val fs = FileSystem.get(conf)
val remoteFs = originalPath.getFileSystem(conf)
var newPath = originalPath
if (!compareFs(remoteFs, fs)) {
if (!compareFs(remoteFs, fs)) {
newPath = new Path(dstDir, originalPath.getName())
logInfo("Uploading " + originalPath + " to " + newPath)
FileUtil.copy(remoteFs, originalPath, fs, newPath, false, conf)
Expand DownExpand Up@@ -250,6 +250,7 @@ trait ClientBase extends Logging {
}
}
}
logInfo("Prepared Local resources " + localResources)
sparkConf.set(ClientBase.CONF_SPARK_YARN_SECONDARY_JARS, cachedSecondaryJarLinks.mkString(","))

UserGroupInformation.getCurrentUser().addCredentials(credentials)
Expand Down
4 changes: 1 addition & 3 deletions ...common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -70,9 +70,7 @@ private[spark] class YarnClientSchedulerBackend(
("--executor-cores", "SPARK_WORKER_CORES", "spark.executor.cores"),
("--executor-cores", "SPARK_EXECUTOR_CORES", "spark.executor.cores"),
("--queue", "SPARK_YARN_QUEUE", "spark.yarn.queue"),
("--name", "SPARK_YARN_APP_NAME", "spark.app.name"),
("--files", "SPARK_YARN_DIST_FILES", "spark.yarn.dist.files"),
("--archives", "SPARK_YARN_DIST_ARCHIVES", "spark.yarn.dist.archives"))
("--name", "SPARK_YARN_APP_NAME", "spark.app.name"))
.foreach { case (optName, envVar, sysProp) => addArg(optName, envVar, sysProp, argsArrayBuf) }

logDebug("ClientArguments called with: " + argsArrayBuf)
Expand Down