Skip to content

Remove hardcoded Spark version string to use SBT defined version instead#600

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pierre-borckmans wants to merge 6 commits into from
Closed

Remove hardcoded Spark version string to use SBT defined version instead #600

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
12769f4
Remove hardcoded Spark version - use the version from the manifest (S…
Apr 30, 2014
f7c2298
Add spark version label in ui (master index page)
Apr 30, 2014
d26bc1e
Merge remote-tracking branch 'upstream/master'
Apr 30, 2014
0f88e95
Changed SparkContext SPARK_VERSION to use Utils.getSparkVersion
Apr 30, 2014
142be89
Modified Spark REPL to display Utils.getSparkVersion in Welcome message
Apr 30, 2014
da8886e
Changed getSparkVersion signature to omit the parentheses
Apr 30, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions core/src/main/scala/org/apache/spark/SparkContext.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1253,7 +1253,7 @@ class SparkContext(config: SparkConf) extends Logging {
*/
object SparkContext extends Logging {

private[spark] val SPARK_VERSION = "1.0.0"
private[spark] val SPARK_VERSION = Utils.getSparkVersion

private[spark] val SPARK_JOB_DESCRIPTION = "spark.job.description"

Expand DownExpand Up@@ -1565,4 +1565,3 @@ private[spark] class WritableConverter[T](
val writableClass: ClassTag[T] => Class[_ <: Writable],
val convert: Writable => T)
extends Serializable

1 change: 1 addition & 0 deletions core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -72,6 +72,7 @@ private[spark] class MasterPage(parent: MasterWebUI) extends WebUIPage("") {
<div class="row-fluid">
<div class="span12">
<ul class="unstyled">
<li><strong>Version:</strong> {Utils.getSparkVersion}</li>
<li><strong>URL:</strong> {state.uri}</li>
<li><strong>Workers:</strong> {state.workers.size}</li>
<li><strong>Cores:</strong> {state.workers.map(_.cores).sum} Total,
Expand Down
5 changes: 5 additions & 0 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1037,6 +1037,11 @@ private[spark] object Utils extends Logging {
obj.getClass.getSimpleName.replace("$", "")
}

/** Return the current version of Spark */
def getSparkVersion: String = {
getClass.getPackage.getImplementationVersion

@pwendellpwendellApr 30, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this also work in the maven build?

@pierre-borckmanspierre-borckmansApr 30, 2014

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not check as I was not really aware of the Maven build.
Apparently, I always overlooked this page:
http://spark.apache.org/docs/latest/building-with-maven.html

I'll check this now.

@pierre-borckmanspierre-borckmansApr 30, 2014

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to work as well.
In fact, the build system does not matter:
http://docs.oracle.com/javase/7/docs/api/java/lang/Package.html
The version string is retrieved from the JAR manifest.

@aarondavaarondavApr 30, 2014

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this manifest property must be correctly set from the build system, but it's good to know if both systems do so.

@pierre-borckmanspierre-borckmansApr 30, 2014

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed.
I confirm that both sbt assembly and mvn package produce a fat jar in which the manifest contains the version.

unzip -p assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop2.0.0-mr1-cdh4.4.0.jar META-INF/MANIFEST.MF

outputs:

Implementation-Vendor: The Apache Software Foundation
Implementation-Title: Spark Project Core
Implementation-Version: 1.0.0-SNAPSHOT
Implementation-Vendor-Id: org.apache.spark
Built-By: ria
Build-Jdk: 1.6.0_27
Specification-Vendor: The Apache Software Foundation
Specification-Title: Spark Project Core
Created-By: Apache Maven 3.0.4
Specification-Version: 1.0.0-SNAPSHOT
Archiver-Version: Plexus Archiver

in both cases

}

/** Return an option that translates JNothing to None */
def jsonOption(json: JValue): Option[JValue] = {
json match {
Expand Down
6 changes: 4 additions & 2 deletions repl/src/main/scala/org/apache/spark/repl/SparkILoopInit.scala
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,6 +14,8 @@ import scala.reflect.internal.util.Position
import scala.util.control.Exception.ignoring
import scala.tools.nsc.util.stackTraceString

import org.apache.spark.util.Utils

/**
* Machinery for the asynchronous initialization of the repl.
*/
Expand All@@ -26,9 +28,9 @@ trait SparkILoopInit {
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.0.0-SNAPSHOT
/___/ .__/\_,_/_/ /_/\_\ version %s
/_/
""")
""".format(Utils.getSparkVersion))
import Properties._
val welcomeMsg = "Using Scala %s (%s, Java %s)".format(
versionString, javaVmName, javaVersion)
Expand Down