SPARK-1119 and other build improvements

[pwendell]

1. Makes assembly and examples jar naming consistent in maven/sbt.
2. Updates make-distribution.sh to use Maven and fixes some bugs.
3. Updates the create-release script to call make-distribution script.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14365/

[witgo]

Why change using Maven build? The PR 480 has some relevant changes.

[pwendell]

@witgo we've been building releases with maven for several versions of spark due to better stability. We can look at #480 as well...

[pwendell]

Okay merged.

[witgo]

@berngp@pwendell ,

Whether we can delete the yarn.version, only using hadoop.version.
Like this,

 <profile>
<id>yarn</id>
<properties>
<hadoop.major.version>2</hadoop.major.version>
<hadoop.version>2.2.0</hadoop.version>
<protobuf.version>2.5.0</protobuf.version>
</properties>
<modules>
<module>yarn</module>
</modules>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.jboss.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.jboss.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.jboss.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</dependencyManagement>
</profile>

will cause any problems?

[berngp]

Yes it will, not every one uses the same version of HDFS vs YARN.

On Wednesday, April 23, 2014, Li Guoqiang notifications@github.com wrote:

@berngphttps://github.com/berngp @pwendellhttps://github.com/pwendell,
Whether we can delete the yarn.version, only using hadoop.version. This
will cause any problems?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/502#issuecomment-41237973
.

[witgo]

@berngp
Most of the people uses the same version of HDFS vs YARN.
We can be so

 <hadoop.version>1.0.4</hadoop.version>
<yarn.version>${hadoop.version}</yarn.version>

 <profile>
<id>yarn-alpha</id>
<properties>
<hadoop.major.version>2</hadoop.major.version> <hadoop.version>0.23.7</hadoop.version>
</properties>
<modules>
<module>yarn</module>
</modules>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>${yarn.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-common</artifactId>
<version>${yarn.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-client</artifactId>
<version>${yarn.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
</profile>

Most of the people use mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests clean package.
Others usemvn -Pyarn-alpha -Dhadoop.version=2.3.0 -DskipTests -Dyarn.version=0.23.9 clean package.

[berngp]

Makes sense to me.

On Wednesday, April 23, 2014, Li Guoqiang notifications@github.com wrote:

@berngphttps://github.com/berngp
Most of the people uses the same version of HDFS vs YARN.
We can be so

<hadoop.version>1.0.4</hadoop.version>
<yarn.version>${hadoop.version}</yarn.version>

yarn-alpha 2 0.23.7 yarn org.apache.hadoop hadoop-yarn-api ${yarn.version} org.apache.hadoop hadoop-yarn-common ${yarn.version} org.apache.hadoop hadoop-yarn-client ${yarn.version}

Most of the people use mvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests
clean package.
Others usemvn -Pyarn -Dhadoop.version=2.3.0 -DskipTests
-Dyarn.version=0.23.9 clean package.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/502#issuecomment-41239453
.