[ZEPPELIN-870] Improvement build and pom normalized

[shijinkui]

What is this PR for?

1. frontend-maven-plugin upgrade to 1.0, as low version occur error when build on osx
2. upgrade spark version from 1.4.1 to 1.6.1
3. define maven dependency scala version as property, used by dependency version include scala version
4. dependencyManagement define junit and scala-test dependency, same to pluginManagement
5. add spark.iscached property, used by profile when need download spark dist by download-maven-plugin.
6. add spark-2.0 and hadoop-2.7 profile

What type of PR is it?

[Improvement]

Todos

What is the Jira issue?

ZEPPELIN-870

How should this be tested?

run build command:
export MAVEN_OPTS="-Xmx3g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512M"; mvn clean package -Pbuild-distr -DskipTests -Pspark-1.6 -Phadoop-2.7 -Ppyspark -Pvendor-repo -Dignite.version=1.1.0-incubating -Pscalding -Pspark-dist-cached -Drat.skip=true

Questions:

[shijinkui]

who can review this pr? @catap@jonbuffington

[Leemoonsoo]

Thanks @shijinkui for the contribution. I want to point out that change spark.version here is probably not the good way to make 1.6.1 as a default build. while all other dependency versions are remained for spark 1.4.1.

I think activating profile spark-1.6 by default, could be a possible way to make 1.6.1 default.

[shijinkui]

good idea

[bzz]

Thank you for contribution, @shijinkui, would you be willing to create a JIRA issue as well and link it?

It is very sound and useful, please see the comments above for more detailed feedback.

The only thing that can be improved to merge it faster is - if we could separate it on 2 parts:

1. "no side-effects" one: normalization&variable reconciliation, which does not change or affect any build behavior
2. other changes affecting\optimizing the build, that do have side effects (add phases\change defaults\etc)

that would simplify the review and make things even smoother.

[shijinkui]

hi, @bzz thanks for review. had add spark-1.6 default active and validate phrase.
For the changes not more enough to separate, I think in one pr may be suitable.

I looked though most of code, the code format of java and scala code not well-formed. I'll have a new issue to refactor, that will be more code changes. :)

[shijinkui]

hi, @bzz the bower error, my local compile is ok, how can i continue

[ERROR] bower angular-ui-ace#0.1.1 invalid-meta angular-ui-ace is missing "ignore" entry in bower.json
[ERROR] bower ng-focus-if#~1.0.2 mismatch Version declared in the json (1.0.4) is different than the resolved one (1.0.5)
[ERROR] bower angular-websocket#~1.0.13 mismatch Version declared in the json (1.0.13) is different than the resolved one (1.0.14)
[ERROR] bower ace-builds#1.1.9 invalid-meta ace-builds is missing "main" entry in bower.json
[ERROR] bower ace-builds#1.1.9 invalid-meta ace-builds is missing "ignore" entry in bower.json
[ERROR] bower perfect-scrollbar extra-resolution Unnecessary resolution: perfect-scrollbar#~0.5.4

[bzz]

@shijinkui I do not think that mentioned fronted build warnings are relevant to CI build failure.
It seems that Selenium integration tests are failing

<<< ERROR!
org.openqa.selenium.ElementNotVisibleException: Element is not currently visible and so may not be interacted with

---

Update

From the embedded data:image/png;base64 screenshot, it looks like ZeppelinIT. testSparkInterpreterDependencyLoading is not able to find Spark interpreter button on the interepreters page

[bzz]

As #862 is merged - it should fix the CI, so could you please rebase on the latest master and see if that helps?

[shijinkui]

@bzz have rebased. Failed all for the spark archive download timeout by the url http://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz

https://travis-ci.org/apache/incubator-zeppelin/jobs/133047225
[INFO] --- download-maven-plugin:1.2.1:wget (download-sparkr-files) @ zeppelin-spark-dependencies ---
Downloading: http://archive.apache.org/dist/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz
org.apache.maven.wagon.providers.http.httpclient.conn.HttpHostConnectException: Connect to archive.apache.org:80 [archive.apache.org/163.172.17.199] failed: Connection refused
Downloading: http://archive.apache.org/dist/spark/spark-1.5.2/spark-1.5.2-bin-without-hadoop.tgz
org.apache.maven.wagon.providers.http.httpclient.conn.HttpHostConnectException: Connect to archive.apache.org:80 [archive.apache.org/163.172.17.199] failed: Connection refused
at org.apache.maven.wagon.providers.http.httpclient.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)
at https://travis-ci.org/apache/incubator-zeppelin/jobs/133047227

[jongyoul]

@shijinkui It becomes green. Could you please trigger it again?

[shijinkui]

@jongyoul I have no authorization。 cc @bzz

[jongyoul]

@shijinkui Just close and reopen it. :-)

[jongyoul]

I agree with @bzz. If your PR has both of parts which change behavior or not, I suggest you divide two seperate PR because of easy review. If you mix them, all reviewers should judge everyline which is normallized part or changing something. I also suggest you contribute changing style at first, and changing behavior is the second one. How about you?