Skip to content

[ZEPPELIN-1544]upgrade spark version to 2.0.1 as profile spark-2.0 - #1511

Closed
LantaoJin wants to merge 2 commits into
apache:masterfrom
LantaoJin:master
Closed

[ZEPPELIN-1544]upgrade spark version to 2.0.1 as profile spark-2.0#1511
LantaoJin wants to merge 2 commits into
apache:masterfrom
LantaoJin:master

Conversation

@LantaoJin

Copy link
Copy Markdown
Contributor

What is this PR for?

A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html

What type of PR is it?

[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]

Todos

  • - Task

What is the Jira issue?

How should this be tested?

Outline the steps to test the PR here.

Screenshots (if appropriate)

Questions:

  • Does the licenses files need update?
  • Is there breaking changes for older versions?
  • Does this needs documentation?

@jongyoul

Copy link
Copy Markdown
Member

LGTM. Merging if there's no more discussion.

//Only one of py4j-0.9-src.zip and py4j-0.8.2.1-src.zip should exist
String[] pythonLibs = new String[]{"pyspark.zip", "py4j-0.9-src.zip", "py4j-0.8.2.1-src.zip",
"py4j-0.10.1-src.zip"};
"py4j-0.10.3-src.zip"};

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't "py4j-0.10.1-src.zip" need to be in the array for in case of spark 2.0.0?

@jongyouljongyoulOct 14, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. My mistake. We need to keep old py4j for backward compatibility. I've checked that 2.0.0 has python/lib/py4j-0.10.1-src.zip

@LantaoJinLantaoJinOct 15, 2016

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You means that add "py4j-0.10.3-src.zip" after "py4j-0.10.1-src.zip"? Can it confuse the logic below?
// Distribute two libraries(pyspark.zip and py4j-*.zip) to workers
// when spark version is less than or equal to 1.4.1
if (pythonLibUris.size() == 2) {
If this semantics is the same as before adding. I will follow your comment to modify.

@jongyoul

Copy link
Copy Markdown
Member

@LantaoJin Can you edit your description following the guide?

@Leemoonsoo

Copy link
Copy Markdown
Member

Thanks @LantaoJin for the contribution.
CI failure is not relevant.

LGTM and merge if there're no more discussions.

darionyaphet pushed a commit to darionyaphet/zeppelin that referenced this pull request Oct 27, 2016
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: LantaoJin <jinlantao@gmail.com>
Closesapache#1511 from LantaoJin/master and squashes the following commits:
671cd68 [LantaoJin] keep old py4j for backward compatibility
3bd84cc [LantaoJin] [ZEPPELIN-1544]upgrade spark version to 2.0.1 as profile spark-2.0
pedrozatta pushed a commit to pedrozatta/zeppelin that referenced this pull request Oct 27, 2016
### What is this PR for?
A few sentences describing the overall goals of the pull request's commits.
First time? Check out the contributing guide - https://zeppelin.apache.org/contribution/contributions.html
### What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* Open an issue on Jira https://issues.apache.org/jira/browse/ZEPPELIN/
* Put link here, and add [ZEPPELIN-*Jira number*] in PR title, eg. [ZEPPELIN-533]
### How should this be tested?
Outline the steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: LantaoJin <jinlantao@gmail.com>
Closesapache#1511 from LantaoJin/master and squashes the following commits:
671cd68 [LantaoJin] keep old py4j for backward compatibility
3bd84cc [LantaoJin] [ZEPPELIN-1544]upgrade spark version to 2.0.1 as profile spark-2.0
asfgit pushed a commit that referenced this pull request Nov 2, 2016
Closes#338 (fixed by #1476)
Closes#1522 (fixed by #1559)
Closes#1527 (fixed by #1511)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@LantaoJin@jongyoul@Leemoonsoo