[SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example. by dongjoon-hyun · Pull Request #12809 · apache/spark

dongjoon-hyun · 2016-04-30T07:21:06Z

What changes were proposed in this pull request?

This PR aims to update Scala/Python/Java examples by replacing SQLContext with newly added SparkSession.

Use SparkSession Builder Pattern in 154(Scala 55, Java 52, Python 47) files.
Add getConf in Python SparkContext class: python/pyspark/context.py
Replace SQLContext Singleton Pattern with SparkSession Singleton Pattern:
- SqlNetworkWordCount.scala
- JavaSqlNetworkWordCount.java
- sql_network_wordcount.py

Now, SQLContexts are used only in R examples and the following two Python examples. The python examples are untouched in this PR since it already fails some unknown issue.

simple_params_example.py
aft_survival_regression.py

How was this patch tested?

Manual.

dongjoon-hyun · 2016-04-30T07:22:00Z

I thought this is a bug fix, but I think I need to update to use SparkSession, too.
I will add a commit very soon.

rxin · 2016-04-30T07:22:11Z

Are there other examples that we should update?

dongjoon-hyun · 2016-04-30T07:25:56Z

Oh, sure. May I proceed this PR for all examples and all testsuites together?

dongjoon-hyun · 2016-04-30T07:26:27Z

If you don't mind, I prefer to do as a single one.
I can update the JIRA and PR description.

dongjoon-hyun · 2016-04-30T07:28:31Z

I mean SparkSession updating here since I didn't run all examples.

SparkQA · 2016-04-30T07:31:41Z

Test build #57422 has finished for PR 12809 at commit f2087fd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T07:36:13Z

Test build #57424 has finished for PR 12809 at commit 7fee97a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-30T07:52:00Z

I'd put all the example changes together, and have a separate pr for the test changes.

dongjoon-hyun · 2016-04-30T08:03:45Z

I see. No problem. Then, I'll use this one for all the example changes (including some fix like here).

dongjoon-hyun · 2016-04-30T08:08:07Z

As you know, for the example, I need to verify the result manually.
I'll proceed testsuite one first because it's automatically verified while this one is marked as [WIP].

SparkQA · 2016-04-30T10:31:54Z

Test build #57438 has finished for PR 12809 at commit 94e5be3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T11:05:50Z

Test build #57440 has finished for PR 12809 at commit 5ebc6a7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-30T11:20:27Z

Test build #57441 has finished for PR 12809 at commit 7a52177.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-04-30T20:27:01Z

Hi, @rxin . For this issue, I'll add new constructor for SparkSession and proceed to Java examples.

 def this(sparkContext: JavaSparkContext) = this(sparkContext.sc)

Please let me know if there is some problem for this.

rxin · 2016-04-30T20:53:09Z

Yea that makes sense.

For this one, I think we should also have a SparkSession ctor that takes in SparkConf, which calls SparkContext.getOrCreate to load a SparkContext. Then users can use this without declaring two things!

dongjoon-hyun · 2016-04-30T20:59:55Z

I updated Java examples with SparkSession(JavaSparkContext). For SparkSession(SparkConf), I'll handle soon.

dongjoon-hyun · 2016-04-30T21:33:12Z

Now, I addressed all comments so far.
Thank you for fast reviews, @rxin .

SparkQA · 2016-04-30T22:23:36Z

Test build #57453 has finished for PR 12809 at commit 627f418.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class JavaSparkSessionSingleton

SparkQA · 2016-04-30T22:52:45Z

Test build #57454 has finished for PR 12809 at commit 113d7ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-05-01T03:28:45Z

Rebased.

SparkQA · 2016-05-01T03:40:25Z

Test build #57462 has finished for PR 12809 at commit c1833bb.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-05-01T05:14:31Z

Test build #57465 has finished for PR 12809 at commit 6ac52c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-05-01T17:37:35Z

Hi, @rxin .
Could you merge this PR please?
Or, please let me know if there is something to do more.

rxin · 2016-05-01T23:02:57Z

Hey @dongjoon-hyun Give me a day or two to think about the API for creating SparkSession. It'd be great to finalize that and then update the examples to reflect that.

My current thinking is to have a factory method that can be used to instantiate SparkSession, something like

SparkSession
.withMaster("local[4]")
.withConfig("...", "...")
.getOrCreate()

dongjoon-hyun · 2016-05-01T23:09:52Z

Thank you for feedback, @rxin . Sure!
Also, congratulation for branching 2.0. :)

rxin · 2016-05-02T22:28:25Z

I've merged #12830

want to update the example using that?

dongjoon-hyun · 2016-05-02T22:42:52Z

Sure! I'm waiting for it. :) Thank you!

dongjoon-hyun · 2016-05-04T18:10:23Z

Rebased to resolve conflicts.

dongjoon-hyun · 2016-05-04T18:15:21Z

Hi, @rxin and @andrewor14 .
Now, Scala/Java/Python examples use new style.
Could you review this PR?

SparkQA · 2016-05-04T18:35:38Z

Test build #57779 has finished for PR 12809 at commit 0346dda.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class JavaSparkSessionSingleton

andrewor14 · 2016-05-04T20:57:10Z

- SparkConf conf = new SparkConf().setAppName("JavaAFTSurvivalRegressionExample");
- JavaSparkContext jsc = new JavaSparkContext(conf);
- SQLContext jsql = new SQLContext(jsc);
+ SparkSession spark = SparkSession.builder().appName("JavaAFTSurvivalRegressionExample").getOrCreate();


line too long, can you break it into multiple lines (here and other places)

Sure! Two or more?
Two Lines
SparkSession spark = SparkSession .builder().appName("JavaAFTSurvivalRegressionExample").getOrCreate();
More Lines
SparkSession spark = SparkSession .builder() .appName("JavaAFTSurvivalRegressionExample") .getOrCreate();

Anyway, thank you for fast review! :)

more lines looks better to me

Thanks. I'll use that format for all languages.

andrewor14 · 2016-05-04T21:09:58Z

Thanks @dongjoon-hyun. This is mostly straightforward and it LGTM. I pointed out a few parts that were not totally straightforward for other reviewers.

dongjoon-hyun · 2016-05-04T21:16:28Z

Thank you for review, again!

andrewor14 · 2016-05-04T21:31:17Z

I'm going to merge this one first to avoid conflicts since it's such a big patch.

andrewor14 · 2016-05-04T21:31:22Z

Merging into master 2.0

## What changes were proposed in this pull request? This PR aims to update Scala/Python/Java examples by replacing `SQLContext` with newly added `SparkSession`. - Use **SparkSession Builder Pattern** in 154(Scala 55, Java 52, Python 47) files. - Add `getConf` in Python SparkContext class: `python/pyspark/context.py` - Replace **SQLContext Singleton Pattern** with **SparkSession Singleton Pattern**: - `SqlNetworkWordCount.scala` - `JavaSqlNetworkWordCount.java` - `sql_network_wordcount.py` Now, `SQLContexts` are used only in R examples and the following two Python examples. The python examples are untouched in this PR since it already fails some unknown issue. - `simple_params_example.py` - `aft_survival_regression.py` ## How was this patch tested? Manual. Author: Dongjoon Hyun <dongjoon@apache.org> Closes#12809 from dongjoon-hyun/SPARK-15031. (cherry picked from commit cdce4e6) Signed-off-by: Andrew Or <andrew@databricks.com>

dongjoon-hyun · 2016-05-04T21:50:05Z

Oh, thank you, @andrewor14 .

This PR removes `sqlContext` in examples. Actual usage was all replaced in #12809 but there are some in comments. Manual style checking. Author: hyukjinkwon <gurwls223@gmail.com> Closes#13006 from HyukjinKwon/minor-docs. (cherry picked from commit 2992a21) Signed-off-by: Andrew Or <andrew@databricks.com>

This PR removes `sqlContext` in examples. Actual usage was all replaced in #12809 but there are some in comments. Manual style checking. Author: hyukjinkwon <gurwls223@gmail.com> Closes#13006 from HyukjinKwon/minor-docs.

…with SparkSession ## What changes were proposed in this pull request? It seems most of Python examples were changed to use SparkSession by #12809. This PR said both examples below: - `simple_params_example.py` - `aft_survival_regression.py` are not changed because it dose not work. It seems `aft_survival_regression.py` is changed by #13050 but `simple_params_example.py` is not yet. This PR corrects the example and make this use SparkSession. In more detail, it seems `threshold` is replaced to `thresholds` here and there by 5a23213. However, when it calls `lr.fit(training, paramMap)` this overwrites the values. So, `threshold` was 5 and `thresholds` becomes 5.5 (by `1 / (1 + thresholds(0) / thresholds(1)`). According to the comment below. this is not allowed, https://github.com/apache/spark/blob/354f8f11bd4b20fa99bd67a98da3525fd3d75c81/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala#L58-L61. So, in this PR, it sets the equivalent value so that this does not throw an exception. ## How was this patch tested? Manully (`mvn package -DskipTests && spark-submit simple_params_example.py`) Author: hyukjinkwon <gurwls223@gmail.com> Closes#13135 from HyukjinKwon/SPARK-15031. (cherry picked from commit e2ec32d) Signed-off-by: Nick Pentreath <nickp@za.ibm.com>

…with SparkSession ## What changes were proposed in this pull request? It seems most of Python examples were changed to use SparkSession by #12809. This PR said both examples below: - `simple_params_example.py` - `aft_survival_regression.py` are not changed because it dose not work. It seems `aft_survival_regression.py` is changed by #13050 but `simple_params_example.py` is not yet. This PR corrects the example and make this use SparkSession. In more detail, it seems `threshold` is replaced to `thresholds` here and there by 5a23213. However, when it calls `lr.fit(training, paramMap)` this overwrites the values. So, `threshold` was 5 and `thresholds` becomes 5.5 (by `1 / (1 + thresholds(0) / thresholds(1)`). According to the comment below. this is not allowed, https://github.com/apache/spark/blob/354f8f11bd4b20fa99bd67a98da3525fd3d75c81/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala#L58-L61. So, in this PR, it sets the equivalent value so that this does not throw an exception. ## How was this patch tested? Manully (`mvn package -DskipTests && spark-submit simple_params_example.py`) Author: hyukjinkwon <gurwls223@gmail.com> Closes#13135 from HyukjinKwon/SPARK-15031.

dongjoon-hyun changed the title ~~[SPARK-15031][EXAMPLE] Fix SQL Python example.~~[WIP][SPARK-15031][EXAMPLE] Fix SQL Python example.Apr 30, 2016

dongjoon-hyun changed the title ~~[WIP][SPARK-15031][EXAMPLE] Fix SQL Python example.~~[SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python example.Apr 30, 2016

dongjoon-hyun changed the title ~~[SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python example.~~[SPARK-15031][EXAMPLE] Use SparkSession in Scala/Python/Java example.Apr 30, 2016

andrewor14 reviewed May 4, 2016
View reviewed changes

asfgit closed this in cdce4e6May 4, 2016

MLnick mentioned this pull request May 5, 2016
[SPARK-14512][DOC] Add python example for QuantileDiscretizer #12281
Closed

yanboliang mentioned this pull request May 5, 2016
[SPARK-15141][EXAMPLE][DOC] Update OneVsRest Examples #12920
Closed

HyukjinKwon mentioned this pull request May 9, 2016
[MINOR][DOCS] Remove remaining sqlContext in documentation at examples #13006
Closed

HyukjinKwon mentioned this pull request May 11, 2016
[SPARK-15031][SPARK-15134][EXAMPLE][DOC] Use SparkSession and update indent in examples #13050
Closed

dongjoon-hyun deleted the SPARK-15031 branch May 12, 2016 01:01

HyukjinKwon mentioned this pull request May 16, 2016
[SPARK-15031][EXAMPLES][FOLLOW-UP] Make Python param example working with SparkSession #13135
Closed

Uh oh!

Conversation

dongjoon-hyun commented Apr 30, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

rxin commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

rxin commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

rxin commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

SparkQA commented Apr 30, 2016

Uh oh!

dongjoon-hyun commented May 1, 2016

Uh oh!

SparkQA commented May 1, 2016

Uh oh!

SparkQA commented May 1, 2016

Uh oh!

dongjoon-hyun commented May 1, 2016

Uh oh!

rxin commented May 1, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 1, 2016

Uh oh!

rxin commented May 2, 2016

Uh oh!

dongjoon-hyun commented May 2, 2016

Uh oh!

dongjoon-hyun commented May 4, 2016

Uh oh!

dongjoon-hyun commented May 4, 2016

Uh oh!

SparkQA commented May 4, 2016

Uh oh!

andrewor14May 4, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunMay 4, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunMay 4, 2016

Choose a reason for hiding this comment

Uh oh!

andrewor14May 4, 2016

dongjoon-hyun commented Apr 30, 2016•
edited
Loading

rxin commented May 1, 2016•
edited
Loading

andrewor14May 4, 2016•
edited
Loading

dongjoon-hyunMay 4, 2016•
edited
Loading