Skip to content

[SPARK-15031][EXAMPLES][FOLLOW-UP] Make Python param example working with SparkSession - #13135

Closed
HyukjinKwon wants to merge 4 commits into
apache:masterfrom
HyukjinKwon:SPARK-15031
Closed

[SPARK-15031][EXAMPLES][FOLLOW-UP] Make Python param example working with SparkSession#13135
HyukjinKwon wants to merge 4 commits into
apache:masterfrom
HyukjinKwon:SPARK-15031

Conversation

@HyukjinKwon

@HyukjinKwonHyukjinKwon commented May 16, 2016

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

It seems most of Python examples were changed to use SparkSession by #12809. This PR said both examples below:

  • simple_params_example.py
  • aft_survival_regression.py

are not changed because it dose not work. It seems aft_survival_regression.py is changed by #13050 but simple_params_example.py is not yet.

This PR corrects the example and make this use SparkSession.

In more detail, it seems threshold is replaced to thresholds here and there by 5a23213. However, when it calls lr.fit(training, paramMap) this overwrites the values. So, threshold was 5 and thresholds becomes 5.5 (by 1 / (1 + thresholds(0) / thresholds(1)).

According to the comment below. this is not allowed,

* Note: Calling this with threshold p is equivalent to calling `setThresholds(Array(1-p, p))`.
* When [[setThreshold()]] is called, any user-set value for [[thresholds]] will be cleared.
* If both [[threshold]] and [[thresholds]] are set in a ParamMap, then they must be
* equivalent.
.

So, in this PR, it sets the equivalent value so that this does not throw an exception.

How was this patch tested?

Manully (mvn package -DskipTests && spark-submit simple_params_example.py)

Loading
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@HyukjinKwon@SparkQA@yanboliang@MLnick@zhengruifeng