Uh oh!
There was an error while loading. Please reload this page.
[SPARK-20899][PySpark] PySpark supports stringIndexerOrderType in RFormula - #18122
[SPARK-20899][PySpark] PySpark supports stringIndexerOrderType in RFormula#18122actuaryzhang wants to merge 6 commits into
Conversation
actuaryzhang
commented
May 26, 2017
SparkQA
commented
May 26, 2017
Test build #77428 has finished for PR 18122 at commit
|
| typeConverter=TypeConverters.toBoolean) | ||
| stringIndexerOrderType = Param(Params._dummy(), "stringIndexerOrderType", | ||
| "How to order categories of a string FEATURE column used by " + |
There was a problem hiding this comment.
FEATURE capitalize is common here?
There was a problem hiding this comment.
Changed it to lower case now.
SparkQA
commented
May 26, 2017
Test build #77440 has finished for PR 18122 at commit
|
yanboliang
left a comment
There was a problem hiding this comment.
One minor comment, otherwise LGTM. Thanks!
| |0.0|2.0| b|[2.0,1.0]| 0.0| | ||
| |0.0|0.0| a|(2,[],[])| 0.0| | ||
| +---+---+---+---------+-----+ | ||
| ... |
There was a problem hiding this comment.
Could you move the newly added test to tests.py? We keep the basic doc tests here both for test and example, other tests should be placed at tests.py. Thanks.
SparkQA
commented
May 29, 2017
Test build #77506 has finished for PR 18122 at commit
|
SparkQA
commented
May 29, 2017
Test build #77508 has finished for PR 18122 at commit
|
SparkQA
commented
May 30, 2017
Test build #77509 has finished for PR 18122 at commit
|
actuaryzhang
commented
May 30, 2017
@yanboliang I have moved the tests to the test file. Please let me know if there is anything else needed. Thanks. |
viirya
commented
May 30, 2017
LGTM |
yanboliang
left a comment
There was a problem hiding this comment.
One very minor comment, thanks!
| observed = transformedDF.select("features").collect() | ||
| expected = [[1.0, 0.0], [2.0, 1.0], [0.0, 0.0]] | ||
| for i in range(0, len(expected)): | ||
| self.assertTrue((observed[i]["features"].toArray() == expected[i]).all()) |
There was a problem hiding this comment.
Minor: Usually we're more prefer to use self.assertTrue(all(observed[i]["features"].toArray() == expected[i])).
SparkQA
commented
May 30, 2017
Test build #77537 has finished for PR 18122 at commit
|
yanboliang
commented
May 30, 2017
Merged into master, thanks for all. |
What changes were proposed in this pull request?
PySpark supports stringIndexerOrderType in RFormula as in #17967.
How was this patch tested?
docstring test