[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None by jiayue-zhang · Pull Request #16225 · apache/spark

jiayue-zhang · 2016-12-09T05:33:51Z

What changes were proposed in this pull request?

Allow DataFrame.replace() to replace with None/null values.

How was this patch tested?

Python doctest and unit test. Scala unit test.

HyukjinKwon · 2016-12-09T15:10:48Z

Would we need a test for Scala too? I checked this by myself and it seems working fine with Scala though. I could argue that this affects the language-specific functions of both Python and Scala as a not strong opinion.

JoshRosen · 2016-12-09T17:38:37Z

- case _ => replacement.map { case (k, v) => (convertToDouble(k), convertToDouble(v)) }
- }
+ val replacementMap: Map[_, _] =
+ if (replacement.head._2 == null)


I think you can just write case null => here and thereby avoid the ifelse

I tried that but failed. Scala doesn't allow pattern matching with null, scala.Nothing and scala.Null

Oh, wait case null => works. I tried case v null which doesn't. Let me modify this. Thanks!

SparkQA · 2016-12-10T16:55:38Z

Test build #3490 has finished for PR 16225 at commit 0b15c8f.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-05-23T17:37:39Z

ok to test

gatorsmile · 2017-05-23T19:06:47Z

@bravo-zhang Could you resolve the conflicts? I will review it then. Thanks!

SparkQA · 2017-05-23T20:32:54Z

Test build #77261 has finished for PR 16225 at commit 2c532c3.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

jiayue-zhang · 2017-05-24T00:53:10Z

Thanks for taking a look, @gatorsmile The conflicts have been resolved.
I appreciate if @zero323 can take a look as well since you made improvement on this function recently.

SparkQA · 2017-05-24T02:30:45Z

Test build #77273 has finished for PR 16225 at commit 43fb6bd.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-24T05:06:07Z

Test build #77282 has finished for PR 16225 at commit b5424d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk

Thanks for working on this, sorry its taken so long to review. I did a first read through the Python side and I've got two minor questions. Hopefully @zero323, @davie , or @gatorsmile can also have some time once the current release is finished to take a look.

holdenk · 2017-06-15T09:49:38Z

 "Got {0}".format(type(to_replace)))

- if not isinstance(value, valid_types) and not isinstance(to_replace, dict):
+ if not isinstance(value, valid_types) and value is not None \


So slightly jet lagged style question: would this be clearer if we just add type(None) to L1398? I know PEP8 says we should only use isis not for checking if something is none rather than depending on the implicit conversion to boolean -- but since really checking the type here we aren't really in danger of that. (This is just a suggestion to make it easier to read - if others think its easier to read this way thats fine :)). @davies ?

holdenk · 2017-06-15T09:50:25Z

 to_replace = [to_replace]

- if isinstance(value, (float, int, long, basestring)):
+ if isinstance(value, (float, int, long, basestring)) or value is None:


same as above

holdenk · 2017-06-15T09:58:10Z

- if not any(all_of_type(rep_dict.keys()) and all_of_type(rep_dict.values())
+ if not any(all_of_type(rep_dict.keys())
+ and (all_of_type(rep_dict.values())
+ or list(rep_dict.values()).count(None) == len(rep_dict))


In the Scala code null is allowed in to be the replacement value for some of the elements but not in the Python code. Is this intentional? If so we should document it clearly and expand on the error message bellow (otherwise we should make it more flexible).

gatorsmile · 2017-06-27T06:42:40Z

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

jiayue-zhang · 2017-08-02T16:03:25Z

@holdenk Thanks for review. I'll combine type(None) in the isinstance. I also made Scala and Python to accept null more generally and in the same way.
PR is reopened at: #18820
@gatorsmile please also take a look.

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 I believe the author in apache#14807 removed his account. Closesapache#7075Closesapache#8927Closesapache#9202Closesapache#9366Closesapache#10861Closesapache#11420Closesapache#12356Closesapache#13028Closesapache#13506Closesapache#14191Closesapache#14198Closesapache#14330Closesapache#14807Closesapache#15839Closesapache#16225Closesapache#16685Closesapache#16692Closesapache#16995Closesapache#17181Closesapache#17211Closesapache#17235Closesapache#17237Closesapache#17248Closesapache#17341Closesapache#17708Closesapache#17716Closesapache#17721Closesapache#17937 Added: Closesapache#14739Closesapache#17139Closesapache#17445Closesapache#18042Closesapache#18359 Added: Closesapache#16450Closesapache#16525Closesapache#17738 Added: Closesapache#16458Closesapache#16508Closesapache#17714 Added: Closesapache#17830Closesapache#14742 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18417 from HyukjinKwon/close-stale-pr.

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None
2653750

Scala test for df.replace with null
2eac8b9

JoshRosen reviewed Dec 9, 2016
View reviewed changes

jiayue-zhang added 2 commits December 9, 2016 09:44

Use pattern matching for null case
7949292

Fix indentation
0b15c8f

Fix Python style check
2c532c3

jiayue-zhang added 2 commits May 23, 2017 17:09

Merge branch 'master' into spark-14932
5ab39cc

Improve scala doc and pyspark test
43fb6bd

Fix python3 dict.values() syntax
b5424d9

holdenk reviewed Jun 15, 2017
View reviewed changes

HyukjinKwon mentioned this pull request Jun 25, 2017
[INFRA] Close stale PRs #18417
Closed

asfgit closed this in b32bd00Jun 27, 2017

jiayue-zhang mentioned this pull request Aug 2, 2017
[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None #18820
Closed

Uh oh!

Conversation

jiayue-zhang commented Dec 9, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Dec 9, 2016

Uh oh!

JoshRosenDec 9, 2016

Choose a reason for hiding this comment

Uh oh!

jiayue-zhangDec 9, 2016

Choose a reason for hiding this comment

Uh oh!

jiayue-zhangDec 9, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 10, 2016

Uh oh!

gatorsmile commented May 23, 2017

Uh oh!

gatorsmile commented May 23, 2017

Uh oh!

SparkQA commented May 23, 2017

Uh oh!

jiayue-zhang commented May 24, 2017

Uh oh!

SparkQA commented May 24, 2017

Uh oh!

SparkQA commented May 24, 2017

Uh oh!

holdenk left a comment

Choose a reason for hiding this comment

Uh oh!

holdenkJun 15, 2017

Choose a reason for hiding this comment

Uh oh!

holdenkJun 15, 2017

Choose a reason for hiding this comment

Uh oh!

holdenkJun 15, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jun 27, 2017

Uh oh!

jiayue-zhang commented Aug 2, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jiayue-zhang commented Dec 9, 2016•
edited
Loading