[SPARK-8674] [MLlib] Implementation of a 2 sample Kolmogorov Smirnov Test by josepablocam · Pull Request #7075 · apache/spark

josepablocam · 2015-06-28T20:35:48Z

The current patch implements a 2-sample, 2-sided Kolmogorov Smirnov test. Similarly to the 1-sample implementation, we seek to reduce the shuffles necessary for computation. The user can provide 2 RDD[Double] and the Statistics.ksTest function allows them to test the null hypothesis that both samples came from the same probability distribution.

This patch includes the 1-sample test (so that reviewers can see the broader context of the change), however, that portion (and relevant tests) are being reviewed at https://issues.apache.org/jira/browse/SPARK-8598.

srowen · 2015-07-07T18:40:08Z

So this isn't to be merged in its current form? Put [WIP] in the title. This should probably just be reviewed later if/when the other PR is merged.

sryza · 2015-07-13T22:30:42Z

Hey @josepablocam can you rebase this on current master?

josepablocam · 2015-07-13T22:36:57Z

@sryza yes, will do.

…k relevant from 1-sample test

mengxr · 2015-07-30T01:30:09Z

@sryza Do you want to make another pass? Please sign off if you think this is ready:)

sryza · 2015-07-31T01:17:37Z

Call this unionedData

Would it be clearer to write true and false instead of isSample1 and !isSample1? I don't have a strong opinion.
Can the map functions be like .map((_, isSample1)) for tidiness or does that syntax not work here?
Finally I wonder if .union is clearer than ++ here? I don't have a strong opinion either, just am somehow used to method invocations on RDDs and ++-like syntax for Scala collections.

sryza · 2015-07-31T01:36:06Z

@mengxr @josepablocam oops thought it was still a WIP for some reason. Just took a pass. It looks mostly done - I just had a bunch of nits and a test request.

josepablocam · 2015-07-31T04:28:33Z

Mmm. I seem to be having some issues building and testing on my laptop. It keeps failing when building Catalyst. I'll try this first thing in the morning at work and push if it passes tests.

srowen · 2015-07-31T07:13:25Z

.sortByKey? I think the mapPartitions doesn't need braces, just parens, but that's tiny.

…to account for aliasing of commons' KS test

sryza · 2015-08-03T20:08:54Z

Nit: I'd make the above one "Sample follows the theoretical distribution" or make the bottom one "Both samples follow same distribution".

fixed. Modified the first. Thanks

…ical functions are package private and can be tested more directly

sryza · 2015-08-04T20:52:35Z

jenkins, test this please

sryza · 2015-08-04T20:52:48Z

This LGTM pending jenkins

mengxr · 2015-08-05T15:06:58Z

test this please

SparkQA · 2015-08-05T17:49:41Z

Test build #39872 has finished for PR 7075 at commit 16ba96e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

josepablocam · 2015-08-05T17:53:22Z

ugh, did not reword the tests in pyspark after we slightly cleaned up the grammar in the 2 sample test. I will make the ks 2 sample test hypothesis statement match the grammar in the first. Sorry about this!

… unit test failure by reversing prior grammar change

sryza · 2015-08-17T13:36:35Z

@mengxr is it too late to get this in to 1.5?

@josepablocam are you able to resolve merge conflicts?

josepablocam · 2015-08-17T18:40:40Z

@sryza fixed merge conflicts

SparkQA · 2016-05-17T21:10:11Z

Test build #58715 has finished for PR 7075 at commit feacda0.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-11-29T16:08:51Z

Test build #69329 has finished for PR 7075 at commit feacda0.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

HyukjinKwon · 2017-06-19T03:44:10Z

@josepablocam, it looks the conflicts were not resolved cleanly. Would you resolve them?

josepablocam · 2017-06-19T07:06:12Z

I'm not involved in this anymore. If you're interested in making those fixes, feel free. Thanks.

…

On Jun 19, 2017 5:44 AM, "Hyukjin Kwon" ***@***.***> wrote: @josepablocam <https://github.com/josepablocam>, it looks the conflicts were not resolved cleanly. Would you resolve them? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7075 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE3GlO5eHVTNq47py-_XuJbhLTFkn89Eks5sFe6tgaJpZM4FNtLA> .

gatorsmile · 2017-06-27T06:39:09Z

We are closing it due to inactivity. please do reopen if you want to push it forward. Thanks!

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 I believe the author in apache#14807 removed his account. Closesapache#7075Closesapache#8927Closesapache#9202Closesapache#9366Closesapache#10861Closesapache#11420Closesapache#12356Closesapache#13028Closesapache#13506Closesapache#14191Closesapache#14198Closesapache#14330Closesapache#14807Closesapache#15839Closesapache#16225Closesapache#16685Closesapache#16692Closesapache#16995Closesapache#17181Closesapache#17211Closesapache#17235Closesapache#17237Closesapache#17248Closesapache#17341Closesapache#17708Closesapache#17716Closesapache#17721Closesapache#17937 Added: Closesapache#14739Closesapache#17139Closesapache#17445Closesapache#18042Closesapache#18359 Added: Closesapache#16450Closesapache#16525Closesapache#17738 Added: Closesapache#16458Closesapache#16508Closesapache#17714 Added: Closesapache#17830Closesapache#14742 ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18417 from HyukjinKwon/close-stale-pr.

josepablocam changed the title ~~[SPARK-8674] [MLlib] Implementation of a 2 sample Kolmogorov Smirnov Test~~[SPARK-8674] [MLlib] [WIP] Implementation of a 2 sample Kolmogorov Smirnov TestJul 7, 2015

cleaned up 2 sample kolmogorov test implementation to include feedbac…
e0686a7
…k relevant from 1-sample test

josepablocam force-pushed the twoSampleKSTest branch from 5c9da55 to e0686a7Compare July 14, 2015 02:37

renamed kolmogorovSmirnov2 to kolmogorovSmirnov2Sample
7a43f49

josepablocam changed the title ~~[SPARK-8674] [MLlib] [WIP] Implementation of a 2 sample Kolmogorov Smirnov Test~~[SPARK-8674] [MLlib] Implementation of a 2 sample Kolmogorov Smirnov TestJul 21, 2015

sryza reviewed Jul 31, 2015
View reviewed changes

josepablocam added 2 commits July 30, 2015 21:11

incorporated latest feedback from@sryza
88831fa

Merge remote-tracking branch 'upstream/master' into test
34c75af

srowen reviewed Jul 31, 2015
View reviewed changes

incorporated PR feedback from@syrzaand@srowen. Merged in upstream …
748e9d3
…to account for aliasing of commons' KS test

sryza reviewed Aug 3, 2015
View reviewed changes

further PR feedback from@sryza. Modified tests now that certain crit…
16ba96e
…ical functions are package private and can be tested more directly

minor grammar fix in TwoSampleTwoSided Null hypothesis, fixes pyspark…
6a27682
… unit test failure by reversing prior grammar change

resolved merge conflicts for 2 sample ks
feacda0

HyukjinKwon mentioned this pull request Jun 25, 2017
[INFRA] Close stale PRs #18417
Closed

asfgit closed this in b32bd00Jun 27, 2017

Uh oh!

Conversation

josepablocam commented Jun 28, 2015

Uh oh!

srowen commented Jul 7, 2015

Uh oh!

sryza commented Jul 13, 2015

Uh oh!

josepablocam commented Jul 13, 2015

Uh oh!

mengxr commented Jul 30, 2015

Uh oh!

sryzaJul 31, 2015

Choose a reason for hiding this comment

Uh oh!

josepablocamJul 31, 2015

Choose a reason for hiding this comment

Uh oh!

srowenJul 31, 2015

Choose a reason for hiding this comment

Uh oh!

sryza commented Jul 31, 2015

Uh oh!

josepablocam commented Jul 31, 2015

Uh oh!

srowenJul 31, 2015

Choose a reason for hiding this comment

Uh oh!

sryzaAug 3, 2015

Choose a reason for hiding this comment

Uh oh!

josepablocamAug 4, 2015

Choose a reason for hiding this comment

Uh oh!

sryza commented Aug 4, 2015

Uh oh!

sryza commented Aug 4, 2015

Uh oh!

mengxr commented Aug 5, 2015

Uh oh!

SparkQA commented Aug 5, 2015

Uh oh!

josepablocam commented Aug 5, 2015

Uh oh!

sryza commented Aug 17, 2015

Uh oh!

josepablocam commented Aug 17, 2015

Uh oh!

SparkQA commented May 17, 2016

Uh oh!

SparkQA commented Nov 29, 2016

Uh oh!

HyukjinKwon commented Jun 19, 2017

Uh oh!

josepablocam commented Jun 19, 2017 via email

Uh oh!

gatorsmile commented Jun 27, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants