[SPARK-15750][MLLib][PYSPARK] Constructing FPGrowth fails when no numPartitions specified in pyspark

[zjffdu]

What changes were proposed in this pull request?

Change FPGrowth from private to private[spark]. If no numPartitions is specified, then default value -1 is used. But -1 is only valid in the construction function of FPGrowth, but not in setNumPartitions. So I make this change and use the constructor directly rather than using set method.

How was this patch tested?

Unit test is added

[SparkQA]

Test build #59923 has finished for PR 13493 at commit 56cd7cf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MechCoder]

lgtm cc: @MLnick

[mengxr]

Shall we change the logic in the Python wrapper and set numPartitions correctly if it is -1? Please also update the PR description to add more details to the changes.

[SparkQA]

Test build #3236 has finished for PR 13493 at commit 56cd7cf.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[jkbradley]

Pinging! Is this still active?

[zjffdu]

@jkbradley I will update the PR soon

[zjffdu]

@mengxr@jkbradley I think we should keep as less logic in python side as possible. So I didn't update the PR, please help review. Thanks

[SparkQA]

Test build #65071 has finished for PR 13493 at commit f0d37c8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

This looks pretty reasonable to me, the only thing is (and its relatively minor) testing the default param might be better as a tests.py test rather than as a doc test since the doctests are part of the documentation and we might not need this in the docs along with the current example.

[jkbradley]

Same here, looks fine except for @holdenk 's comment. Thanks!

[jkbradley]

Ping --- let me know if you'd like someone to take it over. Thanks!

[zjffdu]

Sorry, I am in vacation. Will update it next week.

[holdenk]

@zjffdu Enjoy vacation and no worries - I don't think this is blocking anything and we can circle back in a week :)

[SparkQA]

Test build #66589 has finished for PR 13493 at commit e8fefc0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zjffdu]

PR is updated, @holdenk@jkbradley

[zjffdu]

ping @jkbradley@holdenk

[SparkQA]

Test build #81076 has finished for PR 13493 at commit e8fefc0.

• This patch fails PySpark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[WeichenXu123]

ping @zjffdu Looks reasonable fix, But pls resolve the conflicts! Thanks!

[WeichenXu123]

This line is useless.

[SparkQA]

Test build #82315 has finished for PR 13493 at commit 3575444.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #82319 has finished for PR 13493 at commit fbbcd26.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

retest this please

[SparkQA]

Test build #82330 has finished for PR 13493 at commit fbbcd26.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[lu-wang-dl]

The bug is confirmed. The fix looks pretty reasonable to me. ping @jkbradley .

[jkbradley]

LGTM pending fresh tests
Sorry for the delay!

[SparkQA]

Test build #4167 has finished for PR 13493 at commit fbbcd26.

• This patch fails Spark unit tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[zjffdu]

Thanks @jkbradley The failed tests seems unrelated.

[lu-wang-dl]

LGTM retest this please

[SparkQA]

Test build #90147 has finished for PR 13493 at commit fbbcd26.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[WeichenXu123]

LGTM!

[jkbradley]

Merging with master
Thanks all!