[SPARK-14238][ML][MLLIB][PYSPARK] Add binary toggle Param to PySpark HashingTF in ML & MLlib

[yongtang]

What changes were proposed in this pull request?

This fix tries to add binary toggle Param to PySpark HashingTF in ML & MLlib. If this toggle is set, then all non-zero counts will be set to 1.

Note: This fix (SPARK-14238) is extended from SPARK-13963 where Scala implementation was done.

How was this patch tested?

This fix adds two tests to cover the code changes. One for HashingTF in PySpark's ML and one for HashingTF in PySpark's MLLib.

[MLnick]

ok to test

[SparkQA]

Test build #54623 has finished for PR 12079 at commit e58d1a2.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #54631 has finished for PR 12079 at commit 1e24a68.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yanboliang]

We should keep the doc of the Param consistent with Scala.

[SparkQA]

Test build #54642 has finished for PR 12079 at commit a71f59b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yongtang]

Rebased to fix conflicts.

[SparkQA]

Test build #55524 has finished for PR 12079 at commit 829c87e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

One minor note:Often we want to go with Scala first then Python, but in either direction if we are only doing one at a time it can be good practice to create either a follow up JIRA or a subtask on the existing JIRA to also expose the implementation in the other language.

[holdenk]

We probably want to mention the default value here (namely false).

[yongtang]

Thanks @holdenk this issue has been addressed.

[holdenk]

Great! Looking at the incoming PRs it seems there is a second PR also adding a binary feature to another model - it might make sense to move this to a shared param instead of having it be per-model (although it will require coordination with the other PR timing wise).

[MLnick]

See #12308 (comment)

[MLnick]

if true -> if True

[yongtang]

@holdenk The Scala implementation has ben completed in SPARK-13963. I updated the description of this pull request to show the linkage between this issue (SPARK-14238) and SPARK-13963.

[SparkQA]

Test build #55587 has finished for PR 12079 at commit 9c2b4ab.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

The style seems to be . Default False rather than . (default: False). @BryanCutler@holdenk thoughts?

Though I must say I'd prefer (default: X). across the board myself.

[MLnick]

A few minor comments, otherwise LGTM.

@holdenk@BryanCutler we could merge this and #12308, and then update the param to be shared (if we can do the different doc thing?).

[yongtang]

Thanks @MLnick I just updated the pull request to address several minor issues. With respect to . Default False vs . (default: False), I changed it to . Default False for now. But if you want to see (default: X) I can change it (including the rest of the file) to it as well.

[SparkQA]

Test build #55608 has finished for PR 12079 at commit 551cc6e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BryanCutler]

@holdenk@BryanCutler we could merge this and #12308, and then update the param to be shared (if we can do the different doc thing?).

I think that will be better and maybe then we can change the param to be shared on both the Scala and Python side.

[holdenk]

@BryanCutler / @yongtang That sounds reasonable :)

[MLnick]

As per @jkbradley's #12308 (comment), let's keep them separate params.

[MLnick]

jenkins retest this please