[SPARK-13967] [PYSPARK][ML] Added binary Param to Python CountVectorizer

[BryanCutler]

Added binary toggle param to CountVectorizer feature transformer in PySpark.

Created a unit test for using CountVectorizer with the binary toggle on.

[BryanCutler]

cc @MLnick

[holdenk]

Since we seem to be adding the same param to multiple models, would it maybe make sense to make this a shared param?

[SparkQA]

Test build #55541 has finished for PR 12308 at commit 7d362ed.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BryanCutler]

Since we seem to be adding the same param to multiple models, would it maybe make sense to make this a shared param?

Maybe, I'm not sure if there are other uses besides HashingTF but I know @MLnick was proposing a more general feature hasher that seems like it would reduce a lot more code duplication..

[MLnick]

Will take a look

[MLnick]

@holdenk@BryanCutler I'd say we could make binary shared, but the only thing is currently the doc is a bit different between them (the doc for CountVectorizer mentions the binarization occurs after minTF is applied, while minTF doesn't exist for HashingTF). So if there's a way to easily make them shared but have different doc, then go ahead.

We could later add minTF to HashingTF I guess, in which case it can definitely be a shared param.

[MLnick]

LGTM otherwise.

[jkbradley]

+1 for not sharing the Param if the docs (and semantics) differ

[mengxr]

Need a backslash at the end. Please check the generated HTML doc.

[BryanCutler]

Need a backslash at the end. Please check the generated HTML doc

This should have cause the doc checks to fail right?

[SparkQA]

Test build #55633 has finished for PR 12308 at commit 3907475.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[holdenk]

@BryanCutler it seems that (due to a bug introduced during code review) right now we only hault on doc build errors, not warnings as originally intended. Thanks for noticing this I'll make a follow up JIRA to deal with this.

[BryanCutler]

It looks like the option to treat warnings as errors is there, it just gets overwritten in the makefile

[holdenk]

Yup I've got a fix but going to cleanup the warnings that got in the meantime too.

[MLnick]

@mengxr@jkbradley anything further?

[jkbradley]

LGTM, feel free to merge, thanks!

[MLnick]

Merged to master. Thanks!