Clarify docstring for Pyspark's foreachPartition

[tdhopper]

Due to the underlying use of mapPartitions which requires a function that maps partitions to partitions, foreachPartition requires the function passed to be a generator function or return an iterable (although these results are discarded).

This is currently not stated in the documentation except through the unexplained example. It would help users to understand that example and not waste time with this error:

TypeError: 'NoneType' object is not iterable

[AmplabJenkins]

Can one of the admins verify this patch?

[JoshRosen]

Actually, we might want to just fix this and allow foreachPartition and foreach to accept those types of UDFs, too.

[tdhopper]

Oh. Now that I look at master, @JoshRosen, I see that it's already been fixed by @davishere. The fix just isn't in 1.1. I guess we should close this?

[JoshRosen]

Maybe we can backport SPARK-2871 to 1.1, since it looks like it also fixes a bunch of preservesPartitioning bugs.

[JoshRosen]

@davies Do you think we should backport #2093 to branch-1.1 in order to fix this issue?

[davies]

@JoshRosen It will be better if we could easily backport them.

[tdhopper]

I'd love to see this happen.

[davis]

Ah - tdhopper, i think you meant @davies :)

[JoshRosen]

@davis I think someone mentioned you instead of @davies (an off-by-one-character error).

[JoshRosen]

(Imagine what @Override's notification inbox must look like...)

[JoshRosen]

If you don't mind, could you close this PR since it has been subsumed by another commit? If we want to track the progress / backport status of a different fix, then we should do that in JIRA.

[tdhopper]

@JoshRosen: Yup. Thanks.