Uh oh!
There was an error while loading. Please reload this page.
[SPARK-15811] [SQL] fix the Python UDF in Scala 2.10 - #13717
Conversation
davies
commented
Jun 16, 2016
cc @JoshRosen |
davies
commented
Jun 16, 2016
@rxin Another thing I found is that the SerializationDebugger ran into a infinite loop (or very very slow) before the fix, I have to disable it to release that an Iterator can't be serialized. cc @JoshRosen for a jenkins build for 2.10 (we could merge it then verity that) |
damnMeddlingKid
commented
Jun 16, 2016
Just to take a step back, Is the suite lacking coverage for this feature ?. This sort of thing should have been caught in the unit tests. |
davies
commented
Jun 16, 2016
@damnMeddlingKid This feature is covered well by unit tests (PR builder ran with Scala 2.11 and Hadoop 2.x), but we did not have a jenkins build to ran with Scala 2.0, will have one to run again master. |
| // Check to make sure that the UDF can be evaluated with only the input of this child. | ||
| udf.references.subsetOf(child.outputSet) | ||
| } | ||
| }.toArray // force the iterator to an array, that can't be serialized in Scala 2.10 |
There was a problem hiding this comment.
do u mean the iterator couldn't have been serialized in scala 2.10?
maybe say "Turn it into an array since iterators cannot be serialized in Scala 2.10"
SparkQA
commented
Jun 16, 2016
Test build #60659 has finished for PR 13717 at commit
|
SparkQA
commented
Jun 17, 2016
Test build #60685 has finished for PR 13717 at commit
|
rxin
commented
Jun 17, 2016
Merging in master/2.0. |
## What changes were proposed in this pull request? Iterator can't be serialized in Scala 2.10, we should force it into a array to make sure that . ## How was this patch tested? Build with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build). Author: Davies Liu <davies@databricks.com> Closes#13717 from davies/fix_udf_210. (cherry picked from commit ef43b4e) Signed-off-by: Reynold Xin <rxin@databricks.com>
What changes were proposed in this pull request?
Iterator can't be serialized in Scala 2.10, we should force it into a array to make sure that .
How was this patch tested?
Build with Scala 2.10 and ran all the Python unit tests manually (will be covered by a jenkins build).