Uh oh!
There was an error while loading. Please reload this page.
[SPARK-18891][SQL] Support for specific Java List subtypes - #18009
[SPARK-18891][SQL] Support for specific Java List subtypes#18009michalsenkyr wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
This is not supported intentionally. If we want to support java classes and scala classes mixedly, we should unify the ScalaRelfection and JavaTypeInference, which is a lot of effort.
For now I think it's more important to completely support specific collections type in scala, and then try to unify the ScalaRelfection and JavaTypeInference.
what do you think?
There was a problem hiding this comment.
Alternatively you can improve JavaTypeInference to support specific java list and test it with java bean in JavaDatasetSuite
There was a problem hiding this comment.
I did not notice that there is separate inference code for Java classes. It would certainly be nice to unify the code for Java and Scala classes. Is this already being worked on/planned?
I moved the List support to JavaTypeInference and rewrote tests accordingly.
There was a problem hiding this comment.
it's planned but no one is working on it yet. You can start by creating a JIRA ticket if you are interested :)
cloud-fan
commented
Jun 10, 2017
ping @michalsenkyr |
Remove specific Java List support from ScalaReflection Remove implicit encoder for Java Lists Add relevant tests to JavaDatasetSuite Remove tests from ScalaReflectionSuite and DatasetPrimitiveSuite
1edaf85 to
881e636Comparecloud-fan
commented
Jun 10, 2017
ok to test |
cloud-fan
commented
Jun 10, 2017
LGTM, pending test |
SparkQA
commented
Jun 10, 2017
Test build #77879 has finished for PR 18009 at commit
|
cloud-fan
commented
Jun 12, 2017
thanks, merging to master! |
## What changes were proposed in this pull request? Add support for specific Java `List` subtypes in deserialization as well as a generic implicit encoder. All `List` subtypes are supported by using either the size-specifying constructor (one `int` parameter) or the default constructor. Interfaces/abstract classes use the following implementations: * `java.util.List`, `java.util.AbstractList` or `java.util.AbstractSequentialList` => `java.util.ArrayList` ## How was this patch tested? ```bash build/mvn -DskipTests clean package && dev/run-tests ``` Additionally in Spark shell: ``` scala> val jlist = new java.util.LinkedList[Int]; jlist.add(1) jlist: java.util.LinkedList[Int] = [1] res0: Boolean = true scala> Seq(jlist).toDS().map(_.element()).collect() res1: Array[Int] = Array(1) ``` Author: Michal Senkyr <mike.senkyr@gmail.com> Closesapache#18009 from michalsenkyr/dataset-java-lists.
What changes were proposed in this pull request?
Add support for specific Java
Listsubtypes in deserialization as well as a generic implicit encoder.All
Listsubtypes are supported by using either the size-specifying constructor (oneintparameter) or the default constructor.Interfaces/abstract classes use the following implementations:
java.util.List,java.util.AbstractListorjava.util.AbstractSequentialList=>java.util.ArrayListHow was this patch tested?
build/mvn -DskipTests clean package && dev/run-testsAdditionally in Spark shell: