Uh oh!
There was an error while loading. Please reload this page.
[SPARK-2674] [SQL] [PySpark] support datetime type for SchemaRDD - #1601
[SPARK-2674] [SQL] [PySpark] support datetime type for SchemaRDD#1601davies wants to merge 4 commits into
Conversation
SparkQA
commented
Jul 26, 2014
QA tests have started for PR 1601. This patch merges cleanly. |
SparkQA
commented
Jul 26, 2014
QA results for PR 1601: |
There was a problem hiding this comment.
This pair of extra brackets can be removed.
SparkQA
commented
Jul 27, 2014
QA tests have started for PR 1601. This patch merges cleanly. |
SparkQA
commented
Jul 27, 2014
QA results for PR 1601: |
java.util.{List,Set} => Seq
java.util.Map => Map
but it can not convert Seq into java.util.Set, so set() and tuple()
and array() can not been handled gracefully (back with the original
type).
We can not access items in ArrayType by position, but this is not defined
for set().
Do we still want to support set()/tuple()/array() ?SparkQA
commented
Jul 28, 2014
QA tests have started for PR 1601. This patch merges cleanly. |
SparkQA
commented
Jul 28, 2014
QA results for PR 1601: |
davies
commented
Jul 28, 2014
marmbrus
commented
Jul 28, 2014
Lets just remove it now. It should be as easy as adding an error and removing the tests in question. |
SparkQA
commented
Jul 28, 2014
QA tests have started for PR 1601. This patch merges cleanly. |
SparkQA
commented
Jul 28, 2014
QA results for PR 1601: |
davies
commented
Jul 29, 2014
cc @kanzhang |
marmbrus
commented
Jul 29, 2014
I've merged this into master. |
Datetime and time in Python will be converted into java.util.Calendar after serialization, it will be converted into java.sql.Timestamp during inferSchema(). In javaToPython(), Timestamp will be converted into Calendar, then be converted into datetime in Python after pickling. Author: Davies Liu <davies.liu@gmail.com> Closesapache#1601 from davies/date and squashes the following commits: f0599b0 [Davies Liu] remove tests for sets and tuple in sql, fix list of list c9d607a [Davies Liu] convert datetype for runtime 709d40d [Davies Liu] remove brackets 96db384 [Davies Liu] support datetime type for SchemaRDD
Datetime and time in Python will be converted into java.util.Calendar after serialization, it will be converted into java.sql.Timestamp during inferSchema().
In javaToPython(), Timestamp will be converted into Calendar, then be converted into datetime in Python after pickling.