Skip to content

[SPARK-20431][SQL] Specify a schema by using a DDL-formatted string - #17719

Closed
maropu wants to merge 4 commits into
apache:masterfrom
maropu:SPARK-20431
Closed

[SPARK-20431][SQL] Specify a schema by using a DDL-formatted string#17719
maropu wants to merge 4 commits into
apache:masterfrom
maropu:SPARK-20431

Conversation

@maropu

@maropumaropu commented Apr 21, 2017•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This pr supported a DDL-formatted string in DataFrameReader.schema.
This fix could make users easily define a schema without importing o.a.spark.sql.types._.

How was this patch tested?

Added tests in DataFrameReaderWriterSuite.

@SparkQA

SparkQA commented Apr 21, 2017

Copy link
Copy Markdown

Test build #76034 has finished for PR 17719 at commit 8447a6d.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Apr 21, 2017

Copy link
Copy Markdown

Test build #76037 has finished for PR 17719 at commit a1a2e35.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu

maropu commented Apr 22, 2017

Copy link
Copy Markdown
MemberAuthor

cc: @gatorsmile

gatorsmile
gatorsmile reviewed Apr 22, 2017
View reviewed changes
Comment threadpython/pyspark/sql/readwriter.py Outdated
elif isinstance(schema, basestring):
self._jreader = self._jreader.schema(schema)
else:
raise TypeError("schema should be StructType")

@gatorsmilegatorsmileApr 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update this message?

@maropumaropuApr 22, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, I'll do

gatorsmile
gatorsmile reviewed Apr 22, 2017
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
*
* @since 2.3.0
*/
def schema(schemaString: String): DataFrameReader = {

@gatorsmilegatorsmileApr 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change will make PySpark API inconsistent with the Scala API

@maropumaropuApr 22, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, but I probably missed your point. What's the API consistency you pointed out here?
I just made the python APIs the same with the Scala ones like:

--- python
>>> from pyspark.sql.types import *
>>> fields = [StructField('a', IntegerType(), True), StructField('b', StringType(), True), StructField('c', DoubleType(), True)]
>>> schema = StructType(fields)
>>> spark.read.schema(schema).csv("/Users/maropu/Desktop/test.csv").show()
+---+----+---+
| a| b| c|
+---+----+---+
| 1| aaa|0.3|
+---+----+---+
>>> spark.read.schema("a INT, b STRING, c DOUBLE").csv("/Users/maropu/Desktop/test.csv").show()
+---+----+---+
| a| b| c|
+---+----+---+
| 1| aaa|0.3|
+---+----+---+
--- scala
scala> import org.apache.spark.sql.types._
scala> fields = StructField("a", IntegerType) :: StructField("b", StringType) :: StructField("c", DoubleType) :: Nil
scala> val schema = StructType(fields)
scala> spark.read.schema(schema).csv("/Users/maropu/Desktop/test.csv").show
+---+----+---+
| a| b| c|
+---+----+---+
| 1| aaa|0.3|
+---+----+---+
scala> spark.read.schema("a INT, b STRING, c DOUBLE").csv("/Users/maropu/Desktop/test.csv").show
+---+----+---+
| a| b| c|
+---+----+---+
| 1| aaa|0.3|
+---+----+---+

@gatorsmilegatorsmileApr 22, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I misread the Python codes.

@SparkQA

SparkQA commented Apr 22, 2017

Copy link
Copy Markdown

Test build #76061 has finished for PR 17719 at commit 5fe5e39.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu

maropu commented Apr 26, 2017

Copy link
Copy Markdown
MemberAuthor

@gatorsmile ping

1 similar comment
@maropu

maropu commented May 10, 2017

Copy link
Copy Markdown
MemberAuthor

@gatorsmile ping

gatorsmile
gatorsmile reviewed May 10, 2017
View reviewed changes
Comment threadpython/pyspark/sql/readwriter.py
inference step, and thus speed up data loading.

:param schema: a :class:`pyspark.sql.types.StructType` object
:param schema: a :class:`pyspark.sql.types.StructType` object or a DDL-formatted string

@gatorsmilegatorsmileMay 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you give an example here to users?

@maropumaropuMay 11, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@gatorsmile

gatorsmile commented May 10, 2017

Copy link
Copy Markdown
Member

In PySpark, we have multiple ways to specify the schema. However, it sounds like we silently pick one of the input schemas. Could you submit a separate PR to fix the potential schema conflicts in PySpark?

Thanks!

@maropu

maropu commented May 11, 2017

Copy link
Copy Markdown
MemberAuthor

Sure, I'd love to do tough, I probably missed your point. What's the scenario of the conflict you described? The current logic in readwriter.py just checks types, then decides a schema. Could you give me a concrete example? Thanks

@SparkQA

SparkQA commented May 11, 2017

Copy link
Copy Markdown

Test build #76773 has finished for PR 17719 at commit cc3afd7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented May 11, 2017

Copy link
Copy Markdown

Test build #76774 has finished for PR 17719 at commit 46994fb.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented May 11, 2017

Copy link
Copy Markdown
Member

nvm. I will fix it later. Thanks!

@gatorsmile

gatorsmile commented May 11, 2017

Copy link
Copy Markdown
Member

LGTM

@gatorsmile

gatorsmile commented May 11, 2017

Copy link
Copy Markdown
Member

Thanks! Merging to master.

robert3005 pushed a commit to palantir/spark that referenced this pull request May 19, 2017
@maropuRobert Kruszewski
[SPARK-20431][SQL] Specify a schema by using a DDL-formatted string
a8d25b6
## What changes were proposed in this pull request?
This pr supported a DDL-formatted string in `DataFrameReader.schema`.
This fix could make users easily define a schema without importing `o.a.spark.sql.types._`.
## How was this patch tested?
Added tests in `DataFrameReaderWriterSuite`.
Author: Takeshi Yamamuro <yamamuro@apache.org>
Closesapache#17719 from maropu/SPARK-20431.
lycplus pushed a commit to lycplus/spark that referenced this pull request May 24, 2017
@maropu@lycplus
[SPARK-20431][SQL] Specify a schema by using a DDL-formatted string
2d26b95
## What changes were proposed in this pull request?
This pr supported a DDL-formatted string in `DataFrameReader.schema`.
This fix could make users easily define a schema without importing `o.a.spark.sql.types._`.
## How was this patch tested?
Added tests in `DataFrameReaderWriterSuite`.
Author: Takeshi Yamamuro <yamamuro@apache.org>
Closesapache#17719 from maropu/SPARK-20431.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@maropu@SparkQA@gatorsmile