Skip to content

[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incompatible schemas. - #10260

Closed
nongli wants to merge 3 commits into
apache:masterfrom
nongli:spark-11271
Closed

[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incompatible schemas.#10260
nongli wants to merge 3 commits into
apache:masterfrom
nongli:spark-11271

Conversation

@nongli

@nonglinongli commented Dec 11, 2015

Copy link
Copy Markdown
Contributor

No description provided.

@nonglinongli changed the title [SPARK-12271][SQL] Improve error message when Dataset.as has incompatible schemas.[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incompatible schemas.Dec 11, 2015
@SparkQA

SparkQA commented Dec 11, 2015

Copy link
Copy Markdown

Test build #47560 has finished for PR 10260 at commit ceed59b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Dec 11, 2015
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala Outdated

@cloud-fancloud-fanDec 11, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the simpleString of attribute includes expr id, which I think is noisy to users, should we generate the string manually or use StructType.fromAttributes to turn schema into StructType and call StrcutType.simpleString?

@nonglinongliDec 14, 2015

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@cloud-fancloud-fan mentioned this pull request Dec 11, 2015
Closed
@holdenk

holdenk commented Dec 14, 2015

Copy link
Copy Markdown
Contributor

Could you maybe update the PR description to include the description from the JIRA?

@SparkQA

SparkQA commented Dec 14, 2015

Copy link
Copy Markdown

Test build #47680 has finished for PR 10260 at commit f177d55.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Dec 15, 2015
@cloud-fan@marmbrus
[SPARK-12274][SQL] WrapOption should not have type constraint for child
9ea1a8e
I think it was a mistake, and we have not catched it so far until #10260 which begin to check if the `fromRowExpression` is resolved.
Author: Wenchen Fan <wenchen@databricks.com>
Closes#10263 from cloud-fan/encoder.
marmbrus
marmbrus reviewed Dec 15, 2015
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala Outdated

@marmbrusmarmbrusDec 15, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason to not just put this in CheckAnalysis and make it more general (i.e. not just Project)? I realize its only used here now, but if we ever do use it elsewhere, its invalid to remain after analysis.

@nonglinongliDec 15, 2015

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

much better!

@marmbrus

marmbrus commented Dec 15, 2015

Copy link
Copy Markdown
Contributor

This looks great now!

LGTM pending tests / conflict resolution

@SparkQA

SparkQA commented Dec 15, 2015

Copy link
Copy Markdown

Test build #47747 has finished for PR 10260 at commit 359b3e2.

  • This patch passes all tests.
  • This patch does not merge cleanly.
  • This patch adds the following public classes (experimental):\n * case class WrapOption(child: Expression, optType: DataType)\n

@SparkQA

SparkQA commented Dec 15, 2015

Copy link
Copy Markdown

Test build #47754 has finished for PR 10260 at commit b5007d7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * case class WrapOption(child: Expression, optType: DataType)\n

@cloud-fan

cloud-fan commented Dec 16, 2015

Copy link
Copy Markdown
Contributor

LGTM

@marmbrus

marmbrus commented Dec 16, 2015

Copy link
Copy Markdown
Contributor

Thanks, merging to master.

@nongli
nongli deleted the spark-11271 branch December 16, 2015 05:04
marmbrus pushed a commit to marmbrus/spark that referenced this pull request Jan 7, 2016
@cloud-fan@marmbrus
[SPARK-12274][SQL] WrapOption should not have type constraint for child
c938a43
I think it was a mistake, and we have not catched it so far until apache#10260 which begin to check if the `fromRowExpression` is resolved.
Author: Wenchen Fan <wenchen@databricks.com>
Closesapache#10263 from cloud-fan/encoder.
marmbrus pushed a commit to marmbrus/spark that referenced this pull request Jan 7, 2016
@nongli@marmbrus
[SPARK-12271][SQL] Improve error message when Dataset.as[ ] has incom…
862cb12
…patible schemas.
Author: Nong Li <nong@databricks.com>
Closesapache#10260 from nongli/spark-11271.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@nongli@SparkQA@holdenk@marmbrus@cloud-fan