Skip to content

[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table - #4792

Closed
liancheng wants to merge 4 commits into
apache:masterfrom
liancheng:spark-5775
Closed

[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutableRow when nested data and partitioned table#4792
liancheng wants to merge 4 commits into
apache:masterfrom
liancheng:spark-5775

Conversation

@liancheng

@lianchengliancheng commented Feb 26, 2015

Copy link
Copy Markdown
Contributor

This PR adapts @anselmevignon's #4697 to master and branch-1.3. Please refer to PR description of #4697 for details.

Review on Reviewable

@SparkQA

SparkQA commented Feb 26, 2015

Copy link
Copy Markdown

Test build #28012 has started for PR 4792 at commit ca6e038.

  • This patch merges cleanly.

@liancheng

liancheng commented Feb 26, 2015

Copy link
Copy Markdown
ContributorAuthor

@marmbrus As discussed offline, I tried to use SpecificMutableRow throughout the whole Parquet support implementation, but it touches to many places, which I think is too risky as 1.3 release is so close. So falled back to current implementation. I can do the refactoring later on master (mainly focus on the CatalystConverter class hierarchy).

@lianchengliancheng mentioned this pull request Feb 26, 2015
Closed
@marmbrus

marmbrus commented Feb 26, 2015

Copy link
Copy Markdown
Contributor

Can't we do the same check that we do inside of parquet to see if the row is all primitives once and switch on that, instead of doing per tuple extraction and run time type checking?

@SparkQA

SparkQA commented Feb 26, 2015

Copy link
Copy Markdown

Test build #28012 has finished for PR 4792 at commit ca6e038.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 26, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28012/
Test PASSed.

@yhuaiyhuai mentioned this pull request Feb 27, 2015
Closed
@yhuai

yhuai commented Feb 27, 2015

Copy link
Copy Markdown
Contributor

liancheng#4

@liancheng

liancheng commented Feb 27, 2015

Copy link
Copy Markdown
ContributorAuthor

@marmbrus@yhuai Thanks for the suggestion and PR, much better!

@liancheng
Merge pull request#4from yhuai/spark-5775-yin
cee55cf
Remove runtime pattern matching.
@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28039 has started for PR 4792 at commit cee55cf.

  • This patch merges cleanly.

@SparkQA

SparkQA commented Feb 27, 2015

Copy link
Copy Markdown

Test build #28039 has finished for PR 4792 at commit cee55cf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 27, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28039/
Test PASSed.

yhuai
yhuai reviewed Feb 27, 2015
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTableOperations.scala Outdated

@yhuaiyhuaiFeb 27, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made a typo at here... if => is.

@SparkQA

SparkQA commented Feb 28, 2015

Copy link
Copy Markdown

Test build #28117 has started for PR 4792 at commit 538f506.

  • This patch merges cleanly.

@liancheng

liancheng commented Feb 28, 2015

Copy link
Copy Markdown
ContributorAuthor

@yhuai Thanks for the review! I've addressed the comments. Will merge this to master and branch-1.3 after Jenkins approves.

@SparkQA

SparkQA commented Feb 28, 2015

Copy link
Copy Markdown

Test build #28117 has finished for PR 4792 at commit 538f506.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins

AmplabJenkins commented Feb 28, 2015

Copy link
Copy Markdown

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28117/
Test PASSed.

asfgit pushed a commit that referenced this pull request Feb 28, 2015
@liancheng
[SPARK-5775] [SQL] BugFix: GenericRow cannot be cast to SpecificMutab…
aa39460
…leRow when nested data and partitioned table
This PR adapts anselmevignon's #4697 to master and branch-1.3. Please refer to PR description of #4697 for details.
<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/4792)
<!-- Reviewable:end -->
Author: Cheng Lian <lian@databricks.com>
Author: Cheng Lian <liancheng@users.noreply.github.com>
Author: Yin Huai <yhuai@databricks.com>
Closes#4792 from liancheng/spark-5775 and squashes the following commits:
538f506 [Cheng Lian] Addresses comments
cee55cf [Cheng Lian] Merge pull request #4 from yhuai/spark-5775-yin
b0b74fb [Yin Huai] Remove runtime pattern matching.
ca6e038 [Cheng Lian] Fixes SPARK-5775
(cherry picked from commit e6003f0)
Signed-off-by: Cheng Lian <lian@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@liancheng@SparkQA@marmbrus@AmplabJenkins@yhuai