Skip to content

[SPARK-27690][SQL] Remove materialized views first in HiveClientImpl.reset - #24592

Closed
wangyum wants to merge 2 commits into
apache:masterfrom
wangyum:SPARK-27690
Closed

[SPARK-27690][SQL] Remove materialized views first in HiveClientImpl.reset#24592
wangyum wants to merge 2 commits into
apache:masterfrom
wangyum:SPARK-27690

Conversation

@wangyum

@wangyumwangyum commented May 13, 2019

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

We should remove materialized view first otherwise(note that Hive 3.1 could reproduce this issue):

Cause: org.apache.derby.shared.common.error.DerbySQLIntegrityConstraintViolationException:DELETE on table 'TBLS' caused a violation of foreign key constraint 'MV_TABLES_USED_FK2'for key (4). The statement has been rolled back.
at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(UnknownSource)
at org.apache.derby.impl.jdbc.Util.generateCsSQLException(UnknownSource)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(UnknownSource)
at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(UnknownSource)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(UnknownSource)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(UnknownSource)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(UnknownSource)
at org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeBatchElement(UnknownSource)
at org.apache.derby.impl.jdbc.EmbedStatement.executeLargeBatch(UnknownSource)

How was this patch tested?

Existing test

dongjoon-hyun
dongjoon-hyun reviewed May 13, 2019
View reviewed changes
Comment threadsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala Outdated
@SparkQA

SparkQA commented May 13, 2019

Copy link
Copy Markdown

Test build #105361 has finished for PR 24592 at commit 6ee5b7a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyundongjoon-hyun changed the title [SPARK-27690][SQL] Refactor HiveClientImpl#reset() to remove materialized view first[SPARK-27690][SQL] Remove materialized view first in HiveClientImpl.resetMay 13, 2019
@dongjoon-hyundongjoon-hyun changed the title [SPARK-27690][SQL] Remove materialized view first in HiveClientImpl.reset[SPARK-27690][SQL] Remove materialized views first in HiveClientImpl.resetMay 13, 2019
@dongjoon-hyun

dongjoon-hyun commented May 13, 2019

Copy link
Copy Markdown
Member

Thank you for updating, @wangyum !

@SparkQA

SparkQA commented May 13, 2019

Copy link
Copy Markdown

Test build #105368 has finished for PR 24592 at commit 953194d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes May 14, 2019
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Merged to master. Thank you, @wangyum .
cc @gatorsmile .

gatorsmile
gatorsmile reviewed May 15, 2019
View reviewed changes
Comment threadsql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
val (mvs, others) = allTables.asScala.map(t => client.getTable("default", t))
.partition(_.getTableType.toString.equals("MATERIALIZED_VIEW"))

// Remove materialized view first, otherwise caused a violation of foreign key constraint.

@gatorsmilegatorsmileMay 15, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have the same issue for Hive indexes? Also, we normally also added a test case in the Version Suite. You can do it by using the method runSqlHive(sql: String) to create materialized views.

@wangyumwangyumMay 15, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test in another PR: #23992

@wangyumwangyumMay 16, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue only occurs with materialized views in Hive 3.1.
I have tested the index and the view.

@wangyum
wangyum deleted the SPARK-27690 branch May 16, 2019 13:01
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatorsmilegatorsmilegatorsmile left review comments
@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@wangyum@SparkQA@dongjoon-hyun@gatorsmile