Skip to content

Hive: Avoid drop table related exceptions in MetaHook - #2191

Merged
pvary merged 7 commits into
apache:masterfrom
marton-bod:drop_table_fix
Feb 5, 2021
Merged

Hive: Avoid drop table related exceptions in MetaHook#2191
pvary merged 7 commits into
apache:masterfrom
marton-bod:drop_table_fix

Conversation

@marton-bod

@marton-bodmarton-bod commented Feb 1, 2021•
edited
Loading

Copy link
Copy Markdown
Collaborator

There are a couple of scenarios where the Hive DDL command DROP TABLE can produce an error unnecessarily.

  1. Currently on Hive4, DROP TABLE can result in a FileNotFoundException exception, because based on Hive2/Hive3 behaviour, the current assumption is that the HiveIcebergMetaHook has to clean up the metadata files after the Hive drop table has completed. However, in Hive4, if external.table.purge=TRUE is set, Hive will wipe the metadata folder as well.
  2. When creating a Hive table on top of an existing table in a different catalog, if we drop the table in Hive but the underlying table does not exist anymore (e.g. in the HadoopCatalog), we'd also end up with an exception.

Extended some of the unit tests with explicit DROP TABLE commands in the end to test for the whole flow of CREATE, INSERT, DROP (which currently triggers problem # 1 on Hive4).

@marton-bod

marton-bod commented Feb 3, 2021

Copy link
Copy Markdown
CollaboratorAuthor

@pvary Can you please take a look? Thanks!

pvary
pvary reviewed Feb 3, 2021
View reviewed changes
Comment threadmr/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergMetaHook.java Outdated
pvary
pvary reviewed Feb 3, 2021
View reviewed changes
Comment threadmr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerLocalScan.java Outdated
pvary
pvary reviewed Feb 4, 2021
View reviewed changes
Comment threadmr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java Outdated
pvary
pvary reviewed Feb 4, 2021
View reviewed changes
Comment threadmr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java Outdated
pvary
pvary reviewed Feb 4, 2021
View reviewed changes
Comment threadmr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java Outdated
pvary
pvary reviewed Feb 4, 2021
View reviewed changes
Comment threadmr/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergStorageHandlerNoScan.java Outdated
pvary
pvary approved these changes Feb 4, 2021
View reviewed changes

@pvarypvary left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a minor comment, but LGTM

@marton-bod

marton-bod commented Feb 4, 2021

Copy link
Copy Markdown
CollaboratorAuthor

Thanks for your review @pvary!

@pvary
pvary merged commit 469f6c3 into apache:masterFeb 5, 2021
@pvary

pvary commented Feb 5, 2021

Copy link
Copy Markdown
Contributor

Thanks for the fix @marton-bod

@pvarypvary added this to the Java 0.11.1 Release milestone Feb 18, 2021
aokolnychyi pushed a commit that referenced this pull request Mar 24, 2021
@marton-bod@aokolnychyi
Hive: Avoid drop table related exceptions in MetaHook (#2191)
cb0e587
coolderli pushed a commit to coolderli/iceberg that referenced this pull request Apr 26, 2021
@marton-bod@coolderli
Hive: Avoid drop table related exceptions in MetaHook (apache#2191)
b5a7961
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pvarypvarypvary approved these changes

Assignees

No one assigned

Labels

MR

Projects

None yet

Milestone

Java 0.11.1 Release

Development

Successfully merging this pull request may close these issues.

2 participants

@marton-bod@pvary