Uh oh!
There was an error while loading. Please reload this page.
fix: list_tables method in glue catalog now only return tables. - #1258
Conversation
omkenge
commented
Oct 28, 2024
Hi Team , @kevinjqliu@sungwy Error Details:When views are included in the output of list_tables and subsequently processed, the absence of the table_type parameter often raises an error similar to: |
Fokko
commented
Oct 30, 2024
Hey @omkenge thanks for pointing this out. I was unaware of this issue. Are you interested in providing a patch to fix this? |
Hi @Fokko |
HonahX
left a comment
There was a problem hiding this comment.
@omkenge Thanks for reporting the issue and providing the fix!
NoSuchPropertyException: Property 'table_type' missing, could not determine type
I was able to reproduce this issue and verify that this PR can fix it.
Let's get this into 0.8.0!
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Honah J. <undefined.newdb.newtable@gmail.com>
omkenge
commented
Oct 31, 2024
HI @HonahX |
kevinjqliu
commented
Oct 31, 2024
Thanks for the contribution @omkenge ! |
…he#1258) * fix: list_tables method only return tables * Update pyiceberg/catalog/glue.py Co-authored-by: Honah J. <undefined.newdb.newtable@gmail.com> --------- Co-authored-by: Honah J. <undefined.newdb.newtable@gmail.com>
…he#1258) * fix: list_tables method only return tables * Update pyiceberg/catalog/glue.py Co-authored-by: Honah J. <undefined.newdb.newtable@gmail.com> --------- Co-authored-by: Honah J. <undefined.newdb.newtable@gmail.com>
Description:
This pull request addresses an issue where the list_tables function was returning views alongside Iceberg tables. Since views lack the table_type property or have it set differently than "iceberg", they were incorrectly included in the list of tables. This change modifies the filtering logic to ensure that only actual Iceberg tables are returned.
Problem:
In the existing list_tables function, both Iceberg tables and Glue views were being returned. Glue views generally do not contain the table_type parameter or may have it unset, which causes them to be mistakenly included when listing Iceberg tables in a namespace. This behavior makes it challenging for users to retrieve only Iceberg tables as intended.
Solution:
The __is_iceberg_table helper method has been modified to:
Check if the table_type parameter is present in each table’s properties.
Ensure that the table_type parameter is explicitly set to "iceberg".
Testing:
Tests were added to verify that:
Only Iceberg tables are returned by list_tables.
Glue views, which lack the table_type parameter or have it set to a non-Iceberg value, are excluded from the list.