Uh oh!
There was an error while loading. Please reload this page.
HIVE-28268: Iceberg: Retrieve row count from iceberg SnapshotSummary in case of iceberg.hive.keep.stats=false - #5215
Merged
Conversation
zhangbutao
marked this pull request as draft
April 25, 2024 14:47
zhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
April 26, 2024 04:45
0c14207 to
1a953e8Comparezhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
April 28, 2024 09:22
1a953e8 to
77d9a7eComparezhangbutao
commented
Apr 28, 2024
| filterExpr: (a = 22) (type: boolean) | ||
| Snapshot ref: branch_test1 | ||
| Statistics: Num rows: 3 Data size: 291 Basic stats: COMPLETE Column stats: COMPLETE | ||
| Statistics: Num rows: 5 Data size: 485 Basic stats: COMPLETE Column stats: COMPLETE |
ContributorAuthor
There was a problem hiding this comment.
Before this PR, we always get row count of branch/tag/timetravel by the current snapshot summary, which is not right.
zhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
April 28, 2024 09:44
77d9a7e to
441db00Comparezhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
April 29, 2024 05:15
441db00 to
9971db5Comparezhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
April 30, 2024 02:34
9971db5 to
0ffc9dfComparezhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
October 11, 2024 15:04
1347fef to
7b2ffb2CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
zhangbutaoforce-pushed
the
iceberg_count_optimize
branch
from
October 25, 2024 06:52
7b2ffb2 to
deb46ffComparedeniskuzZ
approved these changes
Oct 25, 2024
Member
There was a problem hiding this comment.
LGTM +1, pending tests
thanks @zhangbutao for addressing the review comment quickly!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
At present, in case of
iceberg.hive.keep.stats=true&hive.compute.query.using.stats=true, HS2 will do a fetch task to get iceberg table'snumRowsproperty from HMS to optimizecountquery.If
iceberg.hive.keep.stats=false, HS2 will always launch tez task to compute table's row count when filing acountquery.However, as we know, iceberg table's metadata has some stats information, we can also just start a fetch task to retrieve the row count from iceberg's snapshot summary when
iceberg.hive.keep.stats=falseor no stats stored in hms. This can avoid launching tez task to compute the table's row count.BTW, timetravel or branch/tag has different stats from current snapshot, so we need to get the specified snapshotid based on the different iceberg version. Otherwise, we will get the wrong stats when querying the time travel/branch/tag.
Why are the changes needed?
Does this PR introduce any user-facing change?
No
Is the change a dependency upgrade?
No
How was this patch tested?
Qtest