Uh oh!
There was an error while loading. Please reload this page.
[fix](nereids)Solve the problem of pruning wrong partitions in multi-column partition pruning - #43332
Merged
morrySnow merged 3 commits intoNov 12, 2024
Conversation
doris-robot
commented
Nov 6, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
feiniaofeiafei
commented
Nov 6, 2024
CollaboratorAuthor
run buildall |
1 similar comment
feiniaofeiafei
commented
Nov 6, 2024
CollaboratorAuthor
run buildall |
morrySnow
previously approved these changes
Nov 7, 2024
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
feiniaofeiafei
commented
Nov 7, 2024
CollaboratorAuthor
run buildall |
morrySnow
commented
Nov 7, 2024
Contributor
run p0 |
1 similar comment
feiniaofeiafei
commented
Nov 9, 2024
CollaboratorAuthor
run p0 |
feiniaofeiafeiforce-pushed
the
fix_multi_column_partition_prune_with_func
branch
from
November 11, 2024 02:14
85f09cd to
cf9dd4cComparefeiniaofeiafei
commented
Nov 11, 2024
CollaboratorAuthor
run buildall |
morrySnow
approved these changes
Nov 11, 2024
Contributor
PR approved by at least one committer and no changes requested. |
924060929
approved these changes
Nov 12, 2024
github-actionsBot
pushed a commit
that referenced
this pull request
Nov 12, 2024
…column partition pruning (#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
github-actionsBot
pushed a commit
that referenced
this pull request
Nov 12, 2024
…column partition pruning (#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 12, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
feiniaofeiafei added a commit
to feiniaofeiafei/doris
that referenced
this pull request
Nov 12, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
zzzxl1993 pushed a commit
to zzzxl1993/doris
that referenced
this pull request
Nov 12, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
py023 pushed a commit
to py023/doris
that referenced
this pull request
Nov 13, 2024
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
Closed
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…column partition pruning (apache#43332) For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate： WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to: a = 0, dt in ['2024-01-01 00:00:00', +∞) a = 1, dt in (-∞, +∞) a = 2, dt in (-∞, +∞) ... a = 10, dt in (-∞, '2024-01-10 00:00:00') Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned. During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day'). Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition. The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
For example, with a partition defined as PARTITION BY RANGE (a, dt) [(0, '2024-01-01 00:00:00'), (10, '2024-01-10 00:00:00')). With the predicate WHERE a = 0 AND date_trunc(dt, 'day') <= '2024-01-10 00:00:00', partition pruning will expand the partition ranges to:
a = 0, dt in ['2024-01-01 00:00:00', +∞)
a = 1, dt in (-∞, +∞)
a = 2, dt in (-∞, +∞)
...
a = 10, dt in (-∞, '2024-01-10 00:00:00')
Each of these eleven ranges will be evaluated against the predicate. If all evaluations return False, the partition can be pruned.
During the evaluation of the first range (a = 0, dt in ['2024-01-01 00:00:00', +∞)), the range of date_trunc(dt, 'day') is calculated as ['2024-01-01', +∞) and stored in rangeMap. However, subsequent evaluations (e.g., for a = 2, dt in (-∞, +∞)) reuse this range ['2024-01-01', +∞), which is incorrect. For a = 2, the correct range should be (-∞, +∞) for date_trunc(dt, 'day').
Due to this incorrect reuse, the range for a = 2, dt in (-∞, +∞) will incorrectly evaluate to False, causing improper pruning of the partition.
The correct approach is to place rangeMap within the context, so that a new rangeMap is constructed for each evaluation.
Issue Number: close #xxx
Related PR: introduced by #38849
Problem Summary:
Check List (For Committer)
Test
Behavior changed:
Does this need documentation?
Release note
None
Check List (For Reviewer who merge this PR)