Uh oh!
There was an error while loading. Please reload this page.
[feat](Nereids): Optimize query by pushing down aggregation through join on foreign key - #36035
Merged
Merged
Conversation
doris-robot
commented
Jun 7, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
keanji-x
commented
Jun 7, 2024
ContributorAuthor
run buildall |
keanji-xforce-pushed
the
add_agg_push_foreign
branch
from
June 7, 2024 06:27
ddf7409 to
d4631c4Comparekeanji-x
commented
Jun 7, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jun 7, 2024
TPC-H: Total hot run time: 41103 ms |
doris-robot
commented
Jun 7, 2024
TPC-DS: Total hot run time: 173088 ms |
doris-robot
commented
Jun 7, 2024
ClickBench: Total hot run time: 31.15 s |
keanji-xforce-pushed
the
add_agg_push_foreign
branch
from
June 7, 2024 08:59
d4631c4 to
c2f6800Comparekeanji-x
commented
Jun 7, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jun 7, 2024
TPC-H: Total hot run time: 39773 ms |
doris-robot
commented
Jun 7, 2024
TPC-DS: Total hot run time: 172670 ms |
doris-robot
commented
Jun 7, 2024
ClickBench: Total hot run time: 31.26 s |
keanji-xforce-pushed
the
add_agg_push_foreign
branch
from
June 11, 2024 05:49
c2f6800 to
7fa9d2bComparekeanji-x
commented
Jun 11, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jun 11, 2024
TPC-H: Total hot run time: 39239 ms |
doris-robot
commented
Jun 11, 2024
TPC-DS: Total hot run time: 172819 ms |
doris-robot
commented
Jun 11, 2024
ClickBench: Total hot run time: 30.7 s |
xzj7019
reviewed
Jun 14, 2024
Uh oh!
There was an error while loading. Please reload this page.
xzj7019
reviewed
Jun 14, 2024
Uh oh!
There was an error while loading. Please reload this page.
xzj7019
reviewed
Jun 14, 2024
Uh oh!
There was an error while loading. Please reload this page.
xzj7019
reviewed
Jun 14, 2024
Uh oh!
There was an error while loading. Please reload this page.
keanji-xforce-pushed
the
add_agg_push_foreign
branch
from
June 18, 2024 07:06
7fa9d2b to
c235383Comparekeanji-x
commented
Jun 18, 2024
ContributorAuthor
run buildall |
1 similar comment
keanji-x
commented
Jun 18, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jun 19, 2024
TPC-H: Total hot run time: 39697 ms |
doris-robot
commented
Jun 19, 2024
TPC-DS: Total hot run time: 173455 ms |
doris-robot
commented
Jun 19, 2024
ClickBench: Total hot run time: 30.65 s |
keanji-xforce-pushed
the
add_agg_push_foreign
branch
from
June 21, 2024 07:25
f15731d to
fb93934Comparekeanji-x
commented
Jun 21, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jun 30, 2024
TPC-H: Total hot run time: 39547 ms |
doris-robot
commented
Jun 30, 2024
TPC-DS: Total hot run time: 173561 ms |
doris-robot
commented
Jun 30, 2024
ClickBench: Total hot run time: 31.11 s |
morrySnow
approved these changes
Jul 1, 2024
xzj7019
approved these changes
Jul 1, 2024
dataroaring pushed a commit
that referenced
this pull request
Jul 2, 2024
…in on foreign key (#36035) ## Proposed changes This PR optimizes query performance by pushing down aggregations through joins when grouped by a foreign key. This adjustment reduces data processing overhead above the join, improving both speed and resource efficiency. Transformation Example: Before Optimization: ``` Aggregation(group by fk) | Join(pk = fk) / \ pk fk ``` After Optimization: ``` Join(pk = fk) / \ pk Aggregation(group by fk) | fk ```
keanji-x added a commit
that referenced
this pull request
Jul 8, 2024
#37343) intro by #36035 This PR refines the LogicalJoin class by introducing robust input validation. Key improvements: * Implement precise checks for join input validity * Ensure consistency between input slots and output sets * Gracefully handle various join scenarios (left/right) These enhancements bolster query integrity and optimize join operations.
xinyiZzz added a commit
to xinyiZzz/incubator-doris
that referenced
this pull request
Jul 12, 2024
1. get arrow flight result schema use query id instead of instance id. 2. get arrow flight result is a sync method, need wait for data ready and return result, introduced by apache#36035 36667. TODO, waiting for data will block pipeline, so use a request pool to save requests waiting for data.
dataroaring pushed a commit
that referenced
this pull request
Jul 17, 2024
#37343) intro by #36035 This PR refines the LogicalJoin class by introducing robust input validation. Key improvements: * Implement precise checks for join input validity * Ensure consistency between input slots and output sets * Gracefully handle various join scenarios (left/right) These enhancements bolster query integrity and optimize join operations.
16 tasks
morrySnow pushed a commit
that referenced
this pull request
Jan 9, 2026
Related PR: #36035 Problem Summary: The key of the aggregation must include the primary key of the primary key table (or contain a unique key that can form a bijection with the primary key) to push the aggregation to the foreign key table. Before this pr, doris have wrong results in this situation: drop table if exists customer_test; drop table if exists store_sales_test; CREATE TABLE customer_test ( c_customer_sk INT not null , c_first_name VARCHAR(50), c_last_name VARCHAR(50) ); CREATE TABLE store_sales_test ( ss_customer_sk INT, ss_date DATE ); INSERT INTO customer_test VALUES (1, 'John', 'Smith'); INSERT INTO customer_test VALUES (2, 'John', 'Smith'); INSERT INTO store_sales_test VALUES (1, '2024-01-01'); INSERT INTO store_sales_test VALUES (2, '2024-01-01'); alter table customer_test add constraint c_pk primary key (c_customer_sk); alter table store_sales_test add constraint ss_c_fk foreign key (ss_customer_sk) references customer_test(c_customer_sk); show constraints from customer_test; show constraints from store_sales_test; SELECT DISTINCT c_last_name, c_first_name, ss_date FROM store_sales_test inner join customer_test on store_sales_test.ss_customer_sk = customer_test.c_customer_sk; set disable_nereids_rules='PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK'; set disable_nereids_rules=''; Turn on PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK will have different result with turn off PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK before this pr. This is because AGG (group by c_last_name, c_first_name, ss_date) should not be pushed down below the JOIN operation. The original transform was: Agg(group by c_last_name, c_first_name, ss_date ) +--Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--scan(store_sales_test) -> Join +--scan(customer_test) +--Agg(group by ss_customer_sk,ss_date) +--scan(store_sales_test) This is an incorrect rewrite because it is not equivalent. This pr corrects the rewrite, allowing the aggregation to be pushed down below the join only when there is a bijective relationship between the group by key from the primary table and the fields in the foreign table (a functional dependency exists from a to b, and also from b to a, then a and b have a bijective relationship). For example, Agg(group by c_customer_sk, c_first_name, ss_date ) +--Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--scan(store_sales_test) -> Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--Agg(group by ss_customer_sk,ss_date) +--scan(store_sales_test) Since c_customer_sk is the primary key, c_first_name in the group by clause can be removed (based on functional dependencies). Furthermore, due to the equality relationship c_customer_sk = ss_customer_sk, there is a bijective relationship between c_customer_sk and ss_customer_sk. In this case, `group by c_customer_sk, ss_date` can be replaced with `group by ss_customer_sk, ss_date`. The aggregation group by key is entirely replaced with the output of the foreign table. Since a primary key-foreign key join does not expand the rows of the foreign table,In this situation, the aggregation can be pushed down.
github-actionsBot
pushed a commit
that referenced
this pull request
Jan 9, 2026
Related PR: #36035 Problem Summary: The key of the aggregation must include the primary key of the primary key table (or contain a unique key that can form a bijection with the primary key) to push the aggregation to the foreign key table. Before this pr, doris have wrong results in this situation: drop table if exists customer_test; drop table if exists store_sales_test; CREATE TABLE customer_test ( c_customer_sk INT not null , c_first_name VARCHAR(50), c_last_name VARCHAR(50) ); CREATE TABLE store_sales_test ( ss_customer_sk INT, ss_date DATE ); INSERT INTO customer_test VALUES (1, 'John', 'Smith'); INSERT INTO customer_test VALUES (2, 'John', 'Smith'); INSERT INTO store_sales_test VALUES (1, '2024-01-01'); INSERT INTO store_sales_test VALUES (2, '2024-01-01'); alter table customer_test add constraint c_pk primary key (c_customer_sk); alter table store_sales_test add constraint ss_c_fk foreign key (ss_customer_sk) references customer_test(c_customer_sk); show constraints from customer_test; show constraints from store_sales_test; SELECT DISTINCT c_last_name, c_first_name, ss_date FROM store_sales_test inner join customer_test on store_sales_test.ss_customer_sk = customer_test.c_customer_sk; set disable_nereids_rules='PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK'; set disable_nereids_rules=''; Turn on PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK will have different result with turn off PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK before this pr. This is because AGG (group by c_last_name, c_first_name, ss_date) should not be pushed down below the JOIN operation. The original transform was: Agg(group by c_last_name, c_first_name, ss_date ) +--Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--scan(store_sales_test) -> Join +--scan(customer_test) +--Agg(group by ss_customer_sk,ss_date) +--scan(store_sales_test) This is an incorrect rewrite because it is not equivalent. This pr corrects the rewrite, allowing the aggregation to be pushed down below the join only when there is a bijective relationship between the group by key from the primary table and the fields in the foreign table (a functional dependency exists from a to b, and also from b to a, then a and b have a bijective relationship). For example, Agg(group by c_customer_sk, c_first_name, ss_date ) +--Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--scan(store_sales_test) -> Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--Agg(group by ss_customer_sk,ss_date) +--scan(store_sales_test) Since c_customer_sk is the primary key, c_first_name in the group by clause can be removed (based on functional dependencies). Furthermore, due to the equality relationship c_customer_sk = ss_customer_sk, there is a bijective relationship between c_customer_sk and ss_customer_sk. In this case, `group by c_customer_sk, ss_date` can be replaced with `group by ss_customer_sk, ss_date`. The aggregation group by key is entirely replaced with the output of the foreign table. Since a primary key-foreign key join does not expand the rows of the foreign table,In this situation, the aggregation can be pushed down.
zzzxl1993 pushed a commit
to zzzxl1993/doris
that referenced
this pull request
Jan 13, 2026
…#59498) Related PR: apache#36035 Problem Summary: The key of the aggregation must include the primary key of the primary key table (or contain a unique key that can form a bijection with the primary key) to push the aggregation to the foreign key table. Before this pr, doris have wrong results in this situation: drop table if exists customer_test; drop table if exists store_sales_test; CREATE TABLE customer_test ( c_customer_sk INT not null , c_first_name VARCHAR(50), c_last_name VARCHAR(50) ); CREATE TABLE store_sales_test ( ss_customer_sk INT, ss_date DATE ); INSERT INTO customer_test VALUES (1, 'John', 'Smith'); INSERT INTO customer_test VALUES (2, 'John', 'Smith'); INSERT INTO store_sales_test VALUES (1, '2024-01-01'); INSERT INTO store_sales_test VALUES (2, '2024-01-01'); alter table customer_test add constraint c_pk primary key (c_customer_sk); alter table store_sales_test add constraint ss_c_fk foreign key (ss_customer_sk) references customer_test(c_customer_sk); show constraints from customer_test; show constraints from store_sales_test; SELECT DISTINCT c_last_name, c_first_name, ss_date FROM store_sales_test inner join customer_test on store_sales_test.ss_customer_sk = customer_test.c_customer_sk; set disable_nereids_rules='PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK'; set disable_nereids_rules=''; Turn on PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK will have different result with turn off PUSH_DOWN_AGG_THROUGH_JOIN_ON_PKFK before this pr. This is because AGG (group by c_last_name, c_first_name, ss_date) should not be pushed down below the JOIN operation. The original transform was: Agg(group by c_last_name, c_first_name, ss_date ) +--Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--scan(store_sales_test) -> Join +--scan(customer_test) +--Agg(group by ss_customer_sk,ss_date) +--scan(store_sales_test) This is an incorrect rewrite because it is not equivalent. This pr corrects the rewrite, allowing the aggregation to be pushed down below the join only when there is a bijective relationship between the group by key from the primary table and the fields in the foreign table (a functional dependency exists from a to b, and also from b to a, then a and b have a bijective relationship). For example, Agg(group by c_customer_sk, c_first_name, ss_date ) +--Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--scan(store_sales_test) -> Join(c_customer_sk=ss_customer_sk) +--scan(customer_test) +--Agg(group by ss_customer_sk,ss_date) +--scan(store_sales_test) Since c_customer_sk is the primary key, c_first_name in the group by clause can be removed (based on functional dependencies). Furthermore, due to the equality relationship c_customer_sk = ss_customer_sk, there is a bijective relationship between c_customer_sk and ss_customer_sk. In this case, `group by c_customer_sk, ss_date` can be replaced with `group by ss_customer_sk, ss_date`. The aggregation group by key is entirely replaced with the output of the foreign table. Since a primary key-foreign key join does not expand the rows of the foreign table,In this situation, the aggregation can be pushed down.
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
1. get arrow flight result schema use query id instead of instance id. 2. get arrow flight result is a sync method, need wait for data ready and return result, introduced by apache#36035 36667. TODO, waiting for data will block pipeline, so use a request pool to save requests waiting for data.
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…in on foreign key (apache#36035) ## Proposed changes This PR optimizes query performance by pushing down aggregations through joins when grouped by a foreign key. This adjustment reduces data processing overhead above the join, improving both speed and resource efficiency. Transformation Example: Before Optimization: ``` Aggregation(group by fk) | Join(pk = fk) / \ pk fk ``` After Optimization: ``` Join(pk = fk) / \ pk Aggregation(group by fk) | fk ```
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
apache#37343) intro by apache#36035 This PR refines the LogicalJoin class by introducing robust input validation. Key improvements: * Implement precise checks for join input validity * Ensure consistency between input slots and output sets * Gracefully handle various join scenarios (left/right) These enhancements bolster query integrity and optimize join operations.
16 tasks
englefly added a commit
that referenced
this pull request
Jun 29, 2026
…n PushDownAggThroughJoinOnPkFk (#64848) ### What problem does this PR solve? Related PR: #36035 Problem Summary: COUNT(*) has no arguments; calling child(0) on it throws ArrayIndexOutOfBoundsException. Added arity() > 0 guard before accessing Count's child slot for FK rewrite. Regression test testCountStar() added to verify COUNT(*) with PK/FK join does not crash.
github-actionsBot
pushed a commit
that referenced
this pull request
Jun 29, 2026
…n PushDownAggThroughJoinOnPkFk (#64848) ### What problem does this PR solve? Related PR: #36035 Problem Summary: COUNT(*) has no arguments; calling child(0) on it throws ArrayIndexOutOfBoundsException. Added arity() > 0 guard before accessing Count's child slot for FK rewrite. Regression test testCountStar() added to verify COUNT(*) with PK/FK join does not crash.
github-actionsBot
pushed a commit
that referenced
this pull request
Jun 29, 2026
…n PushDownAggThroughJoinOnPkFk (#64848) ### What problem does this PR solve? Related PR: #36035 Problem Summary: COUNT(*) has no arguments; calling child(0) on it throws ArrayIndexOutOfBoundsException. Added arity() > 0 guard before accessing Count's child slot for FK rewrite. Regression test testCountStar() added to verify COUNT(*) with PK/FK join does not crash.
Merged
16 tasks
morrySnow pushed a commit
that referenced
this pull request
Jul 10, 2026
…ee reconstruction (#65172) ### What problem does this PR solve? Related PR: #36035 Problem Summary: This PR fixes two bugs in the PushDownAggThroughJoinOnPkFk rule that handles pushing aggregates through joins using PK-FK constraints on multi-table joins. **Bug 1: Duplicate join nodes in tree reconstruction** When `constructPlan` rebuilt a subtree from flattened inner joins, the merge path did `currentPlan.withChildren(currentPlan, entryJoin)`, which inserted `currentPlan` as a child of itself and also kept `entryJoin` with its original two children. This produced duplicate join nodes and duplicate leaf references in the resulting plan tree. The fix computes the new leaf set (`newBits = entryBitset - currentBitset`) and only attaches the genuinely new child, using `entryJoin.withChildren(newChild, currentPlan)` instead. **Bug 2: First ineligible PK-FK edge aborts the entire rule** `pushAgg` iterates all flattened PK-FK edges, but when `eliminatePrimaryOutput` returned null for an earlier edge, the code did `return null` — aborting the whole rule instead of continuing to try later edges. Since `HashMap<BitSet>` iteration order on Java 17 visits the lower `{t_primary, t_foreign1}` edge before the root `{t_other_primary, t_primary}` edge, a query aggregating a primary-side column (e.g., `SUM(t_primary.pk_id)`) would fail on the first edge and never reach the valid root edge. Fixed by changing `return null` to `continue`.
github-actionsBot
pushed a commit
that referenced
this pull request
Jul 10, 2026
…ee reconstruction (#65172) ### What problem does this PR solve? Related PR: #36035 Problem Summary: This PR fixes two bugs in the PushDownAggThroughJoinOnPkFk rule that handles pushing aggregates through joins using PK-FK constraints on multi-table joins. **Bug 1: Duplicate join nodes in tree reconstruction** When `constructPlan` rebuilt a subtree from flattened inner joins, the merge path did `currentPlan.withChildren(currentPlan, entryJoin)`, which inserted `currentPlan` as a child of itself and also kept `entryJoin` with its original two children. This produced duplicate join nodes and duplicate leaf references in the resulting plan tree. The fix computes the new leaf set (`newBits = entryBitset - currentBitset`) and only attaches the genuinely new child, using `entryJoin.withChildren(newChild, currentPlan)` instead. **Bug 2: First ineligible PK-FK edge aborts the entire rule** `pushAgg` iterates all flattened PK-FK edges, but when `eliminatePrimaryOutput` returned null for an earlier edge, the code did `return null` — aborting the whole rule instead of continuing to try later edges. Since `HashMap<BitSet>` iteration order on Java 17 visits the lower `{t_primary, t_foreign1}` edge before the root `{t_other_primary, t_primary}` edge, a query aggregating a primary-side column (e.g., `SUM(t_primary.pk_id)`) would fail on the first edge and never reach the valid root edge. Fixed by changing `return null` to `continue`.
github-actionsBot
pushed a commit
that referenced
this pull request
Jul 10, 2026
…ee reconstruction (#65172) ### What problem does this PR solve? Related PR: #36035 Problem Summary: This PR fixes two bugs in the PushDownAggThroughJoinOnPkFk rule that handles pushing aggregates through joins using PK-FK constraints on multi-table joins. **Bug 1: Duplicate join nodes in tree reconstruction** When `constructPlan` rebuilt a subtree from flattened inner joins, the merge path did `currentPlan.withChildren(currentPlan, entryJoin)`, which inserted `currentPlan` as a child of itself and also kept `entryJoin` with its original two children. This produced duplicate join nodes and duplicate leaf references in the resulting plan tree. The fix computes the new leaf set (`newBits = entryBitset - currentBitset`) and only attaches the genuinely new child, using `entryJoin.withChildren(newChild, currentPlan)` instead. **Bug 2: First ineligible PK-FK edge aborts the entire rule** `pushAgg` iterates all flattened PK-FK edges, but when `eliminatePrimaryOutput` returned null for an earlier edge, the code did `return null` — aborting the whole rule instead of continuing to try later edges. Since `HashMap<BitSet>` iteration order on Java 17 visits the lower `{t_primary, t_foreign1}` edge before the root `{t_other_primary, t_primary}` edge, a query aggregating a primary-side column (e.g., `SUM(t_primary.pk_id)`) would fail on the first edge and never reach the valid root edge. Fixed by changing `return null` to `continue`.
morrySnow pushed a commit
that referenced
this pull request
Aug 31, 2026
…ee reconstruction (#65172) ### What problem does this PR solve? Related PR: #36035 Problem Summary: This PR fixes two bugs in the PushDownAggThroughJoinOnPkFk rule that handles pushing aggregates through joins using PK-FK constraints on multi-table joins. **Bug 1: Duplicate join nodes in tree reconstruction** When `constructPlan` rebuilt a subtree from flattened inner joins, the merge path did `currentPlan.withChildren(currentPlan, entryJoin)`, which inserted `currentPlan` as a child of itself and also kept `entryJoin` with its original two children. This produced duplicate join nodes and duplicate leaf references in the resulting plan tree. The fix computes the new leaf set (`newBits = entryBitset - currentBitset`) and only attaches the genuinely new child, using `entryJoin.withChildren(newChild, currentPlan)` instead. **Bug 2: First ineligible PK-FK edge aborts the entire rule** `pushAgg` iterates all flattened PK-FK edges, but when `eliminatePrimaryOutput` returned null for an earlier edge, the code did `return null` — aborting the whole rule instead of continuing to try later edges. Since `HashMap<BitSet>` iteration order on Java 17 visits the lower `{t_primary, t_foreign1}` edge before the root `{t_other_primary, t_primary}` edge, a query aggregating a primary-side column (e.g., `SUM(t_primary.pk_id)`) would fail on the first edge and never reach the valid root edge. Fixed by changing `return null` to `continue`.
morrySnow pushed a commit
that referenced
this pull request
Aug 31, 2026
…n PushDownAggThroughJoinOnPkFk (#64848) ### What problem does this PR solve? Related PR: #36035 Problem Summary: COUNT(*) has no arguments; calling child(0) on it throws ArrayIndexOutOfBoundsException. Added arity() > 0 guard before accessing Count's child slot for FK rewrite. Regression test testCountStar() added to verify COUNT(*) with PK/FK join does not crash.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
This PR optimizes query performance by pushing down aggregations through joins when grouped by a foreign key. This adjustment reduces data processing overhead above the join, improving both speed and resource efficiency.
Transformation Example:
Before Optimization:
After Optimization: