Skip to content

[fix](Nereids) could not push down filter through cte producer sometimes - #35463

Merged
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix_cte_push_filter
May 28, 2024
Merged

[fix](Nereids) could not push down filter through cte producer sometimes#35463
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:fix_cte_push_filter

Conversation

@morrySnow

@morrySnowmorrySnow commented May 27, 2024

Copy link
Copy Markdown
Contributor

when consumer side eliminate some consumers from plan, the size of consumers is wrong. so we cannot push down some filter in producer side. this PR fix this problem by update consumer set after rewrite outer side

Issue Number: close#35379

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
[fix](Nereids) could not push down filter through cte producer sometimes
5af7e9d
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
@morrySnow

morrySnow commented May 27, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morrySnow
morrySnowforce-pushed the fix_cte_push_filter branch from a19dc31 to 5af7e9dCompareMay 27, 2024 10:33
@morrySnow

morrySnow commented May 27, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 41921 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5af7e9df57d8e3a85af2e633cb3563244eb5ea1a, data reload: false
------ Round 1 ----------------------------------
q1	17619	4830	4257	4257
q2	2017	192	192	192
q3	10470	1169	1134	1134
q4	10199	803	816	803
q5	7468	2763	2674	2674
q6	222	132	128	128
q7	962	613	626	613
q8	9214	2154	2094	2094
q9	9624	6676	6709	6676
q10	9230	3892	3899	3892
q11	455	242	264	242
q12	427	228	225	225
q13	17458	3203	3236	3203
q14	252	216	219	216
q15	508	467	471	467
q16	505	407	422	407
q17	983	669	722	669
q18	8436	7770	7899	7770
q19	7072	1594	1541	1541
q20	654	310	326	310
q21	5208	4125	4159	4125
q22	370	283	291	283
Total cold run time: 119353 ms
Total hot run time: 41921 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4501	4427	4446	4427
q2	380	254	274	254
q3	3184	2938	2778	2778
q4	1887	1685	1664	1664
q5	5478	5503	5499	5499
q6	220	127	127	127
q7	2212	1855	1823	1823
q8	3296	3350	3398	3350
q9	8705	8701	8560	8560
q10	3940	3820	3836	3820
q11	616	523	506	506
q12	784	623	645	623
q13	16079	3157	3172	3157
q14	308	257	269	257
q15	524	476	480	476
q16	496	408	420	408
q17	1792	1488	1466	1466
q18	7687	7468	7359	7359
q19	1666	1596	1567	1567
q20	2018	1782	1782	1782
q21	7645	4877	4823	4823
q22	584	503	495	495
Total cold run time: 74002 ms
Total hot run time: 55221 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 170148 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5af7e9df57d8e3a85af2e633cb3563244eb5ea1a, data reload: false
query1	922	382	371	371
query2	6418	2535	2403	2403
query3	6631	207	206	206
query4	19929	17324	17268	17268
query5	4197	442	405	405
query6	250	157	159	157
query7	4581	308	288	288
query8	252	197	182	182
query9	8688	2438	2434	2434
query10	462	298	274	274
query11	10484	10191	10034	10034
query12	147	101	97	97
query13	1643	373	372	372
query14	9517	7564	7544	7544
query15	210	173	169	169
query16	7718	270	274	270
query17	1719	524	525	524
query18	1950	275	284	275
query19	197	153	161	153
query20	94	84	89	84
query21	198	130	133	130
query22	4231	3898	3889	3889
query23	33763	32933	33098	32933
query24	10522	2832	2846	2832
query25	568	354	368	354
query26	706	156	155	155
query27	2206	335	332	332
query28	5902	2102	2087	2087
query29	866	617	592	592
query30	250	147	153	147
query31	951	766	744	744
query32	90	73	60	60
query33	652	281	264	264
query34	866	501	483	483
query35	729	637	597	597
query36	1079	938	900	900
query37	102	66	68	66
query38	2896	2784	2770	2770
query39	827	795	790	790
query40	193	125	122	122
query41	45	41	44	41
query42	101	97	102	97
query43	591	525	575	525
query44	1103	725	733	725
query45	177	165	166	165
query46	1079	723	718	718
query47	1873	1769	1764	1764
query48	378	304	301	301
query49	842	379	382	379
query50	779	386	381	381
query51	6928	6769	6775	6769
query52	105	87	89	87
query53	344	299	287	287
query54	836	433	427	427
query55	75	75	75	75
query56	264	237	249	237
query57	1116	1041	1044	1041
query58	236	219	216	216
query59	3483	3204	3093	3093
query60	283	268	284	268
query61	93	84	89	84
query62	624	450	449	449
query63	310	281	287	281
query64	8427	2252	1783	1783
query65	3189	3111	3134	3111
query66	791	335	326	326
query67	15391	14852	14909	14852
query68	4537	540	534	534
query69	490	273	274	273
query70	1180	1107	1161	1107
query71	394	281	278	278
query72	7845	2743	2528	2528
query73	705	324	321	321
query74	6145	5598	5525	5525
query75	3444	2676	2608	2608
query76	3047	988	963	963
query77	579	267	269	267
query78	10239	9872	9906	9872
query79	2254	515	525	515
query80	923	447	434	434
query81	530	222	218	218
query82	714	92	92	92
query83	241	170	185	170
query84	246	90	86	86
query85	1459	271	265	265
query86	459	306	297	297
query87	3361	3100	3193	3100
query88	4243	2444	2427	2427
query89	478	398	378	378
query90	1969	190	190	190
query91	126	99	100	99
query92	57	50	49	49
query93	1916	519	494	494
query94	1193	183	196	183
query95	398	318	312	312
query96	589	269	268	268
query97	3144	3001	3053	3001
query98	245	219	223	219
query99	1191	937	836	836
Total cold run time: 267410 ms
Total hot run time: 170148 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5af7e9df57d8e3a85af2e633cb3563244eb5ea1a, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.49	0.49	0.50
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.47	0.49
query10	0.55	0.55	0.53
query11	0.16	0.12	0.11
query12	0.14	0.13	0.12
query13	0.59	0.59	0.58
query14	0.77	0.78	0.79
query15	0.81	0.80	0.81
query16	0.36	0.36	0.37
query17	0.97	1.03	0.93
query18	0.23	0.23	0.24
query19	1.80	1.74	1.78
query20	0.01	0.01	0.01
query21	15.54	0.68	0.66
query22	4.72	5.97	2.71
query23	18.32	1.43	1.24
query24	2.02	0.23	0.19
query25	0.15	0.11	0.08
query26	0.27	0.16	0.16
query27	0.08	0.08	0.07
query28	13.28	1.00	0.99
query29	13.77	3.31	3.25
query30	0.24	0.05	0.06
query31	2.88	0.37	0.38
query32	3.31	0.46	0.47
query33	2.90	2.83	2.89
query34	17.08	4.41	4.40
query35	4.53	4.50	4.66
query36	0.66	0.46	0.45
query37	0.17	0.14	0.14
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.14	0.13
query41	0.08	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.03
Total cold run time: 111.07 s
Total hot run time: 31.07 s

@morrySnowmorrySnow mentioned this pull request May 27, 2024
Closed
3 tasks
@github-actions

github-actionsBot commented May 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions

github-actionsBot commented May 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 28, 2024
@morrySnow
morrySnow merged commit 0632309 into apache:masterMay 28, 2024
@morrySnow
morrySnow deleted the fix_cte_push_filter branch May 28, 2024 07:13
This was referenced May 28, 2024
Merged
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request May 28, 2024
@morrySnow
[fix](Nereids) could not push down filter through cte producer sometimes
63c136a
pick from master apache#35463
commit id 0632309
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
morrySnow added a commit that referenced this pull request May 28, 2024
@morrySnow
[fix](Nereids) could not push down filter through cte producer someti…
d5e56ba
…mes (#35509)
pick from master #35463
commit id 0632309
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
morrySnow added a commit that referenced this pull request May 28, 2024
@morrySnow
[fix](Nereids) could not push down filter through cte producer someti…
63e63e1
…mes (#35507)
pick from master #35463
commit id 0632309
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
dataroaring pushed a commit that referenced this pull request May 31, 2024
@morrySnow@dataroaring
[fix](Nereids) could not push down filter through cte producer someti…
6837ec6
…mes (#35463)
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
@morningmanmorningman mentioned this pull request Jun 22, 2024
Closed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morrySnowweixingyu12
[fix](Nereids) could not push down filter through cte producer someti…
71002c8
…mes (apache#35509)
pick from master apache#35463
commit id 0632309
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@morrySnow@HappenLee
[fix](Nereids) could not push down filter through cte producer someti…
16075b5
…mes (apache#35463)
when consumer side eliminate some consumers from plan, the size of
consumers is wrong. so we cannot push down some filter in producer side.
this PR fix this problem by update consumer set after rewrite outer side
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@engleflyengleflyenglefly approved these changes
+1 more reviewer
@keanji-xkeanji-xkeanji-x approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.11-mergeddev/2.1.4-mergeddev/3.0.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] predicate can't push down when query cantain ‘union all’

5 participants

@morrySnow@doris-robot@englefly@keanji-x@dataroaring