Skip to content

[fix](Nereids) not generate duplicate exprid after convert outer to anti rule - #52798

Merged
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:remove_outer_to_anti
Jul 15, 2025
Merged

[fix](Nereids) not generate duplicate exprid after convert outer to anti rule #52798
morrySnow merged 1 commit into
apache:masterfrom
morrySnow:remove_outer_to_anti

Conversation

@morrySnow

@morrySnowmorrySnow commented Jul 4, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

this rule could generate duplicate ExprID.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow

morrySnow commented Jul 4, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 4, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33687 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c42620be5e9ed6e9e1aafe7974d30043b4c8ad0, data reload: false
------ Round 1 ----------------------------------
q1	17596	5076	4978	4978
q2	1952	279	183	183
q3	10301	1252	700	700
q4	10222	1048	508	508
q5	7549	2374	2305	2305
q6	175	164	129	129
q7	881	753	612	612
q8	9320	1287	1034	1034
q9	6964	5078	5064	5064
q10	6868	2373	1967	1967
q11	472	290	275	275
q12	335	336	217	217
q13	17746	3646	3112	3112
q14	216	217	217	217
q15	533	469	482	469
q16	419	423	374	374
q17	586	860	360	360
q18	7569	7239	7144	7144
q19	1079	980	536	536
q20	328	348	215	215
q21	3665	3165	2320	2320
q22	1080	1047	968	968
Total cold run time: 105856 ms
Total hot run time: 33687 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5108	5113	5236	5113
q2	249	326	217	217
q3	2185	2688	2304	2304
q4	1430	1793	1369	1369
q5	4169	4414	4585	4414
q6	211	168	124	124
q7	2036	1986	1834	1834
q8	2585	2585	2681	2585
q9	7456	7317	7281	7281
q10	3065	3277	2847	2847
q11	596	514	507	507
q12	697	800	605	605
q13	3565	4024	3323	3323
q14	307	308	278	278
q15	521	479	477	477
q16	440	482	458	458
q17	1168	1651	1421	1421
q18	7928	7702	7616	7616
q19	794	819	905	819
q20	2036	2068	1941	1941
q21	5028	4898	4597	4597
q22	1169	1081	1057	1057
Total cold run time: 52743 ms
Total hot run time: 51187 ms

@doris-robot

doris-robot commented Jul 4, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185903 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c42620be5e9ed6e9e1aafe7974d30043b4c8ad0, data reload: false
query1	1054	396	382	382
query2	6521	1795	1749	1749
query3	6739	214	213	213
query4	26342	23284	22902	22902
query5	4348	553	423	423
query6	297	216	195	195
query7	4626	486	291	291
query8	292	232	222	222
query9	8644	2628	2630	2628
query10	470	312	279	279
query11	15515	15093	14738	14738
query12	152	120	112	112
query13	1658	526	400	400
query14	8755	5731	5735	5731
query15	212	190	175	175
query16	7536	620	502	502
query17	1215	738	600	600
query18	2012	416	335	335
query19	201	198	160	160
query20	124	121	111	111
query21	216	125	113	113
query22	4244	4352	4685	4352
query23	34862	33894	34516	33894
query24	8543	2472	2599	2472
query25	537	489	397	397
query26	1239	274	156	156
query27	2707	516	345	345
query28	4269	2170	2132	2132
query29	741	560	438	438
query30	276	213	180	180
query31	918	852	781	781
query32	67	60	58	58
query33	547	350	309	309
query34	797	823	506	506
query35	778	806	728	728
query36	955	1006	876	876
query37	111	98	72	72
query38	4130	4100	4112	4100
query39	1477	1400	1387	1387
query40	208	118	101	101
query41	54	51	51	51
query42	121	106	113	106
query43	515	512	486	486
query44	1304	838	818	818
query45	175	169	167	167
query46	830	1017	621	621
query47	1770	1820	1730	1730
query48	384	409	316	316
query49	742	485	431	431
query50	645	667	406	406
query51	4078	4133	4156	4133
query52	102	103	94	94
query53	222	248	184	184
query54	560	556	488	488
query55	84	78	77	77
query56	287	285	298	285
query57	1171	1174	1118	1118
query58	259	252	249	249
query59	2693	2762	2712	2712
query60	319	305	304	304
query61	119	117	113	113
query62	826	702	659	659
query63	215	182	183	182
query64	4339	1006	644	644
query65	4234	4231	4166	4166
query66	1119	411	301	301
query67	15791	15743	15244	15244
query68	8218	870	520	520
query69	470	308	273	273
query70	1197	1121	1058	1058
query71	469	328	296	296
query72	5596	4765	4918	4765
query73	738	679	350	350
query74	9299	9124	9045	9045
query75	3908	3155	2657	2657
query76	3703	1143	742	742
query77	770	373	279	279
query78	9753	9711	9112	9112
query79	1960	941	588	588
query80	645	507	437	437
query81	475	252	220	220
query82	424	126	92	92
query83	243	247	237	237
query84	260	104	86	86
query85	794	370	310	310
query86	334	327	288	288
query87	4401	4384	4326	4326
query88	3131	2318	2317	2317
query89	372	315	289	289
query90	1980	214	207	207
query91	141	139	108	108
query92	76	63	52	52
query93	1080	974	578	578
query94	646	405	313	313
query95	377	292	285	285
query96	500	576	295	295
query97	2704	2807	2613	2613
query98	235	211	205	205
query99	1330	1396	1330	1330
Total cold run time: 273522 ms
Total hot run time: 185903 ms

@doris-robot

doris-robot commented Jul 4, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c42620be5e9ed6e9e1aafe7974d30043b4c8ad0, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.63	0.11	0.11
query5	0.42	0.44	0.40
query6	1.19	0.64	0.66
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.60	0.51	0.52
query10	0.57	0.56	0.59
query11	0.16	0.10	0.11
query12	0.16	0.11	0.11
query13	0.63	0.61	0.60
query14	0.79	0.82	0.80
query15	0.87	0.89	0.87
query16	0.40	0.37	0.39
query17	1.03	1.09	1.05
query18	0.22	0.22	0.21
query19	1.94	1.83	1.83
query20	0.01	0.02	0.02
query21	15.39	0.91	0.56
query22	0.75	1.18	0.70
query23	14.89	1.32	0.60
query24	6.77	1.44	1.32
query25	0.48	0.23	0.06
query26	0.67	0.17	0.15
query27	0.07	0.06	0.05
query28	9.18	0.91	0.44
query29	12.62	3.92	3.29
query30	0.25	0.09	0.07
query31	2.84	0.60	0.38
query32	3.23	0.56	0.47
query33	3.07	3.08	3.15
query34	16.06	5.37	4.80
query35	4.85	4.87	4.91
query36	0.70	0.50	0.49
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.15
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.37 s
Total hot run time: 30.04 s

@hello-stephen

hello-stephen commented Jul 4, 2025

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@morrySnow
morrySnowforce-pushed the remove_outer_to_anti branch from 5c42620 to 82608ccCompareJuly 7, 2025 09:04
@morrySnow

morrySnow commented Jul 7, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 7, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32990 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 82608cc618aab19c2f829efb2546f9cf35b50970, data reload: false
------ Round 1 ----------------------------------
q1	17571	5318	5009	5009
q2	1926	288	185	185
q3	10313	1289	735	735
q4	10224	1017	510	510
q5	7927	2430	2298	2298
q6	176	157	131	131
q7	890	738	593	593
q8	9309	1295	1100	1100
q9	6718	5077	5074	5074
q10	6884	2370	1992	1992
q11	493	277	269	269
q12	341	342	221	221
q13	17772	3683	3044	3044
q14	220	219	222	219
q15	525	482	468	468
q16	425	420	373	373
q17	587	861	355	355
q18	7539	7040	7195	7040
q19	1213	949	538	538
q20	340	335	216	216
q21	3767	2503	2331	2331
q22	356	310	289	289
Total cold run time: 105516 ms
Total hot run time: 32990 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5069	5054	5065	5054
q2	236	319	228	228
q3	2180	2677	2269	2269
q4	1382	1781	1332	1332
q5	4219	4173	4621	4173
q6	215	166	133	133
q7	2051	2025	1802	1802
q8	2631	2531	2688	2531
q9	7720	7229	7240	7229
q10	3084	3284	2843	2843
q11	560	513	488	488
q12	678	779	636	636
q13	3589	3970	3305	3305
q14	303	287	280	280
q15	523	497	478	478
q16	452	487	456	456
q17	1179	1532	1431	1431
q18	7917	7602	7499	7499
q19	784	782	859	782
q20	2032	2009	1948	1948
q21	5037	4615	4536	4536
q22	626	618	522	522
Total cold run time: 52467 ms
Total hot run time: 49955 ms

@doris-robot

doris-robot commented Jul 7, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 174820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 82608cc618aab19c2f829efb2546f9cf35b50970, data reload: false
dbgen_version	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:44:00	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL catalog_page	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:58	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL reason	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 22:43:56	2023-12-26 22:44:01	NULL	utf-8	NULL	NULL ============================================
query1	1007	384	373	373
query2	6514	1669	1691	1669
query3	6748	211	211	211
query4	26884	23388	22889	22889
query5	4349	566	450	450
query6	283	208	194	194
query7	4615	488	289	289
query8	268	214	208	208
query9	8612	2618	2647	2618
query10	query11	15679	14990	14678	14678
query12	156	114	104	104
query13	1655	533	395	395
query14	8492	5851	5837	5837
query15	194	190	176	176
query16	7128	443	272	272
query17	1011	730	579	579
query18	1977	395	312	312
query19	198	190	169	169
query20	119	113	111	111
query21	214	122	110	110
query22	4179	4244	4546	4244
query23	34911	33859	33522	33522
query24	8454	2406	2392	2392
query25	531	437	433	433
query26	1234	256	146	146
query27	2786	495	342	342
query28	4280	2136	2098	2098
query29	773	541	429	429
query30	282	216	191	191
query31	911	879	739	739
query32	66	57	55	55
query33	539	330	278	278
query34	790	832	510	510
query35	query36	963	1015	895	895
query37	108	97	70	70
query38	4112	4124	4042	4042
query39	1461	1419	1451	1419
query40	210	115	103	103
query41	56	58	55	55
query42	120	103	103	103
query43	491	479	476	476
query44	1340	830	812	812
query45	query46	841	1002	660	660
query47	1756	1786	1728	1728
query48	377	427	306	306
query49	770	481	404	404
query50	635	685	406	406
query51	4177	4316	4188	4188
query52	99	102	94	94
query53	220	254	180	180
query54	579	577	515	515
query55	85	79	77	77
query56	299	304	286	286
query57	1165	1207	1113	1113
query58	259	254	249	249
query59	2588	2583	2500	2500
query60	337	304	300	300
query61	157	120	119	119
query62	779	711	696	696
query63	230	186	181	181
query64	4434	1157	835	835
query65	4259	4155	4179	4155
query66	1157	396	304	304
query67	15776	15611	15517	15517
query68	8635	870	540	540
query69	518	301	264	264
query70	1218	1167	1049	1049
query71	451	317	288	288
query72	5623	4858	5075	4858
query73	745	694	353	353
query74	8851	9221	8906	8906
query75	4051	3179	2657	2657
query76	3642	1154	740	740
query77	800	381	277	277
query78	query79	911	1005	574	574
query80	548	520	449	449
query81	449	255	219	219
query82	181	121	92	92
query83	240	248	238	238
query84	285	109	85	85
query85	946	383	326	326
query86	328	305	295	295
query87	4409	4462	4423	4423
query88	2833	2283	2277	2277
query89	372	313	281	281
query90	1616	206	213	206
query91	135	141	109	109
query92	61	56	53	53
query93	948	977	596	596
query94	612	310	200	200
query95	373	290	272	272
query96	495	587	284	284
query97	2663	2783	2617	2617
query98	238	207	198	198
query99	1285	1391	1283	1283
Total cold run time: 260141 ms
Total hot run time: 174820 ms

@doris-robot

doris-robot commented Jul 7, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 82608cc618aab19c2f829efb2546f9cf35b50970, data reload: false
query1	0.05	0.04	0.03
query2	0.07	0.04	0.04
query3	0.24	0.08	0.07
query4	1.62	0.11	0.11
query5	0.41	0.43	0.41
query6	1.17	0.67	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.61	0.52	0.50
query10	0.56	0.57	0.59
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.61	0.61
query14	0.79	0.81	0.82
query15	0.90	0.89	0.86
query16	0.37	0.39	0.39
query17	1.09	1.06	1.04
query18	0.22	0.21	0.21
query19	2.04	1.86	1.77
query20	0.02	0.01	0.01
query21	15.40	0.90	0.58
query22	0.75	1.21	0.66
query23	14.94	1.36	0.61
query24	6.66	1.92	1.40
query25	0.49	0.33	0.09
query26	0.54	0.16	0.13
query27	0.06	0.06	0.05
query28	9.82	0.81	0.43
query29	12.55	3.96	3.22
query30	0.25	0.09	0.07
query31	2.93	0.56	0.37
query32	3.25	0.54	0.46
query33	3.10	3.08	3.14
query34	16.06	5.42	4.71
query35	4.82	4.84	4.85
query36	0.69	0.54	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.94 s
Total hot run time: 29.89 s

@morrySnowmorrySnow changed the title [fix](Nereids) remove convert outer to anti rule temporary[fix](Nereids) not generate duplicate exprid after convert outer to anti rule Jul 8, 2025
@morrySnow
morrySnowforce-pushed the remove_outer_to_anti branch from 82608cc to ac9f4d8CompareJuly 8, 2025 11:08
@morrySnow

morrySnow commented Jul 8, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 8, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34042 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac9f4d835ccc794d74c29e3260235cd3d6936428, data reload: false
------ Round 1 ----------------------------------
q1	17587	5217	5073	5073
q2	1934	289	192	192
q3	10333	1315	736	736
q4	10299	1007	521	521
q5	8604	2380	2360	2360
q6	220	157	130	130
q7	894	770	612	612
q8	9360	1299	1109	1109
q9	7219	5173	5108	5108
q10	6970	2407	1986	1986
q11	483	287	273	273
q12	371	371	218	218
q13	17788	3711	3122	3122
q14	241	233	225	225
q15	571	469	482	469
q16	429	425	383	383
q17	602	865	375	375
q18	7517	7030	7105	7030
q19	1500	960	541	541
q20	332	344	221	221
q21	3689	3159	3057	3057
q22	360	329	301	301
Total cold run time: 107303 ms
Total hot run time: 34042 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5212	5187	5035	5035
q2	241	327	220	220
q3	2145	2684	2265	2265
q4	1342	1789	1365	1365
q5	4242	4665	4629	4629
q6	226	165	123	123
q7	2031	1949	1871	1871
q8	2630	2613	2515	2515
q9	7336	7204	7136	7136
q10	3114	3346	2894	2894
q11	645	527	487	487
q12	712	774	631	631
q13	3577	4120	3351	3351
q14	293	289	303	289
q15	521	510	486	486
q16	441	520	470	470
q17	1213	1481	1463	1463
q18	7852	7776	7609	7609
q19	819	838	908	838
q20	2000	2130	1936	1936
q21	4998	4609	4609	4609
q22	664	622	557	557
Total cold run time: 52254 ms
Total hot run time: 50779 ms

@doris-robot

doris-robot commented Jul 8, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 184549 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac9f4d835ccc794d74c29e3260235cd3d6936428, data reload: false
query1	1050	388	387	387
query2	6524	1790	1832	1790
query3	6743	217	225	217
query4	26317	23365	23516	23365
query5	4360	577	446	446
query6	322	235	220	220
query7	4622	478	297	297
query8	294	221	217	217
query9	8628	2602	2619	2602
query10	465	331	277	277
query11	15587	15035	14876	14876
query12	154	104	101	101
query13	1662	536	406	406
query14	9212	5703	5666	5666
query15	203	186	168	168
query16	7387	622	445	445
query17	1188	695	556	556
query18	1979	394	309	309
query19	186	189	151	151
query20	120	117	111	111
query21	212	127	108	108
query22	3922	4174	4665	4174
query23	34715	33627	32977	32977
query24	8511	2341	2388	2341
query25	524	454	378	378
query26	1238	258	143	143
query27	2783	499	345	345
query28	4310	2112	2094	2094
query29	728	571	417	417
query30	285	214	184	184
query31	934	835	772	772
query32	69	62	57	57
query33	544	357	315	315
query34	781	872	500	500
query35	784	808	722	722
query36	985	974	877	877
query37	111	96	74	74
query38	4196	4100	4110	4100
query39	1475	1385	1403	1385
query40	208	115	104	104
query41	53	53	48	48
query42	124	103	106	103
query43	500	497	492	492
query44	1295	815	830	815
query45	188	172	161	161
query46	824	993	612	612
query47	1745	1775	1707	1707
query48	382	410	305	305
query49	731	492	380	380
query50	629	689	410	410
query51	4176	4324	4153	4153
query52	106	102	92	92
query53	216	254	181	181
query54	556	563	490	490
query55	85	82	82	82
query56	298	288	280	280
query57	1167	1181	1117	1117
query58	270	251	254	251
query59	2681	2882	2681	2681
query60	332	319	301	301
query61	131	127	154	127
query62	783	717	645	645
query63	216	188	184	184
query64	4292	1035	670	670
query65	4239	4184	4174	4174
query66	1107	400	329	329
query67	16055	15578	15490	15490
query68	8057	880	528	528
query69	478	309	267	267
query70	1187	1160	1100	1100
query71	445	309	308	308
query72	5588	4764	4825	4764
query73	655	605	352	352
query74	9179	8971	8737	8737
query75	3422	3244	2679	2679
query76	3355	1142	713	713
query77	621	365	297	297
query78	9431	9489	8852	8852
query79	1674	918	570	570
query80	657	510	448	448
query81	482	251	223	223
query82	187	127	93	93
query83	256	248	234	234
query84	247	108	83	83
query85	769	385	314	314
query86	359	299	273	273
query87	4478	4411	4325	4325
query88	2851	2259	2233	2233
query89	367	327	286	286
query90	1779	204	200	200
query91	145	135	109	109
query92	64	60	56	56
query93	1097	967	581	581
query94	614	383	282	282
query95	371	291	289	289
query96	489	561	274	274
query97	2696	2750	2642	2642
query98	231	214	212	212
query99	1356	1382	1271	1271
Total cold run time: 271086 ms
Total hot run time: 184549 ms

@doris-robot

doris-robot commented Jul 8, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.17 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ac9f4d835ccc794d74c29e3260235cd3d6936428, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.05
query3	0.26	0.08	0.07
query4	1.61	0.10	0.10
query5	0.43	0.42	0.42
query6	1.20	0.65	0.64
query7	0.02	0.02	0.01
query8	0.05	0.03	0.04
query9	0.60	0.50	0.51
query10	0.55	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.61	0.61
query14	0.80	0.81	0.81
query15	0.90	0.86	0.88
query16	0.38	0.39	0.40
query17	1.05	1.10	1.06
query18	0.22	0.21	0.21
query19	1.97	1.81	1.86
query20	0.01	0.01	0.01
query21	15.41	0.87	0.55
query22	0.76	1.09	0.79
query23	14.90	1.42	0.58
query24	6.53	1.99	0.32
query25	0.30	0.09	0.09
query26	0.53	0.17	0.13
query27	0.06	0.05	0.05
query28	9.48	0.94	0.45
query29	12.52	4.02	3.33
query30	0.25	0.09	0.06
query31	2.83	0.61	0.38
query32	3.24	0.56	0.47
query33	3.06	3.12	3.12
query34	16.18	5.37	4.81
query35	4.81	4.83	4.83
query36	0.70	0.51	0.49
query37	0.09	0.06	0.07
query38	0.05	0.05	0.04
query39	0.03	0.02	0.03
query40	0.16	0.15	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.14 s
Total hot run time: 29.17 s

@morrySnow
morrySnowforce-pushed the remove_outer_to_anti branch 4 times, most recently from 70eb39a to b657fd2CompareJuly 9, 2025 08:51
@morrySnow

morrySnow commented Jul 9, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33428 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b657fd2369e7fd9ec006ae5b1fe10d3e87eae66e, data reload: false
------ Round 1 ----------------------------------
q1	17552	5266	4999	4999
q2	1920	286	199	199
q3	10287	1278	691	691
q4	10207	1021	506	506
q5	7534	2488	2557	2488
q6	183	159	128	128
q7	881	749	592	592
q8	9313	1344	1197	1197
q9	6772	5080	5096	5080
q10	6868	2368	1961	1961
q11	495	284	277	277
q12	333	352	215	215
q13	17749	3707	3138	3138
q14	246	254	228	228
q15	539	480	479	479
q16	427	425	381	381
q17	583	859	344	344
q18	7444	7183	7087	7087
q19	1063	970	576	576
q20	344	345	224	224
q21	3828	3137	2335	2335
q22	354	325	303	303
Total cold run time: 104922 ms
Total hot run time: 33428 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5127	5139	5111	5111
q2	242	335	227	227
q3	2168	2727	2320	2320
q4	1363	1773	1278	1278
q5	4199	4218	4542	4218
q6	227	169	130	130
q7	2069	1942	1800	1800
q8	2615	2672	2824	2672
q9	7530	7246	7134	7134
q10	3154	3286	3016	3016
q11	633	536	516	516
q12	698	759	656	656
q13	3449	3996	3437	3437
q14	294	325	309	309
q15	539	485	481	481
q16	440	518	465	465
q17	1220	1640	1358	1358
q18	7917	7674	7633	7633
q19	817	838	917	838
q20	2033	2038	1894	1894
q21	5206	4676	4619	4619
q22	653	646	564	564
Total cold run time: 52593 ms
Total hot run time: 50676 ms

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 183352 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b657fd2369e7fd9ec006ae5b1fe10d3e87eae66e, data reload: false
query1	990	412	384	384
query2	6497	1679	1655	1655
query3	6741	210	209	209
query4	26184	23486	22930	22930
query5	4408	591	436	436
query6	321	213	195	195
query7	4625	500	289	289
query8	265	219	213	213
query9	8605	2635	2650	2635
query10	460	338	277	277
query11	15710	14999	14840	14840
query12	151	117	105	105
query13	1655	540	408	408
query14	8477	5926	5834	5834
query15	210	193	173	173
query16	7527	440	268	268
query17	1319	752	584	584
query18	2008	419	336	336
query19	201	194	162	162
query20	122	121	117	117
query21	211	125	106	106
query22	4068	4194	4830	4194
query23	35236	34060	33406	33406
query24	8421	2370	2377	2370
query25	530	454	399	399
query26	1225	266	141	141
query27	2756	503	349	349
query28	4260	2128	2114	2114
query29	735	550	420	420
query30	280	214	187	187
query31	899	832	768	768
query32	70	60	61	60
query33	548	328	281	281
query34	780	836	520	520
query35	621	618	553	553
query36	936	1007	879	879
query37	105	97	71	71
query38	4194	4092	4041	4041
query39	1460	1431	1579	1431
query40	209	117	102	102
query41	56	53	51	51
query42	125	106	110	106
query43	488	501	481	481
query44	1285	826	830	826
query45	172	180	170	170
query46	841	1016	616	616
query47	1741	1788	1719	1719
query48	376	420	318	318
query49	737	516	380	380
query50	646	688	425	425
query51	4155	4233	4034	4034
query52	108	104	94	94
query53	219	243	183	183
query54	573	564	506	506
query55	82	76	80	76
query56	308	296	331	296
query57	1168	1186	1114	1114
query58	268	243	258	243
query59	2515	2598	2542	2542
query60	335	329	302	302
query61	125	122	124	122
query62	805	691	671	671
query63	218	182	181	181
query64	4364	1194	895	895
query65	4283	4187	4134	4134
query66	1118	411	319	319
query67	15900	15595	15460	15460
query68	8518	894	526	526
query69	537	316	270	270
query70	1199	1121	1033	1033
query71	496	325	285	285
query72	5735	2356	5095	2356
query73	782	742	350	350
query74	8879	9178	8965	8965
query75	4008	3326	2658	2658
query76	3703	1141	719	719
query77	793	383	297	297
query78	11078	10925	10161	10161
query79	2173	825	569	569
query80	590	522	449	449
query81	479	250	216	216
query82	479	128	100	100
query83	252	243	233	233
query84	309	106	88	88
query85	835	366	332	332
query86	393	307	286	286
query87	4377	4427	4359	4359
query88	3684	2282	2265	2265
query89	386	318	281	281
query90	1817	212	203	203
query91	138	142	111	111
query92	74	62	65	62
query93	1576	962	591	591
query94	691	315	194	194
query95	369	290	297	290
query96	490	558	279	279
query97	2667	2698	2604	2604
query98	244	212	200	200
query99	1325	1422	1314	1314
Total cold run time: 276107 ms
Total hot run time: 183352 ms

@morrySnow
morrySnowforce-pushed the remove_outer_to_anti branch from b657fd2 to a1994fbCompareJuly 9, 2025 09:31
@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b657fd2369e7fd9ec006ae5b1fe10d3e87eae66e, data reload: false
query1	0.04	0.03	0.04
query2	0.09	0.04	0.04
query3	0.26	0.07	0.07
query4	1.62	0.11	0.11
query5	0.43	0.42	0.41
query6	1.15	0.66	0.67
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.61	0.51	0.51
query10	0.57	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	0.81	0.83	0.82
query15	0.90	0.88	0.86
query16	0.39	0.38	0.38
query17	1.05	1.06	1.05
query18	0.22	0.21	0.21
query19	1.92	1.87	1.82
query20	0.01	0.01	0.01
query21	15.40	0.88	0.54
query22	0.76	1.20	0.69
query23	14.92	1.38	0.64
query24	7.18	1.55	0.56
query25	0.50	0.20	0.08
query26	0.68	0.17	0.13
query27	0.06	0.05	0.05
query28	9.73	0.85	0.47
query29	12.53	3.97	3.32
query30	0.27	0.09	0.06
query31	2.82	0.62	0.38
query32	3.22	0.57	0.48
query33	3.12	3.10	3.05
query34	16.05	5.44	4.82
query35	4.87	4.86	4.90
query36	0.67	0.50	0.49
query37	0.10	0.07	0.07
query38	0.06	0.05	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.15
query41	0.09	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.04
Total cold run time: 104.4 s
Total hot run time: 29.4 s

@morrySnow

morrySnow commented Jul 9, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33443 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1994fb78b4964872593b3985c5cb61754a5237b, data reload: false
------ Round 1 ----------------------------------
q1	17563	5283	5119	5119
q2	1931	280	190	190
q3	10430	1343	743	743
q4	10267	1026	538	538
q5	7900	2349	2353	2349
q6	184	166	133	133
q7	930	752	610	610
q8	9324	1342	1096	1096
q9	6956	5137	5182	5137
q10	6898	2360	1979	1979
q11	475	285	281	281
q12	337	352	213	213
q13	17764	3705	3135	3135
q14	231	239	217	217
q15	538	474	488	474
q16	416	436	381	381
q17	584	870	365	365
q18	7754	7235	7042	7042
q19	1223	952	555	555
q20	348	354	225	225
q21	3956	2608	2358	2358
q22	358	327	303	303
Total cold run time: 106367 ms
Total hot run time: 33443 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5169	5452	5176	5176
q2	251	330	221	221
q3	2185	2664	2282	2282
q4	1412	1797	1374	1374
q5	4218	4402	4586	4402
q6	217	170	130	130
q7	2082	1932	1827	1827
q8	2721	2591	2729	2591
q9	7758	7325	7213	7213
q10	3096	3308	3117	3117
q11	569	514	503	503
q12	680	771	600	600
q13	3492	4181	3260	3260
q14	276	301	281	281
q15	527	502	483	483
q16	433	509	461	461
q17	1207	1658	1362	1362
q18	7900	7771	7634	7634
q19	823	827	820	820
q20	2060	2085	1995	1995
q21	4975	4584	4582	4582
q22	644	602	557	557
Total cold run time: 52695 ms
Total hot run time: 50871 ms

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186088 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1994fb78b4964872593b3985c5cb61754a5237b, data reload: false
query1	1020	384	393	384
query2	6519	1691	1662	1662
query3	6761	210	209	209
query4	26516	24293	23002	23002
query5	4859	595	449	449
query6	314	200	194	194
query7	4625	499	289	289
query8	265	215	202	202
query9	8628	2622	2624	2622
query10	489	319	265	265
query11	15545	15051	14816	14816
query12	150	102	97	97
query13	1640	513	384	384
query14	8585	5841	5829	5829
query15	212	189	172	172
query16	7184	429	254	254
query17	1311	695	602	602
query18	1971	391	292	292
query19	184	181	152	152
query20	118	115	122	115
query21	210	132	104	104
query22	4095	4188	4571	4188
query23	35002	33947	33519	33519
query24	8417	2379	2444	2379
query25	531	484	382	382
query26	1258	255	146	146
query27	2790	513	351	351
query28	4318	2126	2132	2126
query29	758	532	422	422
query30	284	214	195	195
query31	906	846	744	744
query32	74	62	57	57
query33	556	337	289	289
query34	784	847	533	533
query35	607	647	562	562
query36	965	991	892	892
query37	107	92	75	75
query38	4186	4151	4118	4118
query39	1465	1406	1420	1406
query40	210	118	109	109
query41	56	54	49	49
query42	115	101	103	101
query43	498	496	465	465
query44	1292	821	813	813
query45	175	170	164	164
query46	827	1015	635	635
query47	1753	1814	1724	1724
query48	377	435	308	308
query49	762	492	409	409
query50	631	684	428	428
query51	4216	4115	4094	4094
query52	104	101	94	94
query53	220	251	183	183
query54	572	567	495	495
query55	80	78	82	78
query56	306	308	285	285
query57	1187	1194	1095	1095
query58	263	250	260	250
query59	2536	2691	2547	2547
query60	328	312	317	312
query61	128	124	148	124
query62	788	704	670	670
query63	229	186	188	186
query64	4619	1339	985	985
query65	4333	4189	4196	4189
query66	1131	420	332	332
query67	16177	15625	15235	15235
query68	9063	897	533	533
query69	514	307	269	269
query70	1239	1107	1048	1048
query71	462	312	283	283
query72	5540	4800	4767	4767
query73	724	613	353	353
query74	9141	8814	9066	8814
query75	4195	3192	2708	2708
query76	3650	1139	722	722
query77	828	386	304	304
query78	10984	11161	10381	10381
query79	1431	872	585	585
query80	642	513	442	442
query81	464	264	231	231
query82	451	123	97	97
query83	256	269	248	248
query84	248	106	79	79
query85	783	353	317	317
query86	335	298	288	288
query87	4371	4442	4395	4395
query88	2841	2254	2247	2247
query89	380	310	290	290
query90	1929	214	209	209
query91	140	152	118	118
query92	70	60	55	55
query93	1052	954	588	588
query94	679	310	193	193
query95	383	346	289	289
query96	502	559	278	278
query97	2692	2780	2644	2644
query98	228	214	213	213
query99	1427	1405	1262	1262
Total cold run time: 275766 ms
Total hot run time: 186088 ms

@doris-robot

doris-robot commented Jul 9, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.68 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1994fb78b4964872593b3985c5cb61754a5237b, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.25	0.08	0.07
query4	1.63	0.11	0.11
query5	0.44	0.42	0.41
query6	1.16	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.60	0.53	0.52
query10	0.58	0.57	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.60	0.61
query14	0.80	0.79	0.84
query15	0.89	0.87	0.85
query16	0.40	0.38	0.39
query17	1.06	1.08	1.09
query18	0.22	0.21	0.21
query19	1.93	1.84	1.85
query20	0.01	0.02	0.01
query21	15.42	0.91	0.56
query22	0.76	1.14	0.61
query23	15.01	1.40	0.63
query24	6.66	0.95	1.59
query25	0.55	0.07	0.17
query26	0.63	0.16	0.14
query27	0.05	0.06	0.05
query28	10.28	0.93	0.45
query29	12.53	3.95	3.29
query30	0.25	0.10	0.07
query31	2.83	0.61	0.39
query32	3.24	0.56	0.48
query33	3.13	3.06	3.13
query34	15.73	5.36	4.81
query35	4.80	4.85	4.86
query36	0.68	0.50	0.51
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.15 s
Total hot run time: 29.68 s

@morrySnow
morrySnow merged commit 54b170b into apache:masterJul 15, 2025
@morrySnow
morrySnow deleted the remove_outer_to_anti branch July 23, 2025 06:02
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 23, 2025
@morrySnow
branch-3.1: [fix](Nereids) not generate duplicate exprid after conver…
90c6c64
…t outer to anti rule apache#52798
cherry picked from apache#52798
@morrySnowmorrySnow mentioned this pull request Jul 23, 2025
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 24, 2025
@morrySnow
branch-3.1: [fix](Nereids) not generate duplicate exprid after conver…
65c76c2
…t outer to anti rule apache#52798
cherry picked from apache#52798
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 25, 2025
@morrySnow
branch-3.0: [fix](Nereids) not generate duplicate exprid after conver…
f47a7c4
…t outer to anti rule apache#52798
cherry picked from apache#52798
@morrySnowmorrySnow mentioned this pull request Jul 25, 2025
Merged
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 25, 2025
@morrySnow
branch-2.1: [fix](Nereids) not generate duplicate exprid after conver…
e883623
…t outer to anti rule apache#52798
cherry picked from apache#52798
@morrySnowmorrySnow mentioned this pull request Jul 25, 2025
Merged
morrySnow added a commit that referenced this pull request Jul 25, 2025
@morrySnow
branch-3.1: [fix](Nereids) not generate duplicate exprid after conver…
ffcf01f
…t outer to anti rule #52798 (#53773)
cherry picked from #52798
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 28, 2025
@morrySnow
branch-2.1: [fix](Nereids) not generate duplicate exprid after conver…
b9f6795
…t outer to anti rule apache#52798
cherry picked from apache#52798
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jul 28, 2025
@morrySnow
branch-2.1: [fix](Nereids) not generate duplicate exprid after conver…
23edc46
…t outer to anti rule apache#52798
cherry picked from apache#52798
yiguolei pushed a commit that referenced this pull request Aug 1, 2025
@morrySnow
branch-2.1: [fix](Nereids) not generate duplicate exprid after conver…
4a062c4
…t outer to anti rule #52798 (#53901)
cherry picked from #52798
dataroaring pushed a commit that referenced this pull request Aug 12, 2025
@morrySnow
branch-3.0: [fix](Nereids) not generate duplicate exprid after conver…
7d6787f
…t outer to anti rule #52798 (#53898)
cherry picked from #52798
@gavinchougavinchou mentioned this pull request Sep 1, 2025
Closed
@feiniaofeiafeifeiniaofeiafei mentioned this pull request Sep 8, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Sep 10, 2025
@feiniaofeiafei
[enhance](nereids) EliminateGroupByKeyByUniform support rewrite in cte (
294b373
#55680)
### What problem does this PR solve?
Related PR: #43391
Problem Summary:
EliminateGroupByKeyByUniform not support rewrite in cte in #43391, and
the restriction can be removed after
#52798, because #52798 implement the cteconsumer expression rewrite.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@924060929924060929924060929 approved these changes
+1 more reviewer
@feiniaofeiafeifeiniaofeiafeifeiniaofeiafei approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.11-mergeddev/3.0.8-mergeddev/3.1.0-mergedp0_wreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@morrySnow@Thearas@doris-robot@hello-stephen@924060929@feiniaofeiafei@yiguolei@dataroaring