Skip to content

[fix](nereids) fix distinct window compute wrong result - #48987

Closed
924060929 wants to merge 8 commits into
apache:masterfrom
924060929:fix_distinct_window
Closed

[fix](nereids) fix distinct window compute wrong result#48987
924060929 wants to merge 8 commits into
apache:masterfrom
924060929:fix_distinct_window

Conversation

@924060929

@924060929924060929 commented Mar 12, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

fix distinct window compute wrong result, introduced by #14397

select distinctsum(value) over(partition by id)
from (
select100 value, 1 id
union allselect100, 2
)a;
+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100 |
| 100 |
+----------------------------------+

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Mar 12, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929

924060929 commented Mar 12, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Mar 12, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32649 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85708a85f22e03537e8a2dac9f39595a228e3d28, data reload: false
------ Round 1 ----------------------------------
q1	17654	5189	5060	5060
q2	2043	294	175	175
q3	10406	2144	691	691
q4	10216	1029	546	546
q5	7534	2358	2346	2346
q6	187	164	134	134
q7	901	758	597	597
q8	9295	1274	1137	1137
q9	4998	4889	4873	4873
q10	6804	2319	1915	1915
q11	490	273	264	264
q12	348	354	215	215
q13	17769	3662	3064	3064
q14	229	232	215	215
q15	556	497	502	497
q16	612	616	583	583
q17	572	868	340	340
q18	6745	6591	6410	6410
q19	1218	935	558	558
q20	319	324	189	189
q21	2787	2208	1881	1881
q22	1039	1017	959	959
Total cold run time: 102722 ms
Total hot run time: 32649 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5137	5113	5152	5113
q2	230	322	231	231
q3	2163	2689	2291	2291
q4	1431	1799	1356	1356
q5	4233	4112	4190	4112
q6	204	164	124	124
q7	1865	1967	1788	1788
q8	2630	2651	2554	2554
q9	7275	7114	7186	7114
q10	2994	3222	2721	2721
q11	572	509	504	504
q12	670	786	650	650
q13	3465	3925	3272	3272
q14	286	296	272	272
q15	535	491	499	491
q16	635	672	640	640
q17	1149	1586	1362	1362
q18	7768	7644	7579	7579
q19	803	892	1017	892
q20	1959	2040	1893	1893
q21	5506	5031	4919	4919
q22	1097	1031	989	989
Total cold run time: 52607 ms
Total hot run time: 50867 ms

@doris-robot

doris-robot commented Mar 12, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 184344 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85708a85f22e03537e8a2dac9f39595a228e3d28, data reload: false
query1	1023	378	366	366
query2	6542	1826	1881	1826
query3	6792	209	217	209
query4	26447	23336	22974	22974
query5	4413	663	479	479
query6	285	201	195	195
query7	4605	510	304	304
query8	299	252	230	230
query9	8627	2619	2640	2619
query10	488	313	251	251
query11	15792	15125	15041	15041
query12	168	108	107	107
query13	1662	516	409	409
query14	10752	6185	6217	6185
query15	207	189	172	172
query16	7655	624	472	472
query17	1339	726	561	561
query18	1999	400	324	324
query19	208	185	155	155
query20	121	118	118	118
query21	270	119	104	104
query22	4037	4100	4143	4100
query23	33784	32811	32927	32811
query24	7128	2369	2378	2369
query25	506	457	380	380
query26	1235	272	157	157
query27	2428	472	322	322
query28	4306	2405	2394	2394
query29	706	544	427	427
query30	283	224	197	197
query31	965	858	755	755
query32	72	63	63	63
query33	546	382	303	303
query34	787	835	508	508
query35	786	823	791	791
query36	955	1036	895	895
query37	117	101	85	85
query38	4271	4172	3976	3976
query39	1430	1405	1390	1390
query40	208	114	104	104
query41	55	52	52	52
query42	118	102	109	102
query43	481	494	466	466
query44	1263	771	775	771
query45	175	172	164	164
query46	832	1006	621	621
query47	1748	1782	1725	1725
query48	365	409	298	298
query49	791	501	410	410
query50	644	728	408	408
query51	4149	4191	4084	4084
query52	109	104	97	97
query53	235	254	189	189
query54	482	518	423	423
query55	87	82	82	82
query56	273	262	262	262
query57	1136	1117	1059	1059
query58	243	237	239	237
query59	2413	2744	2585	2585
query60	269	274	259	259
query61	121	118	118	118
query62	781	746	677	677
query63	225	183	190	183
query64	4230	1002	661	661
query65	4438	4342	4369	4342
query66	1055	409	354	354
query67	15594	15489	15184	15184
query68	8685	866	500	500
query69	459	293	268	268
query70	1184	1110	1109	1109
query71	453	316	266	266
query72	5693	3467	3657	3467
query73	784	707	350	350
query74	9190	9086	8899	8899
query75	3986	3137	2680	2680
query76	3595	1180	744	744
query77	787	376	283	283
query78	10046	10134	9214	9214
query79	3470	811	564	564
query80	701	521	438	438
query81	496	260	226	226
query82	695	128	98	98
query83	211	175	154	154
query84	298	94	79	79
query85	793	353	304	304
query86	392	291	307	291
query87	4510	4522	4350	4350
query88	3542	2281	2241	2241
query89	381	320	280	280
query90	1858	208	207	207
query91	136	132	109	109
query92	77	60	63	60
query93	1952	1039	572	572
query94	668	487	303	303
query95	358	260	258	258
query96	492	562	275	275
query97	3331	3402	3301	3301
query98	238	208	199	199
query99	1426	1398	1314	1314
Total cold run time: 277443 ms
Total hot run time: 184344 ms

@doris-robot

doris-robot commented Mar 12, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85708a85f22e03537e8a2dac9f39595a228e3d28, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.62	0.10	0.11
query5	0.56	0.54	0.55
query6	1.17	0.71	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.52
query10	0.58	0.60	0.59
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.61	0.59
query14	2.66	2.80	2.79
query15	0.92	0.86	0.85
query16	0.37	0.39	0.37
query17	1.04	1.04	1.03
query18	0.21	0.20	0.19
query19	1.90	1.84	1.97
query20	0.01	0.02	0.01
query21	15.37	0.90	0.55
query22	0.78	1.19	0.64
query23	14.93	1.37	0.62
query24	6.88	1.92	1.39
query25	0.53	0.27	0.09
query26	0.54	0.16	0.14
query27	0.05	0.06	0.05
query28	10.30	0.78	0.42
query29	12.55	3.97	3.34
query30	0.26	0.09	0.08
query31	2.81	0.58	0.38
query32	3.23	0.54	0.46
query33	3.04	3.14	3.12
query34	15.72	5.19	4.57
query35	4.57	4.60	4.57
query36	0.67	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.69 s
Total hot run time: 31.73 s

@924060929

924060929 commented Mar 12, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Mar 12, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32423 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be6aeb661aacf9bc32cc793bea917272639dc1d4, data reload: false
------ Round 1 ----------------------------------
q1	17581	5176	5027	5027
q2	2039	305	171	171
q3	10393	1254	756	756
q4	10203	1032	525	525
q5	7533	2434	2320	2320
q6	191	168	132	132
q7	910	736	589	589
q8	9312	1274	1079	1079
q9	4992	4766	4691	4691
q10	6797	2306	1899	1899
q11	460	268	258	258
q12	345	354	217	217
q13	17777	3723	3132	3132
q14	227	222	211	211
q15	531	487	482	482
q16	635	613	587	587
q17	580	872	361	361
q18	7050	6387	6378	6378
q19	1344	968	551	551
q20	321	338	191	191
q21	2922	2109	1904	1904
q22	1035	1031	962	962
Total cold run time: 103178 ms
Total hot run time: 32423 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5151	5184	5151	5151
q2	233	324	226	226
q3	2180	2666	2312	2312
q4	1421	1802	1379	1379
q5	4226	4121	4201	4121
q6	208	173	124	124
q7	1879	1906	1758	1758
q8	2593	2553	2538	2538
q9	7267	7200	7243	7200
q10	3006	3289	2794	2794
q11	582	511	486	486
q12	683	785	600	600
q13	3368	3932	3306	3306
q14	280	298	271	271
q15	536	475	479	475
q16	657	690	659	659
q17	1133	1609	1335	1335
q18	7698	7504	7552	7504
q19	820	867	1070	867
q20	1971	2006	1912	1912
q21	5533	5017	4794	4794
q22	1061	1087	1083	1083
Total cold run time: 52486 ms
Total hot run time: 50895 ms

@doris-robot

doris-robot commented Mar 12, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192680 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be6aeb661aacf9bc32cc793bea917272639dc1d4, data reload: false
query1	1426	1013	976	976
query2	6139	1869	1874	1869
query3	11169	4567	4807	4567
query4	54656	25674	23799	23799
query5	5314	512	481	481
query6	371	217	199	199
query7	5074	492	288	288
query8	331	251	241	241
query9	6537	2613	2631	2613
query10	426	313	258	258
query11	15265	15059	14943	14943
query12	163	110	105	105
query13	1189	519	405	405
query14	10173	6518	6980	6518
query15	204	192	184	184
query16	7109	647	503	503
query17	1101	738	599	599
query18	1547	401	331	331
query19	206	199	164	164
query20	132	137	132	132
query21	208	121	109	109
query22	4595	4705	4480	4480
query23	34060	33401	33196	33196
query24	5753	2418	2405	2405
query25	455	449	397	397
query26	713	279	155	155
query27	1732	486	328	328
query28	2798	2467	2443	2443
query29	585	567	425	425
query30	276	214	193	193
query31	889	884	816	816
query32	71	61	59	59
query33	482	356	314	314
query34	757	833	514	514
query35	822	833	747	747
query36	944	999	892	892
query37	119	103	77	77
query38	4288	4146	4267	4146
query39	1462	1443	1430	1430
query40	201	113	101	101
query41	52	50	48	48
query42	127	106	102	102
query43	479	513	477	477
query44	1323	817	813	813
query45	181	178	163	163
query46	835	1036	637	637
query47	1835	1847	1792	1792
query48	389	412	323	323
query49	755	530	430	430
query50	729	752	416	416
query51	4255	4340	4292	4292
query52	114	108	99	99
query53	242	262	190	190
query54	497	483	410	410
query55	82	84	81	81
query56	291	268	255	255
query57	1165	1153	1116	1116
query58	247	256	297	256
query59	2630	2789	2638	2638
query60	287	307	263	263
query61	119	117	119	117
query62	720	745	683	683
query63	235	202	191	191
query64	1799	1039	660	660
query65	4521	4421	4411	4411
query66	719	395	304	304
query67	15736	15757	15314	15314
query68	7807	874	490	490
query69	546	319	259	259
query70	1242	1117	1090	1090
query71	499	286	266	266
query72	5537	3672	3879	3672
query73	1323	745	349	349
query74	9099	9087	8971	8971
query75	3639	3157	2700	2700
query76	4281	1209	744	744
query77	622	356	284	284
query78	10006	10151	9272	9272
query79	2456	818	582	582
query80	691	523	442	442
query81	497	255	223	223
query82	578	132	92	92
query83	322	170	147	147
query84	289	89	79	79
query85	814	359	303	303
query86	379	299	270	270
query87	4388	4597	4453	4453
query88	3311	2229	2227	2227
query89	403	314	276	276
query90	1947	208	205	205
query91	149	145	109	109
query92	78	56	54	54
query93	1247	1058	582	582
query94	697	414	305	305
query95	344	275	260	260
query96	486	559	270	270
query97	3344	3382	3332	3332
query98	239	208	213	208
query99	1436	1392	1281	1281
Total cold run time: 298802 ms
Total hot run time: 192680 ms

@doris-robot

doris-robot commented Mar 12, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be6aeb661aacf9bc32cc793bea917272639dc1d4, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.62	0.10	0.11
query5	0.55	0.54	0.55
query6	1.20	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.52	0.53
query10	0.56	0.63	0.58
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.62	0.60	0.61
query14	2.80	2.82	2.68
query15	0.93	0.85	0.86
query16	0.38	0.37	0.37
query17	1.02	1.03	1.03
query18	0.21	0.19	0.19
query19	1.93	1.80	1.94
query20	0.01	0.02	0.01
query21	15.35	0.87	0.55
query22	0.76	1.20	0.66
query23	14.95	1.37	0.64
query24	6.92	1.23	1.26
query25	0.53	0.22	0.20
query26	0.49	0.15	0.14
query27	0.07	0.05	0.04
query28	9.86	0.85	0.43
query29	12.73	3.97	3.32
query30	0.24	0.08	0.06
query31	2.86	0.58	0.37
query32	3.23	0.54	0.46
query33	2.97	3.03	3.08
query34	15.90	5.16	4.53
query35	4.59	4.57	4.55
query36	0.70	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.74 s
Total hot run time: 31.36 s

@924060929

924060929 commented Mar 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@924060929

924060929 commented Mar 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

924060929 added a commit to 924060929/incubator-doris that referenced this pull request Mar 13, 2025
@924060929
cherry pick fromapache#48987
1390015
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
select 100 value, 1 id
union all
select 100, 2
)a;
+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100 |
| 100 |
+----------------------------------+
```
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Mar 13, 2025
@924060929
cherry pick fromapache#48987
b712a5d
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
select 100 value, 1 id
union all
select 100, 2
)a;
+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100 |
| 100 |
+----------------------------------+
```
This was referenced Mar 13, 2025
Closed
Merged
@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32408 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a54d4d1b3abc8b6b507c50f4bd42752ad58d8779, data reload: false
------ Round 1 ----------------------------------
q1	17638	5193	4993	4993
q2	2041	288	159	159
q3	10433	1260	723	723
q4	10193	1013	562	562
q5	7515	2432	2324	2324
q6	188	167	132	132
q7	923	738	619	619
q8	9320	1271	1101	1101
q9	4899	4666	4709	4666
q10	7064	2365	1911	1911
q11	481	275	279	275
q12	347	355	213	213
q13	17792	3626	3092	3092
q14	227	229	212	212
q15	532	477	475	475
q16	630	627	579	579
q17	586	844	350	350
q18	6991	6398	6396	6396
q19	1546	936	555	555
q20	318	334	193	193
q21	2831	2095	1901	1901
q22	1017	992	977	977
Total cold run time: 103512 ms
Total hot run time: 32408 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5125	5072	5071	5071
q2	243	322	236	236
q3	2125	2618	2280	2280
q4	1406	1796	1401	1401
q5	4199	4130	4183	4130
q6	208	161	123	123
q7	1857	1865	1727	1727
q8	2606	2682	2592	2592
q9	7281	7244	7152	7152
q10	3008	3218	2816	2816
q11	579	501	476	476
q12	675	768	634	634
q13	3360	3950	3291	3291
q14	280	292	285	285
q15	545	500	498	498
q16	661	673	637	637
q17	1152	1548	1362	1362
q18	7750	7564	7640	7564
q19	803	829	965	829
q20	1982	2001	1909	1909
q21	5488	5021	4778	4778
q22	1104	1062	1047	1047
Total cold run time: 52437 ms
Total hot run time: 50838 ms

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191602 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a54d4d1b3abc8b6b507c50f4bd42752ad58d8779, data reload: false
query1	1399	1015	976	976
query2	6179	1868	1848	1848
query3	11066	4583	4398	4398
query4	26415	23744	23309	23309
query5	5156	661	469	469
query6	308	207	202	202
query7	3987	498	290	290
query8	305	244	241	241
query9	8520	2599	2607	2599
query10	507	309	252	252
query11	15905	15020	14935	14935
query12	167	113	115	113
query13	1724	530	424	424
query14	10343	6625	6358	6358
query15	211	193	167	167
query16	7653	669	492	492
query17	1532	776	596	596
query18	2081	402	309	309
query19	202	190	157	157
query20	133	127	128	127
query21	201	126	106	106
query22	4580	4599	4312	4312
query23	34561	33349	33296	33296
query24	7106	2455	2428	2428
query25	467	466	419	419
query26	1133	258	155	155
query27	2352	512	325	325
query28	4139	2444	2415	2415
query29	562	534	435	435
query30	282	226	198	198
query31	928	876	776	776
query32	78	65	62	62
query33	557	373	299	299
query34	796	873	497	497
query35	806	856	749	749
query36	1016	1006	922	922
query37	123	104	75	75
query38	4285	4200	4202	4200
query39	1506	1441	1422	1422
query40	210	119	103	103
query41	53	51	50	50
query42	126	110	109	109
query43	501	493	490	490
query44	1290	794	781	781
query45	184	177	164	164
query46	843	1038	644	644
query47	1842	1897	1776	1776
query48	382	413	312	312
query49	803	507	423	423
query50	702	747	415	415
query51	4270	4278	4266	4266
query52	111	105	95	95
query53	266	257	184	184
query54	505	517	406	406
query55	88	82	84	82
query56	281	262	257	257
query57	1174	1169	1132	1132
query58	252	245	238	238
query59	2753	2676	2886	2676
query60	296	283	258	258
query61	119	119	120	119
query62	793	764	665	665
query63	231	195	186	186
query64	4012	1044	753	753
query65	4523	4451	4421	4421
query66	967	400	318	318
query67	16545	15434	15229	15229
query68	8526	861	492	492
query69	477	297	261	261
query70	1225	1128	1131	1128
query71	430	296	253	253
query72	5016	3584	3794	3584
query73	1170	746	356	356
query74	9409	9015	8893	8893
query75	3964	3181	2725	2725
query76	4690	1191	738	738
query77	913	359	266	266
query78	9759	10058	9244	9244
query79	1559	875	567	567
query80	660	518	445	445
query81	464	257	217	217
query82	255	126	97	97
query83	171	177	151	151
query84	282	94	75	75
query85	743	364	307	307
query86	332	309	285	285
query87	4521	4733	4421	4421
query88	2874	2258	2266	2258
query89	410	308	281	281
query90	2029	206	210	206
query91	135	136	114	114
query92	74	59	54	54
query93	1324	1049	575	575
query94	663	404	307	307
query95	350	262	255	255
query96	483	553	273	273
query97	3309	3451	3314	3314
query98	222	202	199	199
query99	1348	1402	1356	1356
Total cold run time: 281180 ms
Total hot run time: 191602 ms

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a54d4d1b3abc8b6b507c50f4bd42752ad58d8779, data reload: false
query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.56	0.55	0.57
query6	1.20	0.72	0.73
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.52	0.53
query10	0.61	0.60	0.57
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.66	2.68	2.80
query15	0.94	0.85	0.87
query16	0.38	0.39	0.37
query17	1.03	1.02	1.06
query18	0.20	0.19	0.19
query19	1.91	1.79	2.02
query20	0.01	0.01	0.01
query21	15.36	0.92	0.53
query22	0.76	1.12	0.64
query23	15.00	1.39	0.65
query24	7.37	1.33	1.16
query25	0.47	0.17	0.21
query26	0.60	0.16	0.12
query27	0.05	0.06	0.05
query28	9.86	0.85	0.43
query29	12.53	4.11	3.39
query30	0.26	0.09	0.07
query31	2.82	0.59	0.39
query32	3.23	0.55	0.50
query33	2.97	3.00	3.04
query34	15.70	5.10	4.48
query35	4.52	4.54	4.53
query36	0.68	0.51	0.48
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.71 s
Total hot run time: 31.25 s

@924060929

924060929 commented Mar 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32274 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 71d1738d6a45f31a651df4c7d5db6a92ccfbcb66, data reload: false
------ Round 1 ----------------------------------
q1	17589	5289	5089	5089
q2	2035	281	161	161
q3	10447	1382	720	720
q4	10207	1023	519	519
q5	7550	2413	2340	2340
q6	192	167	137	137
q7	923	734	608	608
q8	9299	1281	1059	1059
q9	4927	4572	4886	4572
q10	6813	2301	1904	1904
q11	464	272	254	254
q12	353	341	223	223
q13	17791	3668	3056	3056
q14	229	218	206	206
q15	526	473	475	473
q16	649	628	578	578
q17	578	880	338	338
q18	6755	6408	6412	6408
q19	1206	955	548	548
q20	316	326	195	195
q21	2792	2203	1902	1902
q22	1045	1018	984	984
Total cold run time: 102686 ms
Total hot run time: 32274 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5208	5171	5119	5119
q2	236	326	234	234
q3	2165	2716	2290	2290
q4	1469	1844	1385	1385
q5	4234	4122	4169	4122
q6	208	165	125	125
q7	1889	1916	1806	1806
q8	2602	2548	2595	2548
q9	7262	7109	7240	7109
q10	3021	3168	2732	2732
q11	580	524	517	517
q12	700	749	630	630
q13	3500	3903	3274	3274
q14	284	289	267	267
q15	521	476	479	476
q16	661	684	647	647
q17	1153	1603	1359	1359
q18	7736	7587	7418	7418
q19	824	884	1123	884
q20	1980	2009	1868	1868
q21	5406	4919	4903	4903
q22	1095	1015	984	984
Total cold run time: 52734 ms
Total hot run time: 50697 ms

@924060929

924060929 commented Mar 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

924060929 added a commit to 924060929/incubator-doris that referenced this pull request Mar 13, 2025
@924060929
cherry pick fromapache#48987
f670357
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
select 100 value, 1 id
union all
select 100, 2
)a;
+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100 |
| 100 |
+----------------------------------+
```
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Mar 13, 2025
@924060929
cherry pick fromapache#48987
ee5ff07
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
select 100 value, 1 id
union all
select 100, 2
)a;
+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100 |
| 100 |
+----------------------------------+
```
@924060929

924060929 commented Mar 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32064 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 36e73bb3784d7cf20e7395d85d95f2d8ac756d5a, data reload: false
------ Round 1 ----------------------------------
q1	17570	5206	5017	5017
q2	2052	285	164	164
q3	10423	1295	697	697
q4	10211	1042	524	524
q5	7518	2385	2296	2296
q6	182	160	131	131
q7	934	730	607	607
q8	9294	1276	1046	1046
q9	4968	4565	4610	4565
q10	6817	2311	1867	1867
q11	470	267	261	261
q12	354	341	218	218
q13	17778	3705	3096	3096
q14	229	225	214	214
q15	522	481	481	481
q16	637	605	573	573
q17	564	842	342	342
q18	6882	6453	6441	6441
q19	1373	952	544	544
q20	322	319	189	189
q21	2842	2136	1851	1851
q22	1035	984	940	940
Total cold run time: 102977 ms
Total hot run time: 32064 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5103	5035	5053	5035
q2	236	335	226	226
q3	2159	2658	2268	2268
q4	1452	1841	1389	1389
q5	4220	4147	4146	4146
q6	207	165	125	125
q7	1882	1821	1815	1815
q8	2597	2621	2596	2596
q9	7303	7141	7142	7141
q10	2990	3193	2710	2710
q11	584	495	508	495
q12	723	789	622	622
q13	3439	3831	3180	3180
q14	285	299	276	276
q15	519	487	464	464
q16	653	682	666	666
q17	1126	1594	1334	1334
q18	7794	7515	7565	7515
q19	789	815	907	815
q20	1950	1991	1966	1966
q21	5286	4896	4813	4813
q22	1077	1001	1012	1001
Total cold run time: 52374 ms
Total hot run time: 50598 ms

924060929 added a commit to 924060929/incubator-doris that referenced this pull request Mar 13, 2025
@924060929
cherry pick fromapache#48987
f6832ee
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
select 100 value, 1 id
union all
select 100, 2
)a;
+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100 |
| 100 |
+----------------------------------+
```
@924060929

924060929 commented Mar 13, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32211 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7fad9295f56fda5f2d77a99bdc04c46affba362a, data reload: false
------ Round 1 ----------------------------------
q1	17627	5177	5027	5027
q2	2045	292	161	161
q3	10582	1289	689	689
q4	10228	999	549	549
q5	7620	2317	2327	2317
q6	184	162	132	132
q7	894	717	621	621
q8	9299	1219	1038	1038
q9	4923	4872	4817	4817
q10	6815	2306	1888	1888
q11	472	264	250	250
q12	341	354	214	214
q13	17770	3659	3048	3048
q14	226	223	216	216
q15	538	475	480	475
q16	618	625	599	599
q17	559	856	320	320
q18	6964	6529	6236	6236
q19	1243	937	532	532
q20	308	324	192	192
q21	2707	2089	1939	1939
q22	1038	997	951	951
Total cold run time: 103001 ms
Total hot run time: 32211 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5117	5091	5098	5091
q2	234	329	224	224
q3	2141	2659	2291	2291
q4	1412	1844	1351	1351
q5	4205	4128	4094	4094
q6	209	167	122	122
q7	1846	1802	1804	1802
q8	2576	2663	2567	2567
q9	7221	7084	7261	7084
q10	2954	3249	2796	2796
q11	568	512	502	502
q12	737	788	593	593
q13	3556	3920	3268	3268
q14	276	305	274	274
q15	520	475	480	475
q16	671	692	656	656
q17	1125	1496	1406	1406
q18	7750	7583	7560	7560
q19	788	832	869	832
q20	1963	2025	1921	1921
q21	5439	4899	4916	4899
q22	1053	1009	995	995
Total cold run time: 52361 ms
Total hot run time: 50803 ms

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 185145 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7fad9295f56fda5f2d77a99bdc04c46affba362a, data reload: false
query1	964	391	381	381
query2	6520	1956	1900	1900
query3	6801	217	213	213
query4	27125	23703	23071	23071
query5	4299	663	498	498
query6	301	204	194	194
query7	4605	509	290	290
query8	306	252	256	252
query9	8595	2643	2631	2631
query10	465	311	263	263
query11	15597	15245	14831	14831
query12	165	114	119	114
query13	1666	534	418	418
query14	8781	6564	6636	6564
query15	217	192	177	177
query16	6927	600	481	481
query17	1209	705	566	566
query18	1213	377	319	319
query19	200	195	164	164
query20	124	120	113	113
query21	215	134	109	109
query22	4217	4227	3997	3997
query23	33781	33112	33116	33112
query24	8032	2410	2434	2410
query25	534	445	395	395
query26	1241	275	157	157
query27	2609	479	326	326
query28	4338	2430	2383	2383
query29	674	561	425	425
query30	285	227	188	188
query31	940	832	777	777
query32	77	65	66	65
query33	587	373	310	310
query34	791	842	513	513
query35	816	810	738	738
query36	965	994	918	918
query37	125	102	79	79
query38	4134	4277	4117	4117
query39	1476	1437	1403	1403
query40	219	121	105	105
query41	57	51	49	49
query42	122	105	103	103
query43	505	496	465	465
query44	1282	783	787	783
query45	176	169	166	166
query46	837	1037	629	629
query47	1749	1778	1685	1685
query48	374	414	304	304
query49	779	507	410	410
query50	710	724	423	423
query51	4157	4202	4156	4156
query52	113	106	98	98
query53	243	277	191	191
query54	487	505	416	416
query55	84	79	82	79
query56	281	276	262	262
query57	1128	1138	1050	1050
query58	251	247	262	247
query59	2542	2522	2584	2522
query60	290	290	267	267
query61	124	123	117	117
query62	809	747	673	673
query63	231	198	210	198
query64	4434	998	663	663
query65	4534	4356	4340	4340
query66	1125	397	309	309
query67	15704	15444	15228	15228
query68	8604	937	502	502
query69	473	293	271	271
query70	1247	1142	1116	1116
query71	475	306	260	260
query72	5686	3569	3582	3569
query73	808	753	343	343
query74	9136	9008	8640	8640
query75	3960	3184	2691	2691
query76	3721	1186	776	776
query77	774	380	286	286
query78	10026	10122	9250	9250
query79	2753	824	601	601
query80	651	519	474	474
query81	495	265	227	227
query82	612	125	97	97
query83	221	170	148	148
query84	290	92	78	78
query85	787	346	326	326
query86	401	289	289	289
query87	4572	4579	4260	4260
query88	3575	2257	2246	2246
query89	396	315	279	279
query90	1825	219	222	219
query91	145	136	110	110
query92	74	61	61	61
query93	1868	1070	576	576
query94	666	394	319	319
query95	361	274	264	264
query96	484	567	283	283
query97	3326	3378	3347	3347
query98	231	213	203	203
query99	1417	1428	1234	1234
Total cold run time: 275324 ms
Total hot run time: 185145 ms

@doris-robot

doris-robot commented Mar 13, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7fad9295f56fda5f2d77a99bdc04c46affba362a, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.58	0.56	0.56
query6	1.20	0.73	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.60	0.52	0.50
query10	0.57	0.61	0.60
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.60
query14	2.81	2.69	2.66
query15	0.92	0.86	0.84
query16	0.39	0.41	0.37
query17	1.04	1.01	1.02
query18	0.21	0.19	0.19
query19	1.86	1.89	1.95
query20	0.01	0.01	0.01
query21	15.38	0.92	0.54
query22	0.76	1.25	0.65
query23	15.10	1.39	0.62
query24	6.75	2.35	0.56
query25	0.54	0.23	0.13
query26	0.65	0.15	0.14
query27	0.05	0.05	0.05
query28	9.63	0.88	0.42
query29	12.56	3.98	3.30
query30	0.25	0.09	0.06
query31	2.82	0.60	0.39
query32	3.23	0.54	0.47
query33	3.05	3.04	3.05
query34	15.85	5.13	4.53
query35	4.59	4.54	4.54
query36	0.65	0.49	0.50
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.46 s
Total hot run time: 30.69 s

yiguolei pushed a commit that referenced this pull request Mar 14, 2025
@924060929
branch-2.1: [fix](nereids) fix distinct window compute wrong result (#…
7123bbf
…48987) (#49010)
cherry pick from #48987
morrySnow
morrySnow reviewed Mar 14, 2025
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ProjectToGlobalAggregate.java
Comment threadregression-test/suites/nereids_p0/aggregate/select_distinct.groovy
sql "select distinct sum(value) over(partition by id) from (select 100 value, 1 id union all select 100, 2)a"
result([[100L]])
}

@morrySnowmorrySnowMar 14, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add more case to cover all if branch in rules, such as

  • qualify with alias
  • qualify with new window function
  • qualify with window function which has exsited in project list
  • having

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ProjectToGlobalAggregate.java
Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ProjectToGlobalAggregate.java
Comment thread...re/src/main/java/org/apache/doris/nereids/rules/analysis/ReplaceExpressionByChildOutput.java
@924060929
924060929 marked this pull request as draft March 14, 2025 07:06
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Mar 14, 2025
@924060929
Revert "branch-2.1: [fix](nereids) fix distinct window compute wrong …
3f21793
…result (apache#48987) (apache#49010)"
This reverts commit 7123bbf.
@924060929924060929 mentioned this pull request Mar 14, 2025
Merged
924060929 added a commit that referenced this pull request Mar 14, 2025
@924060929
Revert "branch-2.1: [fix](nereids) fix distinct window compute wrong …
8589db3
…result (#48987) (#49010)" (#49079)
Revert "branch-2.1: [fix](nereids) fix distinct window compute wrong
result (#48987) (#49010)"
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowmorrySnow left review comments
@engleflyengleflyAwaiting requested review from engleflyenglefly will be requested when the pull request is marked ready for reviewenglefly is a code owner
@starocean999starocean999Awaiting requested review from starocean999starocean999 will be requested when the pull request is marked ready for reviewstarocean999 is a code owner

Assignees

No one assigned

Labels

usercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@924060929@hello-stephen@doris-robot@morrySnow@yiguolei@wm1581066