[fix](statistics)Use column update rows to decide min/max stats are valid or not. by Jibing-Li · Pull Request #34263 · apache/doris

Jibing-Li · 2024-04-28T13:43:05Z

This is a following pr of #33685
After #33703 merged, need to check update rows in column level instead of table level.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-28T13:43:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Jibing-Li · 2024-04-28T13:49:35Z

run buildall

doris-robot · 2024-04-28T14:07:58Z

TPC-H: Total hot run time: 40214 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 08aac89cda84fa0636e235c1ea6bcfbbc99f4d9f, data reload: false
------ Round 1 ----------------------------------
q1	18087	4509	4406	4406
q2	2593	197	205	197
q3	11144	1192	1223	1192
q4	10205	783	840	783
q5	7534	2860	2733	2733
q6	214	131	135	131
q7	1034	638	619	619
q8	9261	2114	2061	2061
q9	9632	6647	6581	6581
q10	8761	3715	3738	3715
q11	449	234	238	234
q12	426	219	220	219
q13	17772	2976	2966	2966
q14	266	225	229	225
q15	502	486	467	467
q16	524	389	393	389
q17	952	621	719	621
q18	7870	7426	7582	7426
q19	5419	1481	1560	1481
q20	680	319	316	316
q21	4945	3183	3850	3183
q22	340	279	269	269
Total cold run time: 118610 ms
Total hot run time: 40214 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4367	4251	4288	4251
q2	361	273	267	267
q3	3021	2738	2738	2738
q4	1896	1602	1588	1588
q5	5300	5295	5288	5288
q6	211	124	124	124
q7	2272	1886	1895	1886
q8	3225	3344	3336	3336
q9	8584	8512	8551	8512
q10	3917	3761	3713	3713
q11	597	494	481	481
q12	781	609	582	582
q13	17133	2946	2903	2903
q14	308	284	265	265
q15	513	475	482	475
q16	488	432	427	427
q17	1779	1472	1460	1460
q18	7518	7461	7370	7370
q19	1645	1483	1553	1483
q20	2013	1774	1759	1759
q21	4947	4877	4847	4847
q22	575	498	501	498
Total cold run time: 71451 ms
Total hot run time: 54253 ms

doris-robot · 2024-04-28T14:19:59Z

TPC-DS: Total hot run time: 187436 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 08aac89cda84fa0636e235c1ea6bcfbbc99f4d9f, data reload: false
query1	916	368	356	356
query2	6452	2501	2461	2461
query3	6670	209	210	209
query4	22903	21245	21141	21141
query5	4151	430	425	425
query6	282	188	177	177
query7	4600	291	293	291
query8	250	194	191	191
query9	8401	2368	2338	2338
query10	425	245	261	245
query11	14606	14246	14275	14246
query12	140	87	88	87
query13	1632	368	380	368
query14	8644	8589	8296	8296
query15	258	168	177	168
query16	8177	268	265	265
query17	1887	583	543	543
query18	2100	276	261	261
query19	336	144	151	144
query20	113	90	90	90
query21	199	129	127	127
query22	5016	4909	4786	4786
query23	33893	33317	33140	33140
query24	10734	2995	2996	2995
query25	610	395	383	383
query26	1162	156	159	156
query27	2361	322	336	322
query28	6949	2053	2060	2053
query29	870	638	607	607
query30	298	151	152	151
query31	1002	768	717	717
query32	97	54	55	54
query33	754	256	257	256
query34	1021	500	493	493
query35	809	677	689	677
query36	1087	953	904	904
query37	138	67	70	67
query38	3188	3036	3035	3035
query39	1594	1562	1555	1555
query40	207	131	126	126
query41	44	44	41	41
query42	111	95	100	95
query43	575	567	536	536
query44	1217	737	732	732
query45	281	267	283	267
query46	1106	744	734	734
query47	1930	1866	1873	1866
query48	379	293	300	293
query49	1066	405	413	405
query50	779	399	392	392
query51	6723	6609	6630	6609
query52	100	93	95	93
query53	351	287	280	280
query54	319	240	247	240
query55	81	76	76	76
query56	257	228	231	228
query57	1213	1148	1125	1125
query58	242	206	204	204
query59	3464	3265	3405	3265
query60	267	246	251	246
query61	113	108	178	108
query62	631	459	444	444
query63	309	279	288	279
query64	8559	7154	7184	7154
query65	3094	3036	3125	3036
query66	913	335	337	335
query67	15430	15061	14957	14957
query68	9022	557	559	557
query69	539	326	307	307
query70	1293	1094	1071	1071
query71	495	272	273	272
query72	8025	2634	2415	2415
query73	1559	331	331	331
query74	6562	6110	6114	6110
query75	4191	2705	2645	2645
query76	5486	921	980	921
query77	672	269	265	265
query78	11094	10086	10209	10086
query79	13010	521	526	521
query80	2575	437	425	425
query81	517	220	216	216
query82	776	96	90	90
query83	219	164	160	160
query84	261	88	82	82
query85	1525	294	260	260
query86	405	302	268	268
query87	3371	3103	3095	3095
query88	5588	2438	2459	2438
query89	527	365	364	364
query90	1971	202	185	185
query91	130	99	99	99
query92	66	47	46	46
query93	7367	522	505	505
query94	1248	181	183	181
query95	397	303	298	298
query96	617	274	263	263
query97	3139	2899	2914	2899
query98	235	225	222	222
query99	1278	893	898	893
Total cold run time: 308646 ms
Total hot run time: 187436 ms

Jibing-Li · 2024-04-29T04:05:14Z

run buildall

doris-robot · 2024-04-29T05:04:56Z

TPC-H: Total hot run time: 40902 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9c931bec28bcefb46796e85de0987f60c942040, data reload: false
------ Round 1 ----------------------------------
q1	17677	4514	4382	4382
q2	2619	195	207	195
q3	11304	1274	1193	1193
q4	10579	774	790	774
q5	8024	2726	2701	2701
q6	218	135	135	135
q7	971	557	544	544
q8	9321	2148	2144	2144
q9	9809	7358	7145	7145
q10	8868	3789	3736	3736
q11	457	242	239	239
q12	504	223	230	223
q13	17801	2921	2944	2921
q14	279	233	228	228
q15	517	491	474	474
q16	526	384	387	384
q17	988	680	685	680
q18	8131	7519	7453	7453
q19	1651	1548	1549	1548
q20	645	320	312	312
q21	5054	3942	3222	3222
q22	337	269	281	269
Total cold run time: 116280 ms
Total hot run time: 40902 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4319	4180	4198	4180
q2	386	273	273	273
q3	3021	2789	2748	2748
q4	1892	1574	1613	1574
q5	5286	5320	5282	5282
q6	215	123	125	123
q7	1781	1433	1427	1427
q8	3261	3356	3343	3343
q9	9271	9310	9311	9310
q10	3912	3683	3761	3683
q11	592	488	483	483
q12	760	597	620	597
q13	16354	2943	2934	2934
q14	300	275	285	275
q15	507	485	477	477
q16	483	413	416	413
q17	1750	1476	1483	1476
q18	7840	7720	7474	7474
q19	4665	1558	1593	1558
q20	2023	1743	1772	1743
q21	5059	4711	4826	4711
q22	555	484	487	484
Total cold run time: 74232 ms
Total hot run time: 54568 ms

doris-robot · 2024-04-29T05:16:40Z

TPC-DS: Total hot run time: 186519 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9c931bec28bcefb46796e85de0987f60c942040, data reload: false
query1	916	369	346	346
query2	6457	2502	2389	2389
query3	6659	204	204	204
query4	24527	21152	21334	21152
query5	4201	425	408	408
query6	271	187	198	187
query7	4590	295	287	287
query8	252	213	212	212
query9	8419	2383	2364	2364
query10	434	239	246	239
query11	14720	14033	14117	14033
query12	151	93	90	90
query13	1644	374	356	356
query14	10607	8245	8202	8202
query15	258	166	170	166
query16	8170	257	265	257
query17	1904	559	541	541
query18	2121	275	270	270
query19	327	149	151	149
query20	92	86	86	86
query21	194	128	127	127
query22	5128	4897	4932	4897
query23	33585	33173	32968	32968
query24	11596	2961	3061	2961
query25	646	361	360	360
query26	1728	151	147	147
query27	2889	313	328	313
query28	7518	2031	2013	2013
query29	1009	607	579	579
query30	291	151	153	151
query31	959	740	730	730
query32	91	51	55	51
query33	733	249	239	239
query34	1044	473	488	473
query35	812	665	687	665
query36	1058	895	902	895
query37	178	64	69	64
query38	3201	3026	3001	3001
query39	1555	1546	1525	1525
query40	269	126	125	125
query41	43	40	38	38
query42	106	96	100	96
query43	584	526	531	526
query44	1275	723	724	723
query45	277	266	259	259
query46	1089	748	723	723
query47	1929	1889	1892	1889
query48	378	292	295	292
query49	1158	416	393	393
query50	805	377	395	377
query51	6644	6622	6654	6622
query52	99	94	89	89
query53	354	281	275	275
query54	310	225	228	225
query55	75	72	74	72
query56	235	216	217	216
query57	1270	1119	1152	1119
query58	235	194	194	194
query59	3524	3176	3117	3117
query60	257	227	235	227
query61	92	88	87	87
query62	668	439	445	439
query63	301	271	271	271
query64	9502	7123	7142	7123
query65	3083	3030	3017	3017
query66	1367	338	335	335
query67	15360	15097	15290	15097
query68	5193	542	548	542
query69	485	314	293	293
query70	1169	1081	1119	1081
query71	398	277	264	264
query72	7305	2592	2396	2396
query73	709	322	331	322
query74	6590	6162	6037	6037
query75	3393	2672	2678	2672
query76	3330	976	968	968
query77	402	260	258	258
query78	10918	10363	10214	10214
query79	2908	521	515	515
query80	2020	423	423	423
query81	519	217	215	215
query82	767	90	93	90
query83	276	166	173	166
query84	264	84	85	84
query85	2166	265	250	250
query86	532	297	295	295
query87	3267	3095	3087	3087
query88	4391	2409	2395	2395
query89	464	358	369	358
query90	2007	180	178	178
query91	121	95	96	95
query92	62	47	48	47
query93	3869	522	507	507
query94	1266	175	178	175
query95	397	294	296	294
query96	601	265	266	265
query97	3108	2908	2955	2908
query98	235	224	216	216
query99	1254	846	851	846
Total cold run time: 292610 ms
Total hot run time: 186519 ms

Jibing-Li · 2024-04-29T11:45:26Z

run buildall

github-actions · 2024-04-29T11:46:50Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-29T11:46:53Z

PR approved by anyone and no changes requested.

doris-robot · 2024-04-29T12:11:51Z

TPC-H: Total hot run time: 41007 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fba9e9541dc673ae721b027f91778bd59a5e0b9c, data reload: false
------ Round 1 ----------------------------------
q1	17658	4884	4376	4376
q2	2530	202	196	196
q3	11117	1169	1186	1169
q4	10399	800	916	800
q5	7588	2753	2811	2753
q6	218	130	133	130
q7	958	554	550	550
q8	9388	2105	2071	2071
q9	10439	7390	7163	7163
q10	8791	3712	3744	3712
q11	460	246	230	230
q12	422	215	216	215
q13	17872	2941	2962	2941
q14	268	229	235	229
q15	513	461	462	461
q16	525	383	380	380
q17	957	699	727	699
q18	8018	7484	7649	7484
q19	4100	1554	1519	1519
q20	692	329	326	326
q21	5050	3329	3890	3329
q22	336	284	274	274
Total cold run time: 118299 ms
Total hot run time: 41007 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4347	4212	4259	4212
q2	375	265	261	261
q3	3053	2737	2713	2713
q4	1872	1605	1553	1553
q5	5304	5280	5312	5280
q6	213	124	124	124
q7	1764	1427	1426	1426
q8	3191	3347	3325	3325
q9	9329	9235	9234	9234
q10	3836	3702	3687	3687
q11	583	486	488	486
q12	745	598	596	596
q13	16574	2962	2958	2958
q14	307	277	264	264
q15	508	470	478	470
q16	483	401	421	401
q17	1778	1500	1493	1493
q18	7509	7513	7511	7511
q19	1662	1553	1591	1553
q20	1956	1730	1781	1730
q21	4909	4827	4753	4753
q22	551	471	495	471
Total cold run time: 70849 ms
Total hot run time: 54501 ms

doris-robot · 2024-04-29T12:23:37Z

TPC-DS: Total hot run time: 186217 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fba9e9541dc673ae721b027f91778bd59a5e0b9c, data reload: false
query1	917	362	347	347
query2	6445	2331	2374	2331
query3	6651	212	217	212
query4	24047	21166	21371	21166
query5	4149	421	439	421
query6	265	178	174	174
query7	4592	282	281	281
query8	243	181	190	181
query9	8395	2346	2328	2328
query10	438	248	263	248
query11	14750	14100	14230	14100
query12	141	87	84	84
query13	1653	357	355	355
query14	10728	8339	8194	8194
query15	260	161	170	161
query16	8151	258	261	258
query17	1864	579	535	535
query18	2108	272	267	267
query19	265	148	150	148
query20	89	85	84	84
query21	192	126	130	126
query22	5047	4821	4797	4797
query23	33766	33063	33123	33063
query24	10811	2920	2883	2883
query25	622	361	350	350
query26	1366	152	146	146
query27	2924	314	313	313
query28	7643	2004	2003	2003
query29	882	590	587	587
query30	283	148	151	148
query31	1004	724	710	710
query32	92	56	55	55
query33	748	244	237	237
query34	1096	459	482	459
query35	815	691	691	691
query36	1058	890	909	890
query37	135	65	68	65
query38	3128	2966	3000	2966
query39	1593	1538	1549	1538
query40	199	124	126	124
query41	41	39	38	38
query42	103	99	98	98
query43	582	557	527	527
query44	1264	730	749	730
query45	276	256	276	256
query46	1076	709	724	709
query47	1929	1827	1847	1827
query48	369	293	302	293
query49	1067	389	392	389
query50	767	382	373	373
query51	6921	6693	6744	6693
query52	98	94	92	92
query53	352	278	279	278
query54	291	235	264	235
query55	96	72	74	72
query56	240	214	224	214
query57	1188	1155	1113	1113
query58	223	194	196	194
query59	3640	3047	3235	3047
query60	247	230	235	230
query61	88	86	85	85
query62	661	459	438	438
query63	309	274	275	274
query64	9269	7146	7558	7146
query65	3151	3052	3052	3052
query66	1390	340	341	340
query67	15561	15039	15027	15027
query68	6214	526	520	520
query69	518	302	304	302
query70	1141	1155	1117	1117
query71	443	275	287	275
query72	8088	2647	2373	2373
query73	728	317	324	317
query74	6569	6118	6083	6083
query75	3583	2636	2633	2633
query76	3651	1006	1012	1006
query77	630	267	269	267
query78	10936	10187	10242	10187
query79	4501	530	533	530
query80	1138	420	424	420
query81	512	222	215	215
query82	1195	96	91	91
query83	194	167	170	167
query84	265	90	81	81
query85	1408	323	254	254
query86	461	304	295	295
query87	3259	3068	3061	3061
query88	4731	2322	2328	2322
query89	484	378	372	372
query90	1990	182	179	179
query91	125	97	96	96
query92	57	48	52	48
query93	5206	512	494	494
query94	1150	181	177	177
query95	383	296	300	296
query96	595	271	260	260
query97	3165	2932	2913	2913
query98	229	224	215	215
query99	1281	872	855	855
Total cold run time: 295584 ms
Total hot run time: 186217 ms

Jibing-Li · 2024-04-29T13:46:43Z

run external

…valid or not (#34263) This is a following pr of #33685 After #33703 merged, need to check update rows in column level instead of table level.

…ats are valid or not (#34263)" This reverts commit 8db4d48.

…valid or not (#34263) This is a following pr of #33685 After #33703 merged, need to check update rows in column level instead of table level.

…ats are valid or not (#34263)" This reverts commit aeafc99.

…valid or not (apache#34263) This is a following pr of apache#33685 After apache#33703 merged, need to check update rows in column level instead of table level.

Jibing-Li force-pushed the updaterows branch 2 times, most recently from 37882e4 to 08aac89Compare April 28, 2024 13:47

Jibing-Li marked this pull request as ready for review April 28, 2024 13:49

Jibing-Li force-pushed the updaterows branch 2 times, most recently from 5b41509 to b9c931bCompare April 29, 2024 03:54

englefly reviewed Apr 29, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java Outdated

xiaokang added the dev/2.0.x label Apr 29, 2024

Jibing-Li force-pushed the updaterows branch 3 times, most recently from 7b4b00a to f0fcc5dCompare April 29, 2024 11:21

englefly reviewed Apr 29, 2024
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java Outdated

Use column update rows to decide min/max stats are valid or not.
fba9e95

Jibing-Li force-pushed the updaterows branch from f0fcc5d to fba9e95Compare April 29, 2024 11:34

englefly approved these changes Apr 29, 2024
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 29, 2024

morrySnow approved these changes Apr 30, 2024
View reviewed changes

morrySnow merged commit 4d69cb2 into apache:masterApr 30, 2024

morrySnow added not-merge/2.0 do not merge into 2.0 branch and removed dev/2.0.x labels Apr 30, 2024

Jibing-Li deleted the updaterows branch May 6, 2024 02:14

yiguolei pushed a commit that referenced this pull request May 6, 2024
[fix](statistics) Use column update rows to decide min/max stats are …
8db4d48
…valid or not (#34263) This is a following pr of #33685 After #33703 merged, need to check update rows in column level instead of table level.

yiguolei pushed a commit that referenced this pull request May 7, 2024
Revert "[fix](statistics) Use column update rows to decide min/max st…
63cd632
…ats are valid or not (#34263)" This reverts commit 8db4d48.

yiguolei added the 2.1.0-conflict label May 7, 2024

dataroaring added a commit that referenced this pull request May 8, 2024
Revert "[fix](statistics) Use column update rows to decide min/max st…
2f3be24
…ats are valid or not (#34263)" This reverts commit aeafc99.

morrySnow added the dev/2.1.x label May 28, 2024

Jibing-Li added not-merge/2.1 and removed 2.1.0-conflict dev/2.1.x labels May 28, 2024

Uh oh!

Conversation

Jibing-Li commented Apr 28, 2024

Further comments

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

Jibing-Li commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

Jibing-Li commented Apr 29, 2024

Uh oh!

doris-robot commented Apr 29, 2024

Uh oh!

doris-robot commented Apr 29, 2024

Uh oh!

Uh oh!

Uh oh!

Jibing-Li commented Apr 29, 2024

Uh oh!

github-actionsBot commented Apr 29, 2024

Uh oh!

github-actionsBot commented Apr 29, 2024

Uh oh!

doris-robot commented Apr 29, 2024

Uh oh!

doris-robot commented Apr 29, 2024

Uh oh!

Jibing-Li commented Apr 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants