[fix](segment cache) estimate momory consumed by segment by dataroaring · Pull Request #35647 · apache/doris

dataroaring · 2024-05-30T07:37:39Z

The memory consumed in segment cache is 0 after
https://github.com/apache/doris/pull/35432/files.

The pr also tracks memory usage of column readers.

Proposed changes

Issue Number: close #xxx

The memory consumed in segment cache is 0 after https://github.com/apache/doris/pull/35432/files. The pr also tracks memory usage of column readers.

doris-robot · 2024-05-30T07:37:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-30T07:44:22Z

@@ -146,5 +154,12 @@ OrdinalPageIndexIterator OrdinalIndexReader::seek_at_or_before(ordinal_t ordinal
 return OrdinalPageIndexIterator(this, left);
 }



warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
OrdinalIndexReader::~OrdinalIndexReader() { ^

github-actions

clang-tidy made some suggestions

github-actions · 2024-05-30T07:51:10Z

 return OrdinalPageIndexIterator(this, left);
 }

+OrdinalIndexReader::~OrdinalIndexReader() {


warning: use '= default' to define a trivial destructor [modernize-use-equals-default]
OrdinalIndexReader::~OrdinalIndexReader() { ^

dataroaring · 2024-05-30T08:04:26Z

run buildall

dataroaring · 2024-05-30T08:28:37Z

run buildall

doris-robot · 2024-05-30T09:01:46Z

TeamCity be ut coverage result:
Function Coverage: 36.27% (9232/25452)
Line Coverage: 27.63% (75719/274043)
Region Coverage: 26.84% (39177/145963)
Branch Coverage: 23.60% (19894/84282)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c8c7f1c811abaa1e865ae87acc456e5969a74b81_c8c7f1c811abaa1e865ae87acc456e5969a74b81/report/index.html

dataroaring · 2024-05-30T09:14:56Z

run buildall

github-actions

clang-tidy made some suggestions

doris-robot · 2024-05-30T10:28:27Z

TeamCity be ut coverage result:
Function Coverage: 36.27% (9232/25453)
Line Coverage: 27.63% (75728/274053)
Region Coverage: 26.85% (39185/145964)
Branch Coverage: 23.61% (19898/84282)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ba424d231e1b0005eb1d3caca4de42dfb71df08d_ba424d231e1b0005eb1d3caca4de42dfb71df08d/report/index.html

doris-robot · 2024-05-30T10:30:16Z

TPC-H: Total hot run time: 41238 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba424d231e1b0005eb1d3caca4de42dfb71df08d, data reload: false
------ Round 1 ----------------------------------
q1	17604	4338	4216	4216
q2	2043	193	189	189
q3	10460	1268	1183	1183
q4	10197	853	824	824
q5	7536	2793	2654	2654
q6	221	134	136	134
q7	990	639	605	605
q8	9224	2200	2116	2116
q9	9112	6779	6828	6779
q10	9266	3891	3904	3891
q11	460	248	262	248
q12	482	223	224	223
q13	17363	3195	3256	3195
q14	273	244	226	226
q15	512	479	479	479
q16	480	387	390	387
q17	996	632	582	582
q18	8532	7917	7813	7813
q19	5990	1647	1537	1537
q20	656	306	311	306
q21	5174	3309	4228	3309
q22	413	344	342	342
Total cold run time: 117984 ms
Total hot run time: 41238 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4519	4428	4382	4382
q2	379	283	273	273
q3	3110	2924	2935	2924
q4	2057	1693	1619	1619
q5	5363	5546	5523	5523
q6	224	124	122	122
q7	2188	1826	1864	1826
q8	3241	3382	3405	3382
q9	8755	8625	8780	8625
q10	4102	3812	3900	3812
q11	598	495	494	494
q12	777	617	605	605
q13	16935	3122	3148	3122
q14	313	259	273	259
q15	532	477	479	477
q16	480	446	427	427
q17	1802	1529	1490	1490
q18	8205	7697	7373	7373
q19	1754	1604	1618	1604
q20	3046	1806	1803	1803
q21	8274	4842	4631	4631
q22	846	520	510	510
Total cold run time: 77500 ms
Total hot run time: 55283 ms

doris-robot · 2024-05-30T10:41:12Z

TPC-DS: Total hot run time: 172408 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba424d231e1b0005eb1d3caca4de42dfb71df08d, data reload: false
query1	932	379	375	375
query2	6461	2413	2414	2413
query3	6638	209	207	207
query4	20938	17502	17340	17340
query5	4139	456	455	455
query6	255	162	157	157
query7	4598	311	285	285
query8	310	292	281	281
query9	8751	2378	2341	2341
query10	462	303	293	293
query11	10752	10142	10001	10001
query12	136	91	91	91
query13	1639	388	376	376
query14	9228	6196	7750	6196
query15	237	196	190	190
query16	7827	277	272	272
query17	1387	548	540	540
query18	1923	279	289	279
query19	208	160	166	160
query20	97	90	84	84
query21	216	133	137	133
query22	4281	4033	3836	3836
query23	33894	33005	33065	33005
query24	11100	2798	2802	2798
query25	639	354	351	351
query26	1259	150	152	150
query27	2327	320	321	320
query28	6930	2056	2058	2056
query29	899	602	598	598
query30	283	152	155	152
query31	960	779	778	778
query32	99	55	54	54
query33	765	278	287	278
query34	964	500	483	483
query35	746	612	599	599
query36	1076	951	928	928
query37	138	66	72	66
query38	2899	2810	2768	2768
query39	860	772	807	772
query40	203	123	123	123
query41	54	52	50	50
query42	121	100	98	98
query43	574	585	549	549
query44	1212	754	759	754
query45	208	193	176	176
query46	1075	711	707	707
query47	1859	1783	1748	1748
query48	367	300	308	300
query49	995	412	406	406
query50	786	385	398	385
query51	6838	6667	6658	6658
query52	109	89	94	89
query53	354	291	297	291
query54	917	455	459	455
query55	79	75	75	75
query56	282	260	261	260
query57	1145	1015	1030	1015
query58	240	255	249	249
query59	3458	3318	3271	3271
query60	293	305	275	275
query61	89	89	91	89
query62	645	463	464	463
query63	318	297	296	296
query64	8854	2223	1750	1750
query65	3207	3075	3107	3075
query66	811	332	336	332
query67	15140	14770	15010	14770
query68	4902	549	544	544
query69	609	418	370	370
query70	1146	1109	1127	1109
query71	484	282	288	282
query72	8020	6050	5636	5636
query73	764	324	322	322
query74	6018	5654	5530	5530
query75	3653	2639	2687	2639
query76	3430	932	886	886
query77	642	303	307	303
query78	10290	9874	9852	9852
query79	1644	521	514	514
query80	1920	465	450	450
query81	565	222	222	222
query82	711	102	108	102
query83	295	178	173	173
query84	261	88	87	87
query85	1072	270	278	270
query86	375	326	322	322
query87	3344	3148	3130	3130
query88	3426	2459	2434	2434
query89	468	395	393	393
query90	1737	181	193	181
query91	136	97	99	97
query92	61	48	49	48
query93	1610	516	495	495
query94	1218	195	189	189
query95	414	324	315	315
query96	591	276	266	266
query97	3162	3017	3053	3017
query98	241	220	218	218
query99	1259	843	842	842
Total cold run time: 271954 ms
Total hot run time: 172408 ms

doris-robot · 2024-05-30T10:46:42Z

ClickBench: Total hot run time: 30.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba424d231e1b0005eb1d3caca4de42dfb71df08d, data reload: false
query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.50	0.49
query6	1.12	0.72	0.71
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.61
query14	0.80	0.79	0.77
query15	0.83	0.82	0.80
query16	0.37	0.37	0.39
query17	0.95	1.02	1.01
query18	0.23	0.25	0.24
query19	1.92	1.70	1.81
query20	0.01	0.01	0.01
query21	15.75	0.66	0.66
query22	4.07	7.22	1.92
query23	18.34	1.43	1.26
query24	1.28	0.35	0.36
query25	0.15	0.08	0.08
query26	0.27	0.17	0.17
query27	0.08	0.07	0.08
query28	13.24	1.02	1.00
query29	12.76	3.32	3.28
query30	0.24	0.06	0.05
query31	2.86	0.39	0.39
query32	3.28	0.48	0.45
query33	2.93	2.95	2.91
query34	16.71	4.41	4.43
query35	4.48	4.55	4.56
query36	0.66	0.47	0.49
query37	0.18	0.16	0.18
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 108.65 s
Total hot run time: 30.81 s

dataroaring · 2024-05-31T01:56:31Z

run buildall

github-actions · 2024-05-31T01:57:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-31T01:57:04Z

PR approved by anyone and no changes requested.

dataroaring · 2024-05-31T02:48:21Z

run performance

doris-robot · 2024-05-31T03:20:00Z

TeamCity be ut coverage result:
Function Coverage: 36.29% (9234/25448)
Line Coverage: 27.64% (75734/274008)
Region Coverage: 26.85% (39206/146007)
Branch Coverage: 23.60% (19895/84292)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6c88e594081f459308ff38fea29de597bec21f6d_6c88e594081f459308ff38fea29de597bec21f6d/report/index.html

doris-robot · 2024-05-31T03:31:14Z

TPC-H: Total hot run time: 41139 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c88e594081f459308ff38fea29de597bec21f6d, data reload: false
------ Round 1 ----------------------------------
q1	17599	4353	4258	4258
q2	2049	192	192	192
q3	10448	1202	1224	1202
q4	10210	915	824	824
q5	7541	2706	2680	2680
q6	217	131	133	131
q7	984	620	602	602
q8	9229	2145	2081	2081
q9	9115	6676	6642	6642
q10	9031	3904	3862	3862
q11	525	249	278	249
q12	458	236	240	236
q13	18094	3256	3135	3135
q14	269	223	237	223
q15	508	483	478	478
q16	510	414	411	411
q17	987	677	608	608
q18	8221	7864	7812	7812
q19	6076	1475	1489	1475
q20	646	318	329	318
q21	5033	3383	4124	3383
q22	402	353	337	337
Total cold run time: 118152 ms
Total hot run time: 41139 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4530	4439	4384	4384
q2	390	267	277	267
q3	3171	2916	2871	2871
q4	2013	1741	1586	1586
q5	5321	5528	5519	5519
q6	224	127	131	127
q7	2213	1859	1872	1859
q8	3238	3350	3415	3350
q9	8703	8665	8735	8665
q10	4085	3905	3906	3905
q11	583	480	486	480
q12	779	617	631	617
q13	16942	3145	3158	3145
q14	290	275	264	264
q15	531	486	490	486
q16	487	430	445	430
q17	1830	1508	1483	1483
q18	8136	7851	7322	7322
q19	1707	1568	1594	1568
q20	3021	1816	1788	1788
q21	8385	4715	4755	4715
q22	624	515	504	504
Total cold run time: 77203 ms
Total hot run time: 55335 ms

doris-robot · 2024-05-31T03:42:11Z

TPC-DS: Total hot run time: 172316 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c88e594081f459308ff38fea29de597bec21f6d, data reload: false
query1	942	382	374	374
query2	6456	2384	2432	2384
query3	6642	213	214	213
query4	20153	17479	17319	17319
query5	4121	465	454	454
query6	249	162	154	154
query7	4577	306	289	289
query8	323	292	295	292
query9	8718	2375	2388	2375
query10	467	299	284	284
query11	10713	10193	10038	10038
query12	133	92	87	87
query13	1639	376	370	370
query14	9495	6363	7777	6363
query15	247	198	192	192
query16	7751	267	270	267
query17	1371	549	553	549
query18	1945	290	280	280
query19	209	163	161	161
query20	98	89	87	87
query21	214	140	134	134
query22	4321	3871	3974	3871
query23	33666	33180	33126	33126
query24	11176	3020	2851	2851
query25	638	367	357	357
query26	1214	154	159	154
query27	2437	319	329	319
query28	6708	2069	2078	2069
query29	871	625	614	614
query30	275	150	165	150
query31	962	745	749	745
query32	94	51	55	51
query33	780	294	284	284
query34	939	480	475	475
query35	743	640	620	620
query36	1073	946	878	878
query37	158	70	75	70
query38	2917	2787	2787	2787
query39	861	787	790	787
query40	210	125	123	123
query41	51	52	49	49
query42	118	95	94	94
query43	584	563	555	555
query44	1194	720	749	720
query45	201	180	166	166
query46	1074	695	702	695
query47	1855	1765	1786	1765
query48	363	299	293	293
query49	1001	412	409	409
query50	770	378	385	378
query51	6867	6686	6731	6686
query52	113	87	92	87
query53	356	291	288	288
query54	876	451	439	439
query55	72	71	72	71
query56	308	251	255	251
query57	1121	1043	1057	1043
query58	259	269	276	269
query59	3571	3509	3225	3225
query60	298	271	264	264
query61	95	89	87	87
query62	654	443	460	443
query63	311	288	292	288
query64	8892	2248	1765	1765
query65	3187	3085	3173	3085
query66	807	322	326	322
query67	15444	14854	15013	14854
query68	4546	549	555	549
query69	512	409	345	345
query70	1164	1029	1164	1029
query71	456	285	284	284
query72	6924	5855	5492	5492
query73	753	324	321	321
query74	6012	5590	5566	5566
query75	3432	2669	2651	2651
query76	2559	894	937	894
query77	495	289	293	289
query78	10340	9861	9734	9734
query79	2583	502	512	502
query80	2252	459	450	450
query81	565	220	224	220
query82	798	104	105	104
query83	302	171	173	171
query84	260	91	81	81
query85	1761	344	264	264
query86	465	326	316	316
query87	3282	3119	3169	3119
query88	3938	2339	2331	2331
query89	486	394	386	386
query90	1789	187	188	187
query91	126	96	98	96
query92	63	52	51	51
query93	2041	527	507	507
query94	1138	188	182	182
query95	404	305	304	304
query96	589	272	268	268
query97	3158	2998	3053	2998
query98	244	217	221	217
query99	1381	840	820	820
Total cold run time: 271766 ms
Total hot run time: 172316 ms

doris-robot · 2024-05-31T03:47:42Z

ClickBench: Total hot run time: 30.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c88e594081f459308ff38fea29de597bec21f6d, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.49	0.48	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.53	0.49	0.49
query10	0.56	0.55	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.60	0.58	0.59
query14	0.78	0.77	0.76
query15	0.84	0.81	0.81
query16	0.35	0.37	0.41
query17	0.95	0.95	0.95
query18	0.22	0.24	0.24
query19	1.78	1.74	1.74
query20	0.01	0.02	0.01
query21	15.51	0.70	0.65
query22	4.35	7.58	2.10
query23	18.32	1.46	1.25
query24	1.67	0.23	0.28
query25	0.17	0.08	0.08
query26	0.26	0.17	0.17
query27	0.09	0.08	0.07
query28	13.32	1.03	1.00
query29	13.30	3.36	3.27
query30	0.24	0.05	0.06
query31	2.88	0.38	0.38
query32	3.28	0.47	0.47
query33	2.90	2.92	2.86
query34	17.27	4.38	4.41
query35	4.54	4.54	4.55
query36	0.72	0.48	0.46
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.17 s
Total hot run time: 30.75 s

Lchangliang · 2024-05-31T06:48:38Z

LGTM

xinyiZzz · 2024-05-31T06:54:44Z


 namespace doris {

+static bvar::Adder<size_t> g_total_rowset_num("doris_total_rowset_num");


BE metrics have a counter all_rowsets_num, is it same as bvar::doris_total_rowset_num?
update_rowsets_and_segments_num_metrics

xinyiZzz · 2024-05-31T07:02:12Z


 namespace doris::segment_v2 {
-bvar::Adder<size_t> g_total_segment_num("doris_total_segment_num");
+static bvar::Adder<size_t> g_total_segment_num("doris_total_segment_num");


BE metrics have a counter all_segments_num, value is it same as bvar::doris_total_segment_num?

xinyiZzz

LGTM

xinyiZzz · 2024-05-31T07:42:07Z

run buildall

doris-robot · 2024-05-31T08:17:19Z

TeamCity be ut coverage result:
Function Coverage: 36.29% (9234/25447)
Line Coverage: 27.64% (75735/274001)
Region Coverage: 26.85% (39207/145998)
Branch Coverage: 23.60% (19894/84286)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b96981ee9e8306b2af79f0b4568feb089a8d60aa_b96981ee9e8306b2af79f0b4568feb089a8d60aa/report/index.html

doris-robot · 2024-05-31T08:30:14Z

TPC-H: Total hot run time: 40730 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b96981ee9e8306b2af79f0b4568feb089a8d60aa, data reload: false
------ Round 1 ----------------------------------
q1	7217	4324	4270	4270
q2	1315	198	191	191
q3	7181	1097	1182	1097
q4	1073	766	809	766
q5	2671	2688	2874	2688
q6	275	172	169	169
q7	1061	652	626	626
q8	2052	2085	2074	2074
q9	6720	6565	6497	6497
q10	4215	3731	3732	3731
q11	442	246	249	246
q12	386	235	221	221
q13	18560	2969	2987	2969
q14	260	220	227	220
q15	519	482	490	482
q16	481	395	402	395
q17	961	706	723	706
q18	8036	7525	7401	7401
q19	1598	1514	1426	1426
q20	649	300	307	300
q21	5361	3936	3933	3933
q22	388	324	322	322
Total cold run time: 71421 ms
Total hot run time: 40730 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4305	4319	4268	4268
q2	376	266	280	266
q3	3035	2758	2746	2746
q4	1872	1657	1570	1570
q5	5263	5331	5299	5299
q6	216	127	132	127
q7	2148	1752	1779	1752
q8	3204	3307	3329	3307
q9	8425	8322	8409	8322
q10	3872	3690	3664	3664
q11	598	511	489	489
q12	804	625	595	595
q13	13474	2997	3016	2997
q14	283	263	266	263
q15	517	488	471	471
q16	478	404	421	404
q17	1783	1516	1457	1457
q18	7675	7583	7286	7286
q19	1696	1526	1574	1526
q20	1972	1809	1770	1770
q21	4917	4691	4853	4691
q22	623	547	536	536
Total cold run time: 67536 ms
Total hot run time: 53806 ms

doris-robot · 2024-05-31T08:41:07Z

TPC-DS: Total hot run time: 169670 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b96981ee9e8306b2af79f0b4568feb089a8d60aa, data reload: false
query1	932	394	370	370
query2	6460	2519	2438	2438
query3	6657	201	202	201
query4	19098	17498	17395	17395
query5	4182	464	457	457
query6	252	154	156	154
query7	4587	305	294	294
query8	329	285	297	285
query9	8575	2382	2367	2367
query10	471	307	286	286
query11	10675	10094	10081	10081
query12	143	94	87	87
query13	1659	371	376	371
query14	10265	7734	6220	6220
query15	223	187	188	187
query16	7857	266	257	257
query17	1860	542	528	528
query18	1942	272	269	269
query19	201	166	157	157
query20	96	95	86	86
query21	207	139	139	139
query22	4328	4055	3980	3980
query23	33807	33036	33140	33036
query24	11116	2845	2810	2810
query25	619	359	363	359
query26	1451	157	160	157
query27	3012	323	324	323
query28	7410	2069	2085	2069
query29	954	620	613	613
query30	286	155	150	150
query31	931	790	730	730
query32	99	52	56	52
query33	772	288	279	279
query34	976	476	488	476
query35	736	624	620	620
query36	1082	927	934	927
query37	155	69	69	69
query38	2925	2795	2755	2755
query39	843	787	807	787
query40	224	129	126	126
query41	56	52	52	52
query42	122	101	98	98
query43	568	561	547	547
query44	1202	738	743	738
query45	206	172	176	172
query46	1074	715	726	715
query47	1837	1733	1760	1733
query48	366	298	302	298
query49	1094	446	419	419
query50	772	389	382	382
query51	6754	6567	6579	6567
query52	97	96	92	92
query53	372	299	293	293
query54	863	460	457	457
query55	72	75	76	75
query56	279	269	291	269
query57	1127	1017	1061	1017
query58	251	257	264	257
query59	3392	3223	3212	3212
query60	298	276	273	273
query61	94	117	90	90
query62	658	455	461	455
query63	320	292	292	292
query64	9004	2217	1739	1739
query65	3183	3125	3137	3125
query66	1271	357	369	357
query67	15337	14710	14800	14710
query68	4606	573	584	573
query69	481	313	316	313
query70	1162	1128	1146	1128
query71	431	299	286	286
query72	7264	2863	2704	2704
query73	745	328	326	326
query74	6093	5710	5623	5623
query75	3446	2680	2635	2635
query76	2868	963	857	857
query77	492	308	312	308
query78	10352	9870	9748	9748
query79	2278	518	532	518
query80	1715	495	486	486
query81	578	223	224	223
query82	788	109	104	104
query83	333	185	177	177
query84	276	97	88	88
query85	1946	337	322	322
query86	483	331	323	323
query87	3286	3103	3163	3103
query88	4146	2360	2349	2349
query89	494	401	401	401
query90	1798	195	197	195
query91	139	133	113	113
query92	63	53	53	53
query93	2235	534	513	513
query94	1265	196	194	194
query95	420	325	327	325
query96	587	265	268	265
query97	3173	2971	3023	2971
query98	257	227	213	213
query99	1318	852	853	852
Total cold run time: 274584 ms
Total hot run time: 169670 ms

doris-robot · 2024-05-31T08:46:39Z

ClickBench: Total hot run time: 30.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b96981ee9e8306b2af79f0b4568feb089a8d60aa, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.06	0.06
query4	1.65	0.10	0.11
query5	0.51	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.49	0.51
query10	0.55	0.55	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.60	0.58	0.59
query14	0.78	0.80	0.78
query15	0.83	0.80	0.82
query16	0.34	0.35	0.36
query17	1.03	1.03	1.03
query18	0.22	0.22	0.27
query19	1.81	1.71	1.65
query20	0.02	0.01	0.01
query21	15.59	0.69	0.66
query22	4.38	7.14	1.77
query23	18.25	1.30	1.23
query24	1.49	0.37	0.21
query25	0.15	0.08	0.08
query26	0.26	0.16	0.17
query27	0.09	0.08	0.09
query28	13.34	1.00	1.00
query29	13.39	3.26	3.26
query30	0.25	0.05	0.06
query31	2.86	0.38	0.38
query32	3.31	0.48	0.46
query33	2.87	2.92	2.92
query34	17.24	4.42	4.46
query35	4.54	4.53	4.58
query36	0.71	0.50	0.46
query37	0.18	0.15	0.15
query38	0.16	0.14	0.15
query39	0.05	0.03	0.04
query40	0.16	0.15	0.13
query41	0.09	0.04	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.18 s
Total hot run time: 30.41 s

The memory consumed in segment cache is 0 after https://github.com/apache/doris/pull/35432/files. The pr also tracks memory usage of column readers.

…5751) The memory consumed in segment cache is 0 after https://github.com/apache/doris/pull/35432/files. The pr also tracks memory usage of column readers.

The memory consumed in segment cache is 0 after https://github.com/apache/doris/pull/35432/files. The pr also tracks memory usage of column readers.

[fix](segment cache) estimate momory consumed by segment
dddd55f
The memory consumed in segment cache is 0 after https://github.com/apache/doris/pull/35432/files. The pr also tracks memory usage of column readers.

fix
96c7fd0

github-actionsBot reviewed May 30, 2024
View reviewed changes

dataroaring added 3 commits May 30, 2024 15:52

fix
2bb8f8d

fix
4b43b79

fix
218bc63

fix
c8c7f1c

dataroaring added 2 commits May 30, 2024 17:12

fix
9c2c6a5

fix
ba424d2

github-actionsBot reviewed May 30, 2024
View reviewed changes

Comment threadbe/src/olap/base_tablet.cpp

add column reader count
6c88e59

yiguolei approved these changes May 31, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 31, 2024

github-actionsBot added the reviewed label May 31, 2024

Lchangliang approved these changes May 31, 2024
View reviewed changes

xinyiZzz reviewed May 31, 2024
View reviewed changes

xinyiZzz approved these changes May 31, 2024
View reviewed changes

Merge branch 'master' into segment_cache_bvar
b96981e

gavinchou approved these changes May 31, 2024
View reviewed changes

dataroaring merged commit 99388a0 into apache:masterMay 31, 2024

dataroaring added a commit that referenced this pull request May 31, 2024
[fix](segment cache) estimate momory consumed by segment (#35647)
981d2d3
The memory consumed in segment cache is 0 after https://github.com/apache/doris/pull/35432/files. The pr also tracks memory usage of column readers.

yiguolei added the 2.1.0-conflict label May 31, 2024

yiguolei added dev/2.1.4-merged and removed 2.1.0-conflict labels Jun 1, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

		@@ -146,5 +154,12 @@ OrdinalPageIndexIterator OrdinalIndexReader::seek_at_or_before(ordinal_t ordinal
		return OrdinalPageIndexIterator(this, left);
		}


		namespace doris {

		static bvar::Adder<size_t> g_total_rowset_num("doris_total_rowset_num");

Uh oh!

Conversation

dataroaring commented May 30, 2024

Proposed changes

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotMay 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotMay 30, 2024

Choose a reason for hiding this comment

Uh oh!

dataroaring commented May 30, 2024

Uh oh!

dataroaring commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

dataroaring commented May 30, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

doris-robot commented May 30, 2024

Uh oh!

dataroaring commented May 31, 2024

Uh oh!

github-actionsBot commented May 31, 2024

Uh oh!

github-actionsBot commented May 31, 2024

Uh oh!

dataroaring commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

Lchangliang commented May 31, 2024

Uh oh!

xinyiZzzMay 31, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinyiZzzMay 31, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

xinyiZzz commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

doris-robot commented May 31, 2024

Uh oh!

Reviewers

xinyiZzzMay 31, 2024•
edited
Loading

xinyiZzzMay 31, 2024•
edited
Loading