[fix](memory) Fix BlockCompression memory tracking by xinyiZzz · Pull Request #33841 · apache/doris

xinyiZzz · 2024-04-18T09:01:43Z

Proposed changes

BlockCompression memory may be shared between multiple queries.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-18T09:01:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-04-18T09:01:49Z

run buildall

github-actions · 2024-04-18T09:07:07Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-18T10:13:21Z

TeamCity be ut coverage result:
Function Coverage: 35.42% (8911/25158)
Line Coverage: 27.12% (73225/269974)
Region Coverage: 26.25% (37827/144101)
Branch Coverage: 23.06% (19279/83604)
Coverage Report: http://coverage.selectdb-in.cc/coverage/37e4997ad8784881f0666808afbb945d5404d972_37e4997ad8784881f0666808afbb945d5404d972/report/index.html

yiguolei

LGTM

github-actions · 2024-04-18T10:48:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-18T10:48:50Z

PR approved by anyone and no changes requested.

xinyiZzz · 2024-04-18T11:18:11Z

run buildall

github-actions · 2024-04-18T11:23:38Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-18T11:36:31Z

TeamCity be ut coverage result:
Function Coverage: 35.42% (8912/25158)
Line Coverage: 27.13% (73261/270010)
Region Coverage: 26.26% (37841/144119)
Branch Coverage: 23.06% (19276/83604)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fc90fddc481fd7b7cb4cfed2706fa503459bb5ff_fc90fddc481fd7b7cb4cfed2706fa503459bb5ff/report/index.html

xinyiZzz · 2024-04-18T14:55:49Z

run buildall

github-actions · 2024-04-18T15:01:22Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-18T16:00:08Z

TeamCity be ut coverage result:
Function Coverage: 35.43% (8913/25158)
Line Coverage: 27.14% (73272/270013)
Region Coverage: 26.27% (37856/144119)
Branch Coverage: 23.06% (19282/83604)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ec3934c9f4a7b53d7e0099128e53bd060c8cf96c_ec3934c9f4a7b53d7e0099128e53bd060c8cf96c/report/index.html

github-actions · 2024-04-19T03:15:40Z

PR approved by at least one committer and no changes requested.

wangbo

LGTM

xinyiZzz · 2024-04-19T10:56:44Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-19T11:21:34Z

 return &s_instance;
 }
- ~Lz4BlockCompression() { _ctx_pool.clear(); }
+ ~Lz4BlockCompression() {


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
Suggested change
~Lz4BlockCompression() {
~Lz4BlockCompression() override{

github-actions · 2024-04-19T11:21:34Z

 return &s_instance;
 }
- ~Lz4HCBlockCompression() { _ctx_pool.clear(); }
+ ~Lz4HCBlockCompression() {


warning: annotate this function with 'override' or (rarely) 'final' [modernize-use-override]
Suggested change
~Lz4HCBlockCompression() {
~Lz4HCBlockCompression() override{

doris-robot · 2024-04-19T11:44:24Z

TPC-H: Total hot run time: 38963 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c68d65cab14e300f4037950ea10bbb7bddd437a, data reload: false
------ Round 1 ----------------------------------
q1	18009	4488	4244	4244
q2	2012	195	188	188
q3	10788	1167	1181	1167
q4	10620	827	819	819
q5	7530	2779	2700	2700
q6	225	138	137	137
q7	974	547	543	543
q8	9613	2124	2091	2091
q9	7431	6793	6707	6707
q10	8919	3631	3531	3531
q11	441	225	226	225
q12	387	218	214	214
q13	17950	2994	2941	2941
q14	279	236	241	236
q15	522	485	476	476
q16	528	387	381	381
q17	984	700	710	700
q18	7467	6847	6665	6665
q19	4764	1577	1543	1543
q20	672	318	323	318
q21	3569	2836	2986	2836
q22	364	301	307	301
Total cold run time: 114048 ms
Total hot run time: 38963 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4318	4241	4202	4202
q2	381	276	267	267
q3	2988	2737	2790	2737
q4	1880	1580	1581	1580
q5	5351	5345	5332	5332
q6	210	120	121	120
q7	1778	1425	1414	1414
q8	3219	3387	3399	3387
q9	8640	8625	8595	8595
q10	3890	3772	3681	3681
q11	596	480	477	477
q12	741	573	618	573
q13	16528	2959	2932	2932
q14	303	282	271	271
q15	510	461	475	461
q16	469	419	419	419
q17	1794	1528	1456	1456
q18	7632	7553	7482	7482
q19	1660	1575	1568	1568
q20	1957	1757	1740	1740
q21	4968	4878	4851	4851
q22	566	441	448	441
Total cold run time: 70379 ms
Total hot run time: 53986 ms

doris-robot · 2024-04-19T11:52:23Z

TeamCity be ut coverage result:
Function Coverage: 35.41% (8921/25190)
Line Coverage: 27.13% (73324/270255)
Region Coverage: 26.27% (37881/144215)
Branch Coverage: 23.07% (19284/83606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1c68d65cab14e300f4037950ea10bbb7bddd437a_1c68d65cab14e300f4037950ea10bbb7bddd437a/report/index.html

doris-robot · 2024-04-19T11:56:02Z

TPC-DS: Total hot run time: 183732 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c68d65cab14e300f4037950ea10bbb7bddd437a, data reload: false
query1	905	371	363	363
query2	6462	2502	2290	2290
query3	6655	210	211	210
query4	22959	21252	21398	21252
query5	4163	405	411	405
query6	272	172	170	170
query7	4587	284	285	284
query8	238	188	182	182
query9	8418	2320	2297	2297
query10	570	243	238	238
query11	14774	14168	14158	14158
query12	147	93	90	90
query13	1648	373	375	373
query14	8944	8013	7798	7798
query15	268	191	188	188
query16	8184	263	262	262
query17	1922	596	552	552
query18	2120	283	276	276
query19	348	153	160	153
query20	91	91	85	85
query21	240	124	123	123
query22	5011	4800	4773	4773
query23	33767	33134	32923	32923
query24	11722	2977	2888	2888
query25	653	375	366	366
query26	1744	150	148	148
query27	2934	298	313	298
query28	7595	2007	2015	2007
query29	1008	599	573	573
query30	269	174	166	166
query31	936	718	733	718
query32	92	54	58	54
query33	735	243	233	233
query34	1047	460	485	460
query35	826	693	689	689
query36	1020	892	906	892
query37	280	70	75	70
query38	3319	3173	3192	3173
query39	1554	1523	1524	1523
query40	276	120	124	120
query41	43	43	42	42
query42	104	95	98	95
query43	574	524	539	524
query44	1169	713	715	713
query45	276	260	237	237
query46	1069	716	729	716
query47	1939	1836	1826	1826
query48	374	295	292	292
query49	1173	388	375	375
query50	752	372	377	372
query51	6913	6723	6694	6694
query52	103	88	86	86
query53	348	275	278	275
query54	296	229	220	220
query55	78	76	71	71
query56	269	221	215	215
query57	1200	1122	1135	1122
query58	213	193	189	189
query59	3454	3113	3160	3113
query60	263	234	229	229
query61	89	90	85	85
query62	633	451	426	426
query63	298	273	273	273
query64	6246	3612	3893	3612
query65	3067	3026	3015	3015
query66	1387	346	332	332
query67	15391	15445	15056	15056
query68	6650	542	547	542
query69	535	309	301	301
query70	1274	1135	1198	1135
query71	1436	1263	1262	1262
query72	6694	2587	2440	2440
query73	718	315	319	315
query74	6725	6276	6345	6276
query75	4014	2631	2609	2609
query76	4247	935	949	935
query77	670	256	260	256
query78	11046	10122	10328	10122
query79	4497	518	520	518
query80	1191	420	420	420
query81	517	250	241	241
query82	1330	91	102	91
query83	208	170	166	166
query84	264	86	86	86
query85	1549	260	258	258
query86	501	298	304	298
query87	3446	3278	3244	3244
query88	4758	2296	2297	2296
query89	483	375	360	360
query90	1978	177	179	177
query91	128	94	95	94
query92	59	46	45	45
query93	5192	527	506	506
query94	1187	178	182	178
query95	385	304	310	304
query96	602	260	261	260
query97	3132	2916	2950	2916
query98	241	225	212	212
query99	1264	841	866	841
Total cold run time: 293345 ms
Total hot run time: 183732 ms

doris-robot · 2024-04-19T12:01:32Z

ClickBench: Total hot run time: 29.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c68d65cab14e300f4037950ea10bbb7bddd437a, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.68	0.08	0.09
query5	0.49	0.49	0.50
query6	1.47	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.51
query10	0.56	0.55	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.58	0.59
query14	0.77	0.76	0.79
query15	0.82	0.81	0.80
query16	0.35	0.36	0.36
query17	0.96	1.01	0.97
query18	0.21	0.24	0.24
query19	1.80	1.68	1.71
query20	0.02	0.01	0.01
query21	15.42	0.67	0.65
query22	4.94	7.29	1.34
query23	18.32	1.38	1.26
query24	1.56	0.28	0.25
query25	0.14	0.09	0.08
query26	0.25	0.17	0.17
query27	0.08	0.07	0.08
query28	13.38	0.99	0.97
query29	12.58	3.28	3.27
query30	0.25	0.06	0.06
query31	2.86	0.38	0.39
query32	3.28	0.46	0.47
query33	2.81	2.89	2.80
query34	17.25	4.46	4.49
query35	4.48	4.59	4.53
query36	0.64	0.45	0.46
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.99 s
Total hot run time: 29.91 s

doris-robot · 2024-04-19T12:04:40Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1c68d65cab14e300f4037950ea10bbb7bddd437a with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 13.3 seconds inserted 10000000 Rows, about 751K ops/s

github-actions · 2024-04-19T16:43:15Z

PR approved by at least one committer and no changes requested.

pick #33396 #33862 #33853 #33732 #33841 #33933 #34901 #35014 --------- Co-authored-by: yiguolei <676222867@qq.com> Co-authored-by: yiguolei <yiguolei@gmail.com>

yiguolei previously approved these changes Apr 18, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 18, 2024

github-actionsBot added the reviewed label Apr 18, 2024

xinyiZzz dismissed yiguolei’s stale review via fc90fddApril 18, 2024 11:17

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Apr 18, 2024

jacktengg previously approved these changes Apr 19, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 19, 2024

wangbo previously approved these changes Apr 19, 2024
View reviewed changes

1
1c68d65

xinyiZzz dismissed stale reviews from wangbo and jacktengg via 1c68d65April 19, 2024 10:56

xinyiZzz force-pushed the 20240417_fix_tracker_compression branch from ec3934c to 1c68d65Compare April 19, 2024 10:56

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Apr 19, 2024

github-actionsBot reviewed Apr 19, 2024
View reviewed changes

yiguolei approved these changes Apr 19, 2024
View reviewed changes

Gabriel39 approved these changes Apr 19, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 19, 2024

yiguolei merged commit 495f389 into apache:masterApr 19, 2024

yiguolei added the 2.1.0-conflict label Apr 20, 2024

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
[fix](memory) Fix BlockCompression memory trackingapache#33841
7d1fdea

xinyiZzz mentioned this pull request Sep 9, 2024
[branch-2.1](memory) Pick query memory tracker value is equal to 0 and exception handle #40564
Closed

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 9, 2024
[fix](memory) Fix BlockCompression memory trackingapache#33841
6aeb2b6

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
[fix](memory) Fix BlockCompression memory trackingapache#33841
a92a400

yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 10, 2024
[fix](memory) Fix BlockCompression memory trackingapache#33841
04b57f6

yiguolei pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 11, 2024
[fix](memory) Fix BlockCompression memory trackingapache#33841
772b1c3

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 13, 2024
[fix](memory) Fix BlockCompression memory trackingapache#33841
8f38331

xinyiZzz mentioned this pull request Sep 13, 2024
[branch-2.1](memory) Allocator support address sanitizers #40836
Merged

xinyiZzz added dev/2.1.7-merged and removed 2.1.0-conflict labels Sep 20, 2024

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[fix](memory) Fix BlockCompression memory trackingapache#33841
072d07c

	~Lz4HCBlockCompression() {
	~Lz4HCBlockCompression() override{

Uh oh!

Conversation

xinyiZzz commented Apr 18, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 18, 2024

Uh oh!

xinyiZzz commented Apr 18, 2024

Uh oh!

github-actionsBot commented Apr 18, 2024

Uh oh!

doris-robot commented Apr 18, 2024

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Apr 18, 2024

Uh oh!

github-actionsBot commented Apr 18, 2024

Uh oh!

xinyiZzz commented Apr 18, 2024

Uh oh!

github-actionsBot commented Apr 18, 2024

Uh oh!

doris-robot commented Apr 18, 2024

Uh oh!

xinyiZzz commented Apr 18, 2024

Uh oh!

github-actionsBot commented Apr 18, 2024

Uh oh!

doris-robot commented Apr 18, 2024

Uh oh!

github-actionsBot commented Apr 19, 2024

Uh oh!

wangbo left a comment

Choose a reason for hiding this comment

Uh oh!

xinyiZzz commented Apr 19, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 19, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBotApr 19, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

doris-robot commented Apr 19, 2024

Uh oh!

github-actionsBot commented Apr 19, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants