Skip to content

[improvement](spill) avoid occuping too much memory while spill build block during the hash join build phase - #33747

Merged
yiguolei merged 1 commit into
apache:masterfrom
mrhhsg:opt_join_spill
Apr 17, 2024
Merged

[improvement](spill) avoid occuping too much memory while spill build block during the hash join build phase#33747
yiguolei merged 1 commit into
apache:masterfrom
mrhhsg:opt_join_spill

Conversation

@mrhhsg

@mrhhsgmrhhsg commented Apr 17, 2024

Copy link
Copy Markdown
Member

Proposed changes

  1. Use uint32_t as the type of partitioned index.
  2. Spill the build block in small blocks each time.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mrhhsg

mrhhsg commented Apr 17, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Apr 17, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp
@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.49% (8908/25099)
Line Coverage: 27.20% (73156/268922)
Region Coverage: 26.34% (37822/143608)
Branch Coverage: 23.12% (19273/83346)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c23fa4ceacd672983ae9c6facd403669502a7e83_c23fa4ceacd672983ae9c6facd403669502a7e83/report/index.html

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38942 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c23fa4ceacd672983ae9c6facd403669502a7e83, data reload: false
------ Round 1 ----------------------------------
q1	17913	4982	4327	4327
q2	2457	192	190	190
q3	11083	1266	1192	1192
q4	10371	839	887	839
q5	7552	2752	2679	2679
q6	223	134	132	132
q7	1029	612	607	607
q8	9704	2098	2083	2083
q9	7537	6605	6531	6531
q10	8573	3516	3522	3516
q11	449	226	222	222
q12	402	222	216	216
q13	17768	2932	2951	2932
q14	274	223	224	223
q15	536	492	481	481
q16	518	379	371	371
q17	963	755	767	755
q18	7318	6792	6783	6783
q19	5513	1525	1528	1525
q20	647	315	293	293
q21	3535	2742	2803	2742
q22	366	307	303	303
Total cold run time: 114731 ms
Total hot run time: 38942 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4363	4167	4189	4167
q2	369	280	269	269
q3	2980	2756	2732	2732
q4	1869	1577	1575	1575
q5	5358	5379	5326	5326
q6	208	126	126	126
q7	2211	1851	1835	1835
q8	3223	3363	3345	3345
q9	8593	8636	8539	8539
q10	3904	3675	3719	3675
q11	591	464	469	464
q12	740	611	620	611
q13	16392	2932	2927	2927
q14	306	268	273	268
q15	514	478	467	467
q16	462	402	419	402
q17	1758	1504	1467	1467
q18	7662	7513	7470	7470
q19	1619	1541	1554	1541
q20	1946	1776	1720	1720
q21	4868	4702	4787	4702
q22	534	442	478	442
Total cold run time: 70470 ms
Total hot run time: 54070 ms

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 183488 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c23fa4ceacd672983ae9c6facd403669502a7e83, data reload: false
query1	923	374	366	366
query2	6515	2438	2254	2254
query3	6657	207	212	207
query4	22748	21292	21246	21246
query5	4131	398	407	398
query6	272	173	169	169
query7	4573	290	288	288
query8	235	194	172	172
query9	8449	2344	2343	2343
query10	595	249	244	244
query11	14737	14293	14170	14170
query12	134	89	86	86
query13	1642	370	367	367
query14	10005	7858	7522	7522
query15	249	184	186	184
query16	8119	270	265	265
query17	1910	603	556	556
query18	2089	279	276	276
query19	206	150	156	150
query20	92	86	88	86
query21	199	127	126	126
query22	4968	4812	4768	4768
query23	33991	32865	33159	32865
query24	11751	2965	2942	2942
query25	649	362	369	362
query26	1744	151	143	143
query27	3021	308	309	308
query28	7607	2008	1980	1980
query29	1026	606	589	589
query30	321	172	168	168
query31	942	730	719	719
query32	96	55	51	51
query33	739	249	244	244
query34	1053	472	481	472
query35	851	715	687	687
query36	1057	920	895	895
query37	212	69	69	69
query38	3336	3176	3140	3140
query39	1585	1532	1519	1519
query40	269	129	120	120
query41	45	45	44	44
query42	108	93	96	93
query43	575	515	545	515
query44	1188	725	716	716
query45	281	257	227	227
query46	1085	736	727	727
query47	1901	1847	1838	1838
query48	357	297	291	291
query49	1169	398	358	358
query50	737	373	386	373
query51	6663	6582	6521	6521
query52	103	86	94	86
query53	343	272	278	272
query54	311	227	223	223
query55	74	72	69	69
query56	240	232	217	217
query57	1192	1121	1098	1098
query58	221	192	195	192
query59	3266	3051	3073	3051
query60	249	232	233	232
query61	91	86	89	86
query62	637	442	440	440
query63	300	278	276	276
query64	6294	3797	3708	3708
query65	3095	3029	3028	3028
query66	1420	336	332	332
query67	15299	15162	14898	14898
query68	5636	526	536	526
query69	532	296	296	296
query70	1268	1187	1161	1161
query71	1428	1316	1265	1265
query72	6603	2625	2413	2413
query73	735	320	319	319
query74	6773	6363	6349	6349
query75	3588	2678	2588	2588
query76	3713	961	994	961
query77	653	264	254	254
query78	10937	10326	10191	10191
query79	4652	533	526	526
query80	1134	424	423	423
query81	527	245	245	245
query82	1401	101	91	91
query83	210	170	168	168
query84	276	86	86	86
query85	1488	275	260	260
query86	468	285	306	285
query87	3453	3281	3291	3281
query88	4933	2445	2451	2445
query89	479	369	381	369
query90	1986	184	185	184
query91	139	106	110	106
query92	61	48	50	48
query93	5213	507	504	504
query94	1171	185	184	184
query95	394	306	301	301
query96	603	275	264	264
query97	3124	2925	2913	2913
query98	239	226	216	216
query99	1291	885	839	839
Total cold run time: 292050 ms
Total hot run time: 183488 ms

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c23fa4ceacd672983ae9c6facd403669502a7e83, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.68	0.07	0.07
query5	0.49	0.50	0.50
query6	1.45	0.71	0.71
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.54	0.50	0.50
query10	0.56	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.58	0.58
query14	0.76	0.77	0.78
query15	0.83	0.80	0.80
query16	0.37	0.36	0.37
query17	0.98	1.01	0.98
query18	0.22	0.24	0.23
query19	1.79	1.69	1.67
query20	0.02	0.01	0.01
query21	15.42	0.65	0.66
query22	3.99	7.73	1.90
query23	18.31	1.35	1.25
query24	1.81	0.29	0.20
query25	0.16	0.08	0.07
query26	0.26	0.17	0.17
query27	0.07	0.08	0.08
query28	13.31	1.00	0.98
query29	12.64	3.27	3.25
query30	0.26	0.07	0.05
query31	2.95	0.37	0.37
query32	3.22	0.47	0.46
query33	2.83	2.82	2.82
query34	17.06	4.39	4.40
query35	4.50	4.47	4.45
query36	0.65	0.45	0.46
query37	0.19	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.13
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.19 s
Total hot run time: 30.24 s

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c23fa4ceacd672983ae9c6facd403669502a7e83 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s

yiguolei
yiguolei approved these changes Apr 17, 2024
View reviewed changes

@yiguoleiyiguolei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024
@github-actions

github-actionsBot commented Apr 17, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Apr 17, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

jacktengg
jacktengg approved these changes Apr 17, 2024
View reviewed changes

@jacktenggjacktengg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
yiguolei merged commit 73c336a into apache:masterApr 17, 2024
dataroaring pushed a commit that referenced this pull request Apr 17, 2024
@mrhhsg@dataroaring
[improvement](spill) avoid occuping too much memory while spill build…
7621008
… block during the hash join build phase (#33747)
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@mrhhsg@Doris-Extras
[improvement](spill) avoid occuping too much memory while spill build…
bdf9bf0
… block during the hash join build phase (#33747)
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@mrhhsg@Doris-Extras
[improvement](spill) avoid occuping too much memory while spill build…
520f0f5
… block during the hash join build phase (#33747)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@mrhhsg@HappenLee
[improvement](spill) avoid occuping too much memory while spill build…
1054c65
… block during the hash join build phase (apache#33747)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@yiguoleiyiguoleiyiguolei approved these changes
@jacktenggjacktenggjacktengg approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@mrhhsg@doris-robot@yiguolei@jacktengg@dataroaring