Skip to content

[fix](datatype)Fix for unaligned memory in arrow MapArray parsing. - #58248

Merged
morningman merged 1 commit into
apache:masterfrom
Chen768959:fix/arrow_map_mem_alignment
Nov 22, 2025
Merged

[fix](datatype)Fix for unaligned memory in arrow MapArray parsing.#58248
morningman merged 1 commit into
apache:masterfrom
Chen768959:fix/arrow_map_mem_alignment

Conversation

@Chen768959

@Chen768959Chen768959 commented Nov 21, 2025•
edited by morningman
Loading

Copy link
Copy Markdown
Contributor
  1. Fix for unaligned memory in arrow MapArray parsing.
  2. arrow Map unaligned memory ut.
  3. arrow Struct unaligned memory ut.
    Followup [Feature] Support Doris Catalog #55274

@hello-stephen

hello-stephen commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Chen768959
Chen768959force-pushed the fix/arrow_map_mem_alignment branch from 8d5c562 to ea40ab7CompareNovember 21, 2025 14:36
@yiguolei

yiguolei commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

run buildall

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 21, 2025
@github-actions

github-actionsBot commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Nov 21, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34360 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea40ab787b3d93695e5563a92c3ea2b4825a2c46, data reload: false
------ Round 1 ----------------------------------
q1	17635	5039	4915	4915
q2	2049	314	198	198
q3	10255	1272	694	694
q4	10758	930	366	366
q5	7521	2456	2408	2408
q6	183	171	141	141
q7	918	781	645	645
q8	9453	1348	1164	1164
q9	7307	5535	5413	5413
q10	6907	2239	1819	1819
q11	579	303	282	282
q12	337	360	220	220
q13	18099	3627	2990	2990
q14	228	241	216	216
q15	576	506	494	494
q16	1019	994	930	930
q17	567	868	361	361
q18	7452	7212	7104	7104
q19	1092	958	561	561
q20	339	343	224	224
q21	3763	2508	2257	2257
q22	1054	1069	958	958
Total cold run time: 108091 ms
Total hot run time: 34360 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4976	4948	4938	4938
q2	321	370	318	318
q3	2158	2614	2241	2241
q4	1326	1759	1277	1277
q5	4207	4481	4464	4464
q6	239	174	133	133
q7	2048	1994	1772	1772
q8	2662	2561	2455	2455
q9	7525	7639	7555	7555
q10	3020	3281	2772	2772
q11	599	522	513	513
q12	680	762	648	648
q13	3528	3939	3250	3250
q14	279	305	296	296
q15	536	503	519	503
q16	1051	1088	1038	1038
q17	1141	1573	1377	1377
q18	7889	7665	7763	7665
q19	795	814	980	814
q20	1921	1995	1850	1850
q21	4630	4408	4274	4274
q22	1095	1013	991	991
Total cold run time: 52626 ms
Total hot run time: 51144 ms

@doris-robot

doris-robot commented Nov 21, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 186458 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea40ab787b3d93695e5563a92c3ea2b4825a2c46, data reload: false
query1	1019	415	388	388
query2	6571	1674	1691	1674
query3	6762	222	216	216
query4	25859	23504	22649	22649
query5	4481	616	471	471
query6	352	242	222	222
query7	4663	502	303	303
query8	314	274	281	274
query9	8705	2566	2577	2566
query10	501	348	297	297
query11	15436	15049	14777	14777
query12	172	119	115	115
query13	1678	565	482	482
query14	10294	9074	9116	9074
query15	201	194	167	167
query16	7321	692	497	497
query17	1241	740	634	634
query18	1991	405	309	309
query19	218	207	191	191
query20	131	128	125	125
query21	220	136	114	114
query22	4055	4086	3883	3883
query23	33755	32971	33139	32971
query24	8406	2389	2387	2387
query25	611	532	448	448
query26	1234	272	164	164
query27	2743	486	365	365
query28	4405	2181	2168	2168
query29	827	617	470	470
query30	296	228	193	193
query31	909	799	719	719
query32	82	75	74	74
query33	601	377	326	326
query34	778	850	517	517
query35	803	829	734	734
query36	950	972	907	907
query37	123	105	88	88
query38	3533	3508	3485	3485
query39	1470	1431	1407	1407
query40	218	129	119	119
query41	63	60	66	60
query42	125	113	109	109
query43	487	477	477	477
query44	1224	754	759	754
query45	187	183	177	177
query46	886	979	634	634
query47	1755	1799	1716	1716
query48	383	411	315	315
query49	770	466	413	413
query50	648	679	406	406
query51	3880	4005	3884	3884
query52	113	112	105	105
query53	232	267	197	197
query54	328	293	275	275
query55	88	83	82	82
query56	351	327	326	326
query57	1171	1201	1118	1118
query58	284	272	271	271
query59	2470	2574	2476	2476
query60	358	352	331	331
query61	168	162	158	158
query62	785	733	647	647
query63	230	191	191	191
query64	4498	1164	866	866
query65	4066	3917	3924	3917
query66	1190	427	331	331
query67	14873	14731	14774	14731
query68	3414	958	637	637
query69	492	328	286	286
query70	1359	1312	1264	1264
query71	412	337	328	328
query72	6118	5153	5244	5153
query73	681	616	365	365
query74	8863	9053	8765	8765
query75	3314	3340	2823	2823
query76	3289	1149	740	740
query77	511	410	314	314
query78	9562	9779	8856	8856
query79	2140	843	597	597
query80	1681	558	501	501
query81	570	257	238	238
query82	403	158	132	132
query83	358	261	243	243
query84	254	104	104	104
query85	935	504	453	453
query86	430	330	311	311
query87	3729	3698	3550	3550
query88	2782	2246	2244	2244
query89	388	323	299	299
query90	1660	227	209	209
query91	167	163	133	133
query92	73	69	63	63
query93	1219	1024	685	685
query94	719	453	344	344
query95	410	313	326	313
query96	471	575	282	282
query97	2908	2945	2851	2851
query98	238	216	207	207
query99	1282	1381	1231	1231
Total cold run time: 266374 ms
Total hot run time: 186458 ms

@doris-robot

doris-robot commented Nov 21, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ea40ab787b3d93695e5563a92c3ea2b4825a2c46, data reload: false
query1	0.06	0.05	0.04
query2	0.09	0.05	0.05
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.29	0.24	0.26
query6	1.19	0.65	0.66
query7	0.03	0.02	0.02
query8	0.06	0.04	0.05
query9	0.58	0.53	0.52
query10	0.57	0.56	0.57
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.62	0.62
query14	1.01	1.02	1.00
query15	0.88	0.84	0.84
query16	0.41	0.42	0.41
query17	1.02	1.09	1.02
query18	0.21	0.21	0.20
query19	1.92	1.80	1.82
query20	0.02	0.01	0.01
query21	15.43	0.18	0.13
query22	5.14	0.08	0.05
query23	15.63	0.28	0.10
query24	2.67	0.83	1.16
query25	0.07	0.06	0.06
query26	0.15	0.14	0.13
query27	0.07	0.05	0.06
query28	5.50	1.23	0.95
query29	12.56	3.95	3.24
query30	0.27	0.13	0.12
query31	2.82	0.61	0.40
query32	3.23	0.57	0.49
query33	3.08	3.09	3.12
query34	15.91	5.30	4.51
query35	4.57	4.60	4.65
query36	0.69	0.51	0.50
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.03	0.04	0.03
query43	0.04	0.03	0.04
Total cold run time: 99.45 s
Total hot run time: 28.06 s

@doris-robot

doris-robot commented Nov 21, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage52.63% (18281/34733)
Line Coverage38.08% (166355/436860)
Region Coverage33.02% (129299/391626)
Branch Coverage33.83% (55510/164107)

@hello-stephen

hello-stephen commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (12/12) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage71.59% (24409/34094)
Line Coverage58.09% (253854/437026)
Region Coverage53.51% (212401/396934)
Branch Coverage54.82% (90512/165118)

@morningman
morningman merged commit fb8c540 into apache:masterNov 22, 2025
28 of 30 checks passed
morningman pushed a commit to morningman/doris that referenced this pull request Nov 22, 2025
@Chen768959@morningman
[fix](datatype)Fix for unaligned memory in arrow MapArray parsing. (a…
d149e20
…pache#58248)
1. Fix for unaligned memory in arrow MapArray parsing.
2. arrow Map unaligned memory ut.
3. arrow Struct unaligned memory ut.
Followup apache#55274
github-actionsBot pushed a commit that referenced this pull request Nov 22, 2025
@Chen768959Your Name
[fix](datatype)Fix for unaligned memory in arrow MapArray parsing. (#…
06da7c1
…58248)
1. Fix for unaligned memory in arrow MapArray parsing.
2. arrow Map unaligned memory ut.
3. arrow Struct unaligned memory ut.
Followup #55274
@github-actionsgithub-actionsBot mentioned this pull request Nov 22, 2025
Merged
@morningmanmorningman mentioned this pull request Nov 22, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 22, 2025
@github-actions@Chen768959
branch-4.0: [fix](datatype)Fix for unaligned memory in arrow MapArray…
6067568
… parsing. #58248 (#58251)
Cherry-picked from #58248
Co-authored-by: Chen768959 <67011523+Chen768959@users.noreply.github.com>
yiguolei pushed a commit that referenced this pull request Nov 24, 2025
@morningman@Chen768959
branch-4.0: [Feature] Support Doris Catalog (#55274) (#58162)
7989173
bp #55274#58002#58248
---------
Co-authored-by: Chen768959 <67011523+Chen768959@users.noreply.github.com>
luwei16 pushed a commit to luwei16/Doris that referenced this pull request Dec 8, 2025
@Chen768959@luwei16
[fix](datatype)Fix for unaligned memory in arrow MapArray parsing. (a…
624bfed
…pache#58248)
1. Fix for unaligned memory in arrow MapArray parsing.
2. arrow Map unaligned memory ut.
3. arrow Struct unaligned memory ut.
Followup apache#55274
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@Chen768959@nagisa-kunhah
[fix](datatype)Fix for unaligned memory in arrow MapArray parsing. (a…
39aa094
…pache#58248)
1. Fix for unaligned memory in arrow MapArray parsing.
2. arrow Map unaligned memory ut.
3. arrow Struct unaligned memory ut.
Followup apache#55274
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/4.0.2-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Chen768959@hello-stephen@yiguolei@doris-robot@morningman