[Refactor](type) use std::bit_cast to replace union-based type punning by zclllyybb · Pull Request #53135 · apache/doris

zclllyybb · 2025-07-11T16:33:37Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Run on (96 X 3100.01 MHz CPU s)
CPU Caches:
L1 Data 32 KiB (x48)
L1 Instruction 32 KiB (x48)
L2 Unified 1024 KiB (x48)
L3 Unified 36608 KiB (x2)
Load Average: 6.58, 7.45, 10.99
-------------------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
-------------------------------------------------------------------------------------------------------------------------------------
BM_BinaryCast_UI64_to_DateTimeV2/1000000/min_time:2.000/iterations:100 534352472 us 866 us 100 items_per_second=1.15495G/s
BM_BinaryCast_UI64_to_DateTimeV2/10000000/min_time:2.000/iterations:100 534360412 us 8806 us 100 items_per_second=1.1356G/s
BM_OldBinaryCast_UI64_to_DateTimeV2/1000000/min_time:2.000/iterations:100 534361880 us 10274 us 100 items_per_second=97.3361M/s
BM_OldBinaryCast_UI64_to_DateTimeV2/10000000/min_time:2.000/iterations:100 534369846 us 18239 us 100 items_per_second=548.273M/s
BM_BinaryCast_DateTimeV2_to_UI64/1000000/min_time:2.000/iterations:100 534370998 us 19392 us 100 items_per_second=51.568M/s
BM_BinaryCast_DateTimeV2_to_UI64/10000000/min_time:2.000/iterations:100 534379002 us 27395 us 100 items_per_second=365.027M/s
BM_OldBinaryCast_DateTimeV2_to_UI64/1000000/min_time:2.000/iterations:100 534380603 us 28996 us 100 items_per_second=34.4872M/s
BM_OldBinaryCast_DateTimeV2_to_UI64/10000000/min_time:2.000/iterations:100 534388536 us 36929 us 100 items_per_second=270.786M/s

the benchmark result upon actually means: these two ways both has no side-effect and could be optimized out.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-07-11T16:33:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zclllyybb · 2025-07-11T16:33:44Z

run buildall

doris-robot · 2025-07-11T18:21:26Z

TPC-H: Total hot run time: 33082 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e1e5e72e6bdd7bc87cbf3598677b6d911b0d2bd, data reload: false
------ Round 1 ----------------------------------
q1	17579	5142	5001	5001
q2	1940	276	180	180
q3	10330	1318	708	708
q4	10243	1016	523	523
q5	7606	2365	2409	2365
q6	173	159	131	131
q7	903	748	588	588
q8	9308	1304	1058	1058
q9	6860	5074	5088	5074
q10	6886	2382	1962	1962
q11	482	279	264	264
q12	346	350	210	210
q13	17783	3680	3081	3081
q14	227	214	211	211
q15	555	473	470	470
q16	427	422	367	367
q17	584	875	357	357
q18	7734	7236	7142	7142
q19	1339	941	554	554
q20	348	345	222	222
q21	3704	2590	2324	2324
q22	359	318	290	290
Total cold run time: 105716 ms
Total hot run time: 33082 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5062	5053	5010	5010
q2	236	328	220	220
q3	2218	2700	2307	2307
q4	1367	1786	1293	1293
q5	4232	4117	4456	4117
q6	207	167	121	121
q7	2076	2031	1789	1789
q8	2689	2566	2543	2543
q9	7291	7317	7230	7230
q10	3123	3251	2894	2894
q11	567	514	504	504
q12	709	791	658	658
q13	3505	4037	3547	3547
q14	278	288	270	270
q15	519	491	488	488
q16	439	503	456	456
q17	1172	1616	1392	1392
q18	7935	7803	7863	7803
q19	775	803	1017	803
q20	2027	2064	1870	1870
q21	4741	4352	4303	4303
q22	610	593	576	576
Total cold run time: 51778 ms
Total hot run time: 50194 ms

doris-robot · 2025-07-11T18:33:00Z

TPC-DS: Total hot run time: 186386 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e1e5e72e6bdd7bc87cbf3598677b6d911b0d2bd, data reload: false
query1	976	383	385	383
query2	6490	1691	1662	1662
query3	6742	214	216	214
query4	26183	23784	23262	23262
query5	4361	569	412	412
query6	295	219	198	198
query7	4619	477	283	283
query8	274	227	211	211
query9	8654	2641	2625	2625
query10	460	332	261	261
query11	15490	14978	14786	14786
query12	151	106	101	101
query13	1654	515	413	413
query14	8729	5971	6002	5971
query15	197	193	207	193
query16	7746	434	252	252
query17	1338	702	559	559
query18	2053	382	302	302
query19	183	187	153	153
query20	120	116	116	116
query21	210	127	109	109
query22	4092	4214	4080	4080
query23	33988	32849	32986	32849
query24	8374	2345	2330	2330
query25	518	464	390	390
query26	823	261	139	139
query27	2752	501	329	329
query28	4415	2132	2113	2113
query29	656	541	431	431
query30	285	225	191	191
query31	899	853	769	769
query32	72	65	58	58
query33	543	354	287	287
query34	784	827	501	501
query35	594	625	538	538
query36	942	984	891	891
query37	106	101	73	73
query38	4136	4087	4022	4022
query39	1510	1420	1406	1406
query40	209	117	104	104
query41	54	54	53	53
query42	124	109	108	108
query43	506	490	470	470
query44	1322	827	814	814
query45	177	169	167	167
query46	825	1010	628	628
query47	1750	1798	1742	1742
query48	376	413	302	302
query49	698	495	396	396
query50	622	687	432	432
query51	5509	5467	5461	5461
query52	112	109	98	98
query53	217	254	185	185
query54	560	563	488	488
query55	78	80	82	80
query56	289	300	301	300
query57	1180	1197	1117	1117
query58	257	247	251	247
query59	2605	2685	2605	2605
query60	333	333	318	318
query61	187	123	121	121
query62	802	691	644	644
query63	219	185	185	185
query64	3518	1200	820	820
query65	4259	4160	4192	4160
query66	884	409	310	310
query67	15757	15429	15368	15368
query68	7892	855	522	522
query69	505	297	270	270
query70	1202	1093	1058	1058
query71	460	298	296	296
query72	5608	4630	4654	4630
query73	666	582	351	351
query74	8819	8945	8791	8791
query75	3826	3151	2723	2723
query76	3525	1147	698	698
query77	791	378	296	296
query78	11034	11162	10282	10282
query79	1574	816	592	592
query80	567	503	422	422
query81	476	253	216	216
query82	183	124	94	94
query83	265	246	226	226
query84	238	98	82	82
query85	764	356	316	316
query86	340	307	268	268
query87	4341	4436	4267	4267
query88	3051	2268	2264	2264
query89	375	317	280	280
query90	2071	207	203	203
query91	134	139	111	111
query92	71	57	56	56
query93	1321	930	591	591
query94	639	309	198	198
query95	375	288	287	287
query96	490	553	277	277
query97	2664	2743	2670	2670
query98	230	209	194	194
query99	1320	1385	1269	1269
Total cold run time: 271660 ms
Total hot run time: 186386 ms

doris-robot · 2025-07-11T18:38:15Z

ClickBench: Total hot run time: 29.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7e1e5e72e6bdd7bc87cbf3598677b6d911b0d2bd, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.08	0.07
query4	1.62	0.10	0.10
query5	0.44	0.44	0.42
query6	1.18	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.60	0.52	0.51
query10	0.57	0.56	0.58
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.63	0.62	0.60
query14	0.82	0.78	0.82
query15	0.89	0.89	0.85
query16	0.38	0.38	0.38
query17	1.06	1.04	1.06
query18	0.23	0.21	0.21
query19	1.88	1.89	1.81
query20	0.01	0.01	0.01
query21	15.39	0.89	0.53
query22	0.76	1.24	0.74
query23	14.81	1.39	0.62
query24	6.62	1.95	0.57
query25	0.46	0.18	0.10
query26	0.62	0.16	0.13
query27	0.06	0.05	0.05
query28	9.71	0.93	0.43
query29	12.56	4.05	3.39
query30	0.26	0.09	0.07
query31	2.83	0.58	0.38
query32	3.23	0.56	0.49
query33	3.15	3.09	3.07
query34	16.12	5.37	4.73
query35	4.85	4.80	4.78
query36	0.71	0.51	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.15	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.68 s
Total hot run time: 29.27 s

doris-robot · 2025-07-11T19:52:24Z

BE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.45% (15614/27178)
Line Coverage	46.19% (139395/301771)
Region Coverage	35.52% (104197/293344)
Branch Coverage	38.06% (45954/120747)

zclllyybb · 2025-07-12T10:56:04Z

run buildall

doris-robot · 2025-07-12T12:44:48Z

TPC-H: Total hot run time: 33275 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed7777ca819de39e1f4f8ef5cf9fca51da7a104a, data reload: false
------ Round 1 ----------------------------------
q1	17616	5132	5036	5036
q2	1938	279	190	190
q3	10409	1371	737	737
q4	10259	1017	527	527
q5	7833	2499	2314	2314
q6	182	158	126	126
q7	892	741	596	596
q8	9308	1274	1119	1119
q9	7018	5181	5159	5159
q10	6897	2408	1948	1948
q11	478	296	278	278
q12	338	343	219	219
q13	17799	3726	3135	3135
q14	239	222	219	219
q15	551	478	475	475
q16	418	419	386	386
q17	583	936	352	352
q18	7789	7215	7083	7083
q19	1215	939	557	557
q20	347	342	224	224
q21	3792	2649	2304	2304
q22	361	318	291	291
Total cold run time: 106262 ms
Total hot run time: 33275 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5003	5048	5016	5016
q2	244	332	230	230
q3	2171	2718	2265	2265
q4	1381	1778	1335	1335
q5	4220	4308	4525	4308
q6	232	175	126	126
q7	1974	1967	1758	1758
q8	2710	2533	2552	2533
q9	7358	7349	7327	7327
q10	3162	3313	2822	2822
q11	604	528	490	490
q12	672	776	599	599
q13	3573	4025	3514	3514
q14	281	307	299	299
q15	533	491	453	453
q16	441	521	466	466
q17	1192	1596	1399	1399
q18	7892	7808	7390	7390
q19	825	855	1057	855
q20	2044	2050	1867	1867
q21	4774	4324	4245	4245
q22	624	614	534	534
Total cold run time: 51910 ms
Total hot run time: 49831 ms

doris-robot · 2025-07-12T12:56:24Z

TPC-DS: Total hot run time: 187413 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed7777ca819de39e1f4f8ef5cf9fca51da7a104a, data reload: false
query1	987	402	408	402
query2	6530	1673	1654	1654
query3	6752	215	210	210
query4	26964	23683	22977	22977
query5	4317	593	441	441
query6	315	218	204	204
query7	4630	505	292	292
query8	278	237	220	220
query9	8636	2643	2649	2643
query10	491	316	267	267
query11	15600	15044	14792	14792
query12	151	115	104	104
query13	1648	537	403	403
query14	8515	6094	6076	6076
query15	211	188	171	171
query16	7317	430	255	255
query17	1349	704	586	586
query18	1981	408	294	294
query19	215	195	170	170
query20	133	117	110	110
query21	206	120	101	101
query22	4189	4104	4175	4104
query23	34304	33132	33344	33132
query24	8491	2355	2354	2354
query25	538	475	397	397
query26	1231	265	145	145
query27	2750	501	348	348
query28	4315	2117	2088	2088
query29	766	557	430	430
query30	284	222	188	188
query31	914	843	736	736
query32	70	64	59	59
query33	555	332	273	273
query34	804	914	530	530
query35	595	630	572	572
query36	933	1003	885	885
query37	111	100	77	77
query38	4221	4187	4079	4079
query39	1475	1411	1440	1411
query40	209	117	108	108
query41	55	54	56	54
query42	118	111	108	108
query43	507	504	488	488
query44	1298	829	813	813
query45	174	171	177	171
query46	835	1020	627	627
query47	1783	1819	1744	1744
query48	369	410	308	308
query49	747	470	391	391
query50	629	690	422	422
query51	5582	5716	5602	5602
query52	112	106	106	106
query53	221	267	180	180
query54	573	561	500	500
query55	82	77	77	77
query56	278	276	297	276
query57	1200	1208	1153	1153
query58	268	256	259	256
query59	2509	2678	2474	2474
query60	332	320	295	295
query61	125	123	132	123
query62	786	702	673	673
query63	225	183	195	183
query64	4450	1150	813	813
query65	4262	4177	4204	4177
query66	1133	433	305	305
query67	15822	15686	15554	15554
query68	6154	910	528	528
query69	515	304	271	271
query70	1145	1137	1050	1050
query71	385	328	311	311
query72	5649	4724	4834	4724
query73	674	597	346	346
query74	9223	9149	9075	9075
query75	3154	3194	2689	2689
query76	3157	1147	714	714
query77	481	374	299	299
query78	10811	10979	10263	10263
query79	1597	817	575	575
query80	600	516	441	441
query81	475	316	220	220
query82	176	121	96	96
query83	251	251	248	248
query84	244	102	82	82
query85	756	357	323	323
query86	373	308	288	288
query87	4494	4459	4305	4305
query88	2833	2235	2257	2235
query89	376	315	297	297
query90	1803	213	203	203
query91	128	143	112	112
query92	63	57	56	56
query93	1052	940	590	590
query94	619	312	197	197
query95	376	288	282	282
query96	485	559	278	278
query97	2695	2763	2671	2671
query98	223	255	201	201
query99	1381	1422	1282	1282
Total cold run time: 270804 ms
Total hot run time: 187413 ms

doris-robot · 2025-07-12T13:01:42Z

ClickBench: Total hot run time: 29.27 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed7777ca819de39e1f4f8ef5cf9fca51da7a104a, data reload: false
query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.25	0.07	0.07
query4	1.62	0.12	0.11
query5	0.43	0.41	0.42
query6	1.14	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.61	0.51	0.52
query10	0.56	0.57	0.58
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.61	0.62
query14	0.80	0.80	0.82
query15	0.91	0.86	0.90
query16	0.39	0.38	0.40
query17	1.06	1.07	1.07
query18	0.22	0.21	0.21
query19	1.95	1.83	1.81
query20	0.02	0.02	0.02
query21	15.39	0.90	0.53
query22	0.75	1.30	0.68
query23	14.84	1.37	0.62
query24	6.83	2.01	0.40
query25	0.39	0.17	0.09
query26	0.69	0.17	0.15
query27	0.06	0.06	0.05
query28	9.57	0.92	0.44
query29	12.58	4.01	3.40
query30	0.27	0.11	0.06
query31	2.84	0.59	0.39
query32	3.24	0.55	0.47
query33	3.00	3.06	3.12
query34	16.09	5.39	4.80
query35	4.80	4.87	4.86
query36	0.70	0.51	0.50
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.15	0.14
query41	0.07	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.61 s
Total hot run time: 29.27 s

doris-robot · 2025-07-12T14:17:48Z

BE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.44% (15612/27182)
Line Coverage	46.15% (139366/301964)
Region Coverage	35.50% (104190/293526)
Branch Coverage	38.03% (45942/120798)

BiteTheDDDDt

LGTM

github-actions · 2025-07-14T03:45:58Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-14T03:46:00Z

PR approved by anyone and no changes requested.

…k / disabled test Follow-up to the previous two commits. Two latent breakages where code that doesn't ride the default build path still referenced classes deleted by this PR series: 1) be/benchmark/benchmark_zone_map_index.hpp `benchmark_test` (built only with `--benchmark`, default OFF) was wiring up its zone-map writer through the now-deleted `StorageFieldFactory::create`. Drop the wrapper and pass the `TabletColumn*` straight into `ZoneMapIndexWriter::create`, which is what the writer wants anyway after the previous commit's API change. Verified by RELEASE build of the `benchmark_test` target: our hpp compiles cleanly (the binary as a whole still fails on `binary_cast_benchmark.hpp`, but that is fallout from PR apache#53135 union-to-`std::bit_cast` migration and is out of scope here). 2) be/test/storage/segment/column_reader_writer_test.cpp This file is in the `# todo: need fix those ut` REMOVE_ITEM list of be/test/CMakeLists.txt and is not built. It was already broken on the deleted row-based `ColumnBlock` / `ColumnVectorBatch` API from a prior BE refactor; this PR series additionally killed `StorageField` and `CollectionValue` it referenced. Strip out the four pieces tied to the classes this PR deleted, leaving the rest of the file as-is for whoever eventually revives the legacy `ColumnBlock` paths: - the `test_array_nullable_data` helper (used CollectionValue + StorageFieldFactory) - TEST_F(test_array_type) (only caller of that helper) - TEST_F(test_single_empty_array) (constructed CollectionValue) - TEST_F(test_mixed_empty_arrays) (same) The file stays in REMOVE_ITEM so the build state is unchanged.

1
7e1e5e7

zclllyybb changed the title ~~[Refactor](type) use std::bit_cast to replace union-base type punning~~[Refactor](type) use std::bit_cast to replace union-based type punningJul 12, 2025

fix
ed7777c

BiteTheDDDDt approved these changes Jul 14, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 14, 2025

github-actionsBot added the reviewed label Jul 14, 2025

Mryange approved these changes Jul 14, 2025
View reviewed changes

zclllyybb merged commit 41b9f6b into apache:masterJul 14, 2025

zclllyybb deleted the bit_cast branch July 14, 2025 03:54

Uh oh!

Conversation

zclllyybb commented Jul 11, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 11, 2025

Uh oh!

zclllyybb commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

BE UT Coverage Report

Uh oh!

zclllyybb commented Jul 12, 2025

Uh oh!

doris-robot commented Jul 12, 2025

Uh oh!

doris-robot commented Jul 12, 2025

Uh oh!

doris-robot commented Jul 12, 2025

Uh oh!

doris-robot commented Jul 12, 2025

BE UT Coverage Report

Uh oh!

BiteTheDDDDt left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 14, 2025

Uh oh!

github-actionsBot commented Jul 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zclllyybb commented Jul 11, 2025•
edited
Loading