[opt](variant) Optimize VARIANT doc mode load/compaction and add benchmarks by eldenmoon · Pull Request #60511 · apache/doris

eldenmoon · 2026-02-04T17:11:04Z

Improve VARIANT doc-mode ingestion and compaction performance, and add a
write/compaction benchmark UT.

Key changes:

Add config enable_variant_doc_compaction_sparse_write (default: true) to
enable sparse writing for materialized subcolumns during doc-mode compaction.
Optimize doc value column generation: deduplicate paths, sort paths once, and
serialize values directly into the binary column to reduce allocations/copies.
Optimize bucket sharding/stats collection with phmap and cached path->bucket
mapping; remove per-row doc value sorting in writer.
Refactor VariantDocCompactWriter finalize flow with an explicit write plan;
skip invalid array types and fix doc materialization offset handling.
Update variant-related UTs and add VariantDocModeCompactionTest to benchmark
segment import and doc-mode compaction.

UT Testing:

BE UT: VariantColumnWriterReaderTest., VariantUtilTest., ColumnVariantTest.*

Benchmark Testing(Release compile):

BE UT (Release-only, slow): VariantDocModeCompactionTest.*

eldenmoon · 2026-02-04T17:11:15Z

run buildall

hello-stephen · 2026-02-04T17:11:23Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR focuses on optimizing VARIANT “doc mode” ingestion/compaction paths by reducing overhead in doc-value serialization/materialization and by switching several hot-path maps/sets to phmap for better performance, with accompanying new/expanded tests.

Changes:

Optimize doc-value column handling (avoid sorting, faster serialization path, phmap usage in several structures).
Adjust variant compaction validation logic to tolerate doc/sparse extracted columns in output schemas.
Add/extend unit tests, including a new doc-mode compaction test and additional doc-value serialization coverage.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
be/test/olap/variant_doc_mode_compaction_test.cpp	Adds a new end-to-end compaction test for VARIANT doc mode (currently heavy and non-hermetic).
be/test/olap/rowset/segment_v2/variant_util_test.cpp	Adds coverage for `OnlyDocValueColumn` serialization with mixed types.
be/src/vec/common/variant_util.h	Changes `materialize_docs_to_subcolumns_map` return type to `phmap::flat_hash_map`.
be/src/vec/common/variant_util.cpp	Implements faster doc-value serialization and updates doc materialization map implementation.
be/src/vec/columns/subcolumn_tree.h	Switches node children map to `phmap::flat_hash_map`.
be/src/vec/columns/column_variant.h / be/src/vec/columns/column_variant.cpp	Removes `sort_doc_value_column` support.
be/src/olap/rowset/segment_v2/variant/variant_column_writer_impl.cpp	Reworks sparse/doc bucket writing/statistics with `phmap`; introduces offset-iteration changes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

eldenmoon · 2026-02-04T17:49:31Z

run buildall

eldenmoon · 2026-02-04T18:04:48Z

run buildall

doris-robot · 2026-02-04T18:39:36Z

TPC-H: Total hot run time: 31662 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f5ce85ae8d81b365a6bd810ea9834b2b7f79251c, data reload: false
------ Round 1 ----------------------------------
q1	17667	5265	5100	5100
q2	2062	310	195	195
q3	10213	1380	736	736
q4	10233	939	322	322
q5	7887	2186	1900	1900
q6	211	186	153	153
q7	901	720	605	605
q8	9258	1431	1088	1088
q9	5518	4986	4820	4820
q10	6869	1934	1570	1570
q11	517	307	285	285
q12	368	375	221	221
q13	17820	4058	3228	3228
q14	242	247	223	223
q15	934	832	803	803
q16	676	672	616	616
q17	641	820	477	477
q18	6710	6514	6369	6369
q19	1546	986	615	615
q20	389	352	230	230
q21	2647	2114	1836	1836
q22	348	314	270	270
Total cold run time: 103657 ms
Total hot run time: 31662 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5454	5329	5296	5296
q2	258	349	254	254
q3	2161	2695	2286	2286
q4	1345	1723	1320	1320
q5	4293	4179	4314	4179
q6	211	187	173	173
q7	2299	2089	1870	1870
q8	2567	2558	2452	2452
q9	7458	7624	7398	7398
q10	2765	3112	2532	2532
q11	544	484	448	448
q12	657	760	675	675
q13	3929	4525	3601	3601
q14	343	331	287	287
q15	871	804	811	804
q16	671	774	681	681
q17	1191	1337	1319	1319
q18	8058	7957	8031	7957
q19	909	847	866	847
q20	2063	2126	2009	2009
q21	4483	4255	4142	4142
q22	597	550	511	511
Total cold run time: 53127 ms
Total hot run time: 51041 ms

doris-robot · 2026-02-04T18:56:13Z

ClickBench: Total hot run time: 28.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f5ce85ae8d81b365a6bd810ea9834b2b7f79251c, data reload: false
query1	0.05	0.04	0.05
query2	0.09	0.04	0.04
query3	0.26	0.08	0.09
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.16	0.68	0.68
query7	0.03	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.51	0.50
query10	0.55	0.55	0.54
query11	0.14	0.09	0.09
query12	0.14	0.10	0.11
query13	0.64	0.62	0.63
query14	1.07	1.08	1.04
query15	0.87	0.86	0.87
query16	0.38	0.42	0.42
query17	1.20	1.17	1.10
query18	0.23	0.21	0.20
query19	2.07	1.95	2.07
query20	0.02	0.01	0.01
query21	15.40	0.30	0.14
query22	5.25	0.05	0.05
query23	16.00	0.30	0.10
query24	1.94	0.26	0.20
query25	0.09	0.08	0.06
query26	0.15	0.15	0.13
query27	0.05	0.05	0.08
query28	3.89	1.16	0.97
query29	12.60	3.95	3.18
query30	0.28	0.14	0.11
query31	2.82	0.63	0.41
query32	3.23	0.60	0.49
query33	3.19	3.31	3.36
query34	16.09	5.38	4.72
query35	4.78	4.83	4.84
query36	0.64	0.51	0.49
query37	0.12	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.16	0.16
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 98.43 s
Total hot run time: 28.31 s

eldenmoon · 2026-02-05T17:55:09Z

run buildall

doris-robot · 2026-02-05T20:00:02Z

TPC-H: Total hot run time: 31175 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5936af5b0d0bacb601de1486dc59922c6d1dd7c, data reload: false
------ Round 1 ----------------------------------
q1	17635	4476	4297	4297
q2	2027	380	250	250
q3	10117	1298	744	744
q4	10232	821	321	321
q5	8238	2171	1893	1893
q6	231	177	145	145
q7	881	729	585	585
q8	9286	1402	1086	1086
q9	5372	4848	4837	4837
q10	6835	1928	1564	1564
q11	501	298	267	267
q12	388	375	216	216
q13	17819	4096	3262	3262
q14	236	233	215	215
q15	920	813	800	800
q16	694	668	632	632
q17	770	798	476	476
q18	6985	6521	6543	6521
q19	1442	976	615	615
q20	379	352	233	233
q21	2605	2030	1937	1937
q22	351	314	279	279
Total cold run time: 103944 ms
Total hot run time: 31175 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4445	4373	4335	4335
q2	248	336	261	261
q3	2129	2593	2229	2229
q4	1338	1732	1298	1298
q5	4268	4119	4237	4119
q6	214	185	137	137
q7	1866	1825	2038	1825
q8	2632	2556	2516	2516
q9	7519	7481	7539	7481
q10	2877	3130	2632	2632
q11	574	470	488	470
q12	673	739	615	615
q13	3827	4727	3645	3645
q14	291	302	282	282
q15	890	819	863	819
q16	704	734	697	697
q17	1199	1402	1343	1343
q18	8346	7946	7747	7747
q19	907	851	835	835
q20	2059	2156	2006	2006
q21	4707	4330	4084	4084
q22	563	560	500	500
Total cold run time: 52276 ms
Total hot run time: 49876 ms

doris-robot · 2026-02-05T20:16:49Z

ClickBench: Total hot run time: 28.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b5936af5b0d0bacb601de1486dc59922c6d1dd7c, data reload: false
query1	0.06	0.05	0.04
query2	0.09	0.04	0.05
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.28	0.25	0.24
query6	1.16	0.70	0.69
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.51	0.50
query10	0.55	0.56	0.55
query11	0.14	0.09	0.09
query12	0.14	0.11	0.10
query13	0.64	0.61	0.63
query14	1.07	1.06	1.04
query15	0.88	0.86	0.87
query16	0.40	0.38	0.38
query17	1.11	1.11	1.14
query18	0.22	0.21	0.21
query19	2.10	1.94	2.04
query20	0.02	0.02	0.01
query21	15.41	0.26	0.15
query22	5.34	0.05	0.06
query23	16.24	0.29	0.10
query24	0.97	0.24	0.18
query25	0.08	0.10	0.05
query26	0.15	0.13	0.13
query27	0.08	0.06	0.07
query28	3.24	1.15	0.97
query29	12.53	3.95	3.15
query30	0.27	0.13	0.12
query31	2.80	0.65	0.40
query32	3.26	0.60	0.50
query33	3.29	3.23	3.25
query34	16.04	5.39	4.74
query35	4.88	4.76	4.87
query36	0.65	0.50	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.20	0.16	0.16
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.19 s
Total hot run time: 28.13 s

eldenmoon · 2026-02-06T02:32:23Z

run p0 5

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

eldenmoon · 2026-02-06T06:19:59Z

run buildall

doris-robot · 2026-02-06T08:16:41Z

TPC-H: Total hot run time: 30809 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6ae78f1afa51081a546188e0c923f77bd2f568c1, data reload: false
------ Round 1 ----------------------------------
q1	17639	4506	4301	4301
q2	2006	363	271	271
q3	10129	1363	740	740
q4	10252	768	309	309
q5	8313	2253	1909	1909
q6	227	179	147	147
q7	916	744	635	635
q8	9272	1405	1221	1221
q9	4905	4659	4695	4659
q10	6876	1938	1553	1553
q11	529	310	294	294
q12	404	377	229	229
q13	17782	4037	3191	3191
q14	233	247	219	219
q15	881	812	808	808
q16	664	680	620	620
q17	880	878	469	469
q18	6546	6001	6459	6001
q19	1576	1027	670	670
q20	567	534	395	395
q21	2755	2012	1873	1873
q22	361	347	295	295
Total cold run time: 103713 ms
Total hot run time: 30809 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4671	4675	5774	4675
q2	286	359	266	266
q3	2352	2944	2478	2478
q4	1439	1871	1428	1428
q5	4611	4502	4581	4502
q6	222	190	137	137
q7	2052	1871	1806	1806
q8	2570	2503	2425	2425
q9	7421	7296	7465	7296
q10	2976	3012	2581	2581
q11	645	549	457	457
q12	666	744	602	602
q13	3793	4137	3210	3210
q14	268	293	265	265
q15	828	797	778	778
q16	648	688	637	637
q17	1093	1296	1323	1296
q18	7507	7206	7290	7206
q19	846	803	781	781
q20	2009	2038	1866	1866
q21	4472	4245	4166	4166
q22	586	546	587	546
Total cold run time: 51961 ms
Total hot run time: 49404 ms

doris-robot · 2026-02-06T08:33:17Z

ClickBench: Total hot run time: 28.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6ae78f1afa51081a546188e0c923f77bd2f568c1, data reload: false
query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.25	0.09	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.16	0.68	0.68
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.49	0.49
query10	0.55	0.54	0.53
query11	0.14	0.10	0.09
query12	0.15	0.10	0.10
query13	0.64	0.62	0.62
query14	1.05	1.06	1.06
query15	0.89	0.86	0.87
query16	0.42	0.40	0.39
query17	1.11	1.14	1.07
query18	0.22	0.20	0.20
query19	2.03	1.96	1.93
query20	0.02	0.01	0.01
query21	15.41	0.27	0.15
query22	5.08	0.06	0.05
query23	16.03	0.28	0.11
query24	0.95	0.25	0.17
query25	0.08	0.09	0.06
query26	0.14	0.14	0.13
query27	0.06	0.09	0.06
query28	2.98	1.15	0.96
query29	12.57	3.85	3.16
query30	0.27	0.13	0.13
query31	2.82	0.64	0.39
query32	3.23	0.59	0.50
query33	3.25	3.27	3.27
query34	16.13	5.37	4.72
query35	4.77	4.79	4.78
query36	0.65	0.50	0.49
query37	0.11	0.08	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.02
query40	0.19	0.16	0.15
query41	0.09	0.03	0.02
query42	0.04	0.04	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.29 s
Total hot run time: 28.11 s

eldenmoon · 2026-02-06T15:56:20Z

run buildall

doris-robot · 2026-02-06T16:35:04Z

TPC-H: Total hot run time: 30123 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46b88679a10524398325dd2a447b040858976e4f, data reload: false
------ Round 1 ----------------------------------
q1	17617	4555	4320	4320
q2	2041	350	260	260
q3	10124	1268	726	726
q4	10195	761	304	304
q5	7507	2154	1940	1940
q6	192	175	143	143
q7	876	730	625	625
q8	9261	1356	1094	1094
q9	4683	4663	4575	4575
q10	6812	1919	1593	1593
q11	503	314	279	279
q12	334	378	221	221
q13	17977	4076	3293	3293
q14	236	240	231	231
q15	913	820	791	791
q16	684	670	638	638
q17	703	869	445	445
q18	6679	5694	5619	5619
q19	1427	977	618	618
q20	503	488	388	388
q21	2495	1799	1749	1749
q22	361	317	271	271
Total cold run time: 102123 ms
Total hot run time: 30123 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4386	4343	4352	4343
q2	254	341	255	255
q3	2073	2665	2235	2235
q4	1337	1729	1302	1302
q5	4260	4169	4275	4169
q6	220	177	133	133
q7	1850	1803	1673	1673
q8	2465	2636	2463	2463
q9	7623	7490	7516	7490
q10	2798	3012	2750	2750
q11	545	503	494	494
q12	670	756	687	687
q13	3885	4312	3437	3437
q14	309	424	319	319
q15	926	926	826	826
q16	676	709	657	657
q17	1127	1383	1374	1374
q18	8154	7779	8057	7779
q19	878	865	954	865
q20	2134	2105	1998	1998
q21	4745	4494	3988	3988
q22	595	581	513	513
Total cold run time: 51910 ms
Total hot run time: 49750 ms

doris-robot · 2026-02-06T16:51:41Z

ClickBench: Total hot run time: 28.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46b88679a10524398325dd2a447b040858976e4f, data reload: false
query1	0.06	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.18	0.67	0.67
query7	0.04	0.02	0.03
query8	0.06	0.04	0.05
query9	0.57	0.51	0.50
query10	0.54	0.56	0.55
query11	0.14	0.09	0.10
query12	0.13	0.11	0.11
query13	0.62	0.62	0.62
query14	1.08	1.05	1.05
query15	0.89	0.87	0.90
query16	0.39	0.42	0.38
query17	1.16	1.14	1.14
query18	0.22	0.21	0.21
query19	2.14	2.02	2.04
query20	0.02	0.01	0.01
query21	15.39	0.26	0.14
query22	5.26	0.06	0.05
query23	15.95	0.28	0.11
query24	1.04	0.27	0.54
query25	0.13	0.08	0.09
query26	0.15	0.14	0.13
query27	0.06	0.05	0.08
query28	4.08	1.17	0.97
query29	12.56	4.02	3.20
query30	0.28	0.13	0.13
query31	2.81	0.64	0.41
query32	3.24	0.59	0.49
query33	3.28	3.27	3.24
query34	15.92	5.36	4.75
query35	4.75	4.83	4.79
query36	0.67	0.50	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.02
query40	0.20	0.17	0.16
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 97.65 s
Total hot run time: 28.5 s

eldenmoon · 2026-02-07T08:04:14Z

run buildall

Copilot

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 6 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

eldenmoon · 2026-02-09T04:59:05Z

run buildall

…hmarks Improve VARIANT doc-mode ingestion and compaction performance, and add a write/compaction benchmark UT. Key changes: - Add config `enable_variant_doc_compaction_sparse_write` (default: true) to enable sparse writing for materialized subcolumns during doc-mode compaction. - Optimize doc value column generation: deduplicate paths, sort paths once, and serialize values directly into the binary column to reduce allocations/copies. - Optimize bucket sharding/stats collection with phmap and cached path->bucket mapping; remove per-row doc value sorting in writer. - Refactor VariantDocCompactWriter finalize flow with an explicit write plan; skip invalid array types and fix doc materialization offset handling. - Update variant-related UTs and add VariantDocModeCompactionTest to benchmark segment import and doc-mode compaction. Benchmark Testing: - BE UT: VariantColumnWriterReaderTest.*, VariantUtilTest.*, ColumnVariantTest.* - BE UT (Release-only, slow): VariantDocModeCompactionTest.*

eldenmoon · 2026-02-09T05:36:37Z

run buildall

doris-robot · 2026-02-09T07:26:47Z

TPC-H: Total hot run time: 30589 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e80457f76fb5655cc4dc91fdea69dd53199b508e, data reload: false
------ Round 1 ----------------------------------
q1	17671	4567	4275	4275
q2	1999	380	238	238
q3	10155	1339	735	735
q4	10197	780	317	317
q5	7519	2222	1920	1920
q6	209	179	144	144
q7	891	718	611	611
q8	9279	1396	1224	1224
q9	4832	4650	4570	4570
q10	6886	1972	1544	1544
q11	539	321	291	291
q12	404	374	236	236
q13	17785	4109	3260	3260
q14	238	240	208	208
q15	893	825	798	798
q16	698	683	623	623
q17	705	796	563	563
q18	6472	6067	5959	5959
q19	1096	987	605	605
q20	510	494	393	393
q21	2539	1851	1796	1796
q22	365	325	279	279
Total cold run time: 101882 ms
Total hot run time: 30589 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4336	4329	4330	4329
q2	270	335	267	267
q3	2091	2649	2249	2249
q4	1354	1720	1321	1321
q5	4294	4156	4257	4156
q6	214	177	137	137
q7	1846	1783	1676	1676
q8	2559	2603	2523	2523
q9	7655	7464	7458	7458
q10	2827	3260	2672	2672
q11	555	480	457	457
q12	710	763	659	659
q13	4083	4316	3561	3561
q14	346	296	283	283
q15	885	836	823	823
q16	688	843	737	737
q17	1146	1293	1275	1275
q18	8214	7963	7873	7873
q19	870	895	876	876
q20	2137	2169	2034	2034
q21	4824	4619	4529	4529
q22	577	573	562	562
Total cold run time: 52481 ms
Total hot run time: 50457 ms

doris-robot · 2026-02-09T07:43:35Z

ClickBench: Total hot run time: 28.7 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e80457f76fb5655cc4dc91fdea69dd53199b508e, data reload: false
query1	0.05	0.05	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.11	0.11
query5	0.27	0.25	0.26
query6	1.17	0.67	0.68
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.61	0.51	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.65	0.62	0.62
query14	1.08	1.07	1.06
query15	0.89	0.87	0.89
query16	0.40	0.40	0.40
query17	1.17	1.17	1.19
query18	0.23	0.22	0.21
query19	2.02	2.03	2.10
query20	0.02	0.01	0.01
query21	15.40	0.25	0.15
query22	5.16	0.06	0.05
query23	15.89	0.30	0.11
query24	0.94	0.38	0.45
query25	0.07	0.13	0.08
query26	0.14	0.13	0.14
query27	0.05	0.09	0.08
query28	3.96	1.14	0.97
query29	12.56	3.94	3.18
query30	0.27	0.14	0.12
query31	2.80	0.64	0.41
query32	3.23	0.59	0.49
query33	3.19	3.25	3.31
query34	16.48	5.42	4.75
query35	4.81	4.76	4.81
query36	0.65	0.51	0.49
query37	0.11	0.07	0.07
query38	0.08	0.05	0.04
query39	0.05	0.03	0.03
query40	0.20	0.16	0.16
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 97.65 s
Total hot run time: 28.7 s

csun5285

LGTM

github-actions · 2026-02-09T08:05:04Z

PR approved by anyone and no changes requested.

eldenmoon · 2026-02-09T09:12:38Z

run buildall

doris-robot · 2026-02-09T10:51:58Z

TPC-H: Total hot run time: 30829 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit afbdddea56e078b3e96485ea734f6937f6650f4a, data reload: false
------ Round 1 ----------------------------------
q1	17606	4426	4254	4254
q2	2007	347	237	237
q3	10182	1364	748	748
q4	10198	824	314	314
q5	7546	2235	1929	1929
q6	196	184	147	147
q7	914	752	604	604
q8	9273	1411	1261	1261
q9	4883	4701	4644	4644
q10	6844	1947	1591	1591
q11	523	320	292	292
q12	346	383	225	225
q13	17777	4112	3244	3244
q14	237	252	212	212
q15	883	801	807	801
q16	697	680	624	624
q17	714	841	517	517
q18	6784	5863	6268	5863
q19	1295	1060	698	698
q20	577	545	410	410
q21	2772	1971	1915	1915
q22	397	333	299	299
Total cold run time: 102651 ms
Total hot run time: 30829 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4534	4649	4693	4649
q2	259	359	262	262
q3	2354	2923	2472	2472
q4	1467	1893	1409	1409
q5	4718	4645	4696	4645
q6	220	179	135	135
q7	1919	2069	1747	1747
q8	2570	2417	2499	2417
q9	7465	7459	7737	7459
q10	2849	3073	2740	2740
q11	544	462	461	461
q12	739	812	649	649
q13	3799	4530	3529	3529
q14	283	320	281	281
q15	867	801	775	775
q16	646	686	634	634
q17	1090	1274	1306	1274
q18	7670	7358	7273	7273
q19	879	822	816	816
q20	1969	2041	1853	1853
q21	4625	4331	4126	4126
q22	560	544	527	527
Total cold run time: 52026 ms
Total hot run time: 50133 ms

doris-robot · 2026-02-09T11:08:46Z

ClickBench: Total hot run time: 28.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit afbdddea56e078b3e96485ea734f6937f6650f4a, data reload: false
query1	0.05	0.05	0.05
query2	0.09	0.05	0.04
query3	0.24	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.24
query6	1.15	0.68	0.66
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.55	0.55	0.54
query11	0.14	0.09	0.10
query12	0.14	0.11	0.11
query13	0.63	0.61	0.62
query14	1.06	1.06	1.05
query15	0.87	0.87	0.88
query16	0.40	0.37	0.38
query17	1.09	1.17	1.18
query18	0.22	0.21	0.22
query19	2.13	2.05	2.03
query20	0.02	0.01	0.01
query21	15.40	0.27	0.14
query22	5.20	0.06	0.05
query23	15.87	0.29	0.11
query24	1.03	0.60	1.02
query25	0.13	0.06	0.06
query26	0.14	0.14	0.13
query27	0.06	0.06	0.06
query28	5.13	1.14	0.97
query29	12.59	3.96	3.19
query30	0.28	0.16	0.11
query31	2.82	0.64	0.40
query32	3.24	0.61	0.50
query33	3.32	3.29	3.26
query34	16.05	5.41	4.67
query35	4.81	4.81	4.85
query36	0.66	0.51	0.50
query37	0.11	0.07	0.08
query38	0.08	0.05	0.03
query39	0.04	0.03	0.03
query40	0.19	0.16	0.15
query41	0.09	0.03	0.02
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.64 s
Total hot run time: 28.72 s

hello-stephen · 2026-02-09T12:28:27Z

BE UT Coverage Report

Increment line coverage 72.71% (373/513) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.73% (19454/36894)
Line Coverage	36.22% (181151/500179)
Region Coverage	32.59% (140592/431452)
Branch Coverage	33.64% (60964/181215)

hello-stephen · 2026-02-09T13:52:01Z

BE Regression && UT Coverage Report

Increment line coverage 87.52% (449/513) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.36% (26518/36150)
Line Coverage	56.37% (281276/498941)
Region Coverage	53.88% (234827/435834)
Branch Coverage	55.61% (101161/181919)

airborne12

LGTM

github-actions · 2026-02-10T02:47:14Z

PR approved by at least one committer and no changes requested.

…hmarks (apache#60511) Improve VARIANT doc-mode ingestion and compaction performance, and add a write/compaction benchmark UT. Key changes: - Add config `enable_variant_doc_compaction_sparse_write` (default: true) to enable sparse writing for materialized subcolumns during doc-mode compaction. - Optimize doc value column generation: deduplicate paths, sort paths once, and serialize values directly into the binary column to reduce allocations/copies. - Optimize bucket sharding/stats collection with phmap and cached path->bucket mapping; remove per-row doc value sorting in writer. - Refactor VariantDocCompactWriter finalize flow with an explicit write plan; skip invalid array types and fix doc materialization offset handling. - Update variant-related UTs and add VariantDocModeCompactionTest to benchmark segment import and doc-mode compaction. UT Testing: - BE UT: VariantColumnWriterReaderTest.*, VariantUtilTest.*, ColumnVariantTest.* Benchmark Testing(Release compile): - BE UT (Release-only, slow): VariantDocModeCompactionTest.* <img width="902" height="778" alt="image" src="https://github.com/user-attachments/assets/08fb9341-2620-484d-9cf9-f3a64c95d4bd" />

pick from master #59183, #60511, #60518, #60668 --------- Co-authored-by: lihangyu <lihangyu@selectdb.com>

CopilotAI review requested due to automatic review settings February 4, 2026 17:11

Copilotstarted reviewing on behalf of eldenmoon February 4, 2026 17:11 View session

CopilotAI reviewed Feb 4, 2026
View reviewed changes

eldenmoon force-pushed the doc-load-opt branch from 03ef62f to 065fe1fCompare February 4, 2026 17:48

eldenmoon force-pushed the doc-load-opt branch from 065fe1f to f5ce85aCompare February 4, 2026 18:04

eldenmoon force-pushed the doc-load-opt branch from f5ce85a to b5936afCompare February 5, 2026 17:54

eldenmoon requested a review from Copilot February 6, 2026 02:31

Copilotstarted reviewing on behalf of eldenmoon February 6, 2026 02:32 View session

CopilotAI reviewed Feb 6, 2026
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/column_writer.cpp Outdated

eldenmoon force-pushed the doc-load-opt branch from 6ae78f1 to 46b8867Compare February 6, 2026 15:56

eldenmoon requested a review from Copilot February 7, 2026 08:05

Copilotstarted reviewing on behalf of eldenmoon February 7, 2026 08:06 View session

CopilotAI reviewed Feb 7, 2026
View reviewed changes

eldenmoon force-pushed the doc-load-opt branch 2 times, most recently from da975a5 to 6f5e3b1Compare February 9, 2026 04:00

eldenmoon changed the title ~~Doc load opt~~[opt](variant) Optimize VARIANT doc mode load/compaction and add benchmarksFeb 9, 2026

eldenmoon added the dev/4.1.x label Feb 9, 2026

eldenmoon force-pushed the doc-load-opt branch from 6f5e3b1 to e80457fCompare February 9, 2026 05:08

csun5285 approved these changes Feb 9, 2026
View reviewed changes

github-actionsBot added the reviewed label Feb 9, 2026

fix ut
afbddde

airborne12 approved these changes Feb 10, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2026

eldenmoon merged commit 5eddc6c into apache:masterFeb 10, 2026
27 of 28 checks passed

csun5285 mentioned this pull request Mar 12, 2026
[branch-4.1][feature](variant) add variant doc mode (#59183) #61281
Merged

eldenmoon added a commit that referenced this pull request Mar 16, 2026
[branch-4.1][feature](variant) add variant doc mode (#59183) (#61281)
8f280b3
pick from master #59183, #60511, #60518, #60668 --------- Co-authored-by: lihangyu <lihangyu@selectdb.com>

eldenmoon added dev/4.1.0-merged and removed dev/4.1.x labels Mar 17, 2026

Uh oh!

Conversation

eldenmoon commented Feb 4, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eldenmoon commented Feb 4, 2026

Uh oh!

hello-stephen commented Feb 4, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Feb 4, 2026

Uh oh!

eldenmoon commented Feb 4, 2026

Uh oh!

doris-robot commented Feb 4, 2026

Uh oh!

doris-robot commented Feb 4, 2026

Uh oh!

eldenmoon commented Feb 5, 2026

Uh oh!

doris-robot commented Feb 5, 2026

Uh oh!

doris-robot commented Feb 5, 2026

Uh oh!

eldenmoon commented Feb 6, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

eldenmoon commented Feb 6, 2026

Uh oh!

doris-robot commented Feb 6, 2026

Uh oh!

doris-robot commented Feb 6, 2026

Uh oh!

eldenmoon commented Feb 6, 2026

Uh oh!

doris-robot commented Feb 6, 2026

Uh oh!

doris-robot commented Feb 6, 2026

Uh oh!

eldenmoon commented Feb 7, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Feb 9, 2026

Uh oh!

eldenmoon commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

Uh oh!

doris-robot commented Feb 9, 2026

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 9, 2026

Uh oh!

eldenmoon commented Feb 9, 2026

Uh oh!

eldenmoon commented Feb 4, 2026•
edited
Loading