[feat](Variant) add variant_enable_nested_group property and fix variant nullable issue by eldenmoon · Pull Request #61047 · apache/doris

eldenmoon · 2026-03-04T13:22:43Z

No description provided.

Thearas · 2026-03-04T13:22:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

eldenmoon · 2026-03-04T13:23:24Z

run buildall

eldenmoon · 2026-03-04T13:23:38Z

/review

Copilot

Pull request overview

This PR introduces a new VARIANT column property, variant_enable_nested_group, and wires it through FE/BE metadata serialization (Thrift/Proto) and BE write paths to enable/disable nested-group handling.

Changes:

Add variant_enable_nested_group to Thrift (TColumn) and Proto (ColumnPB) descriptors.
Propagate the flag through FE type/model layers (session variable, property parsing, catalog/nereids VariantType conversion, Column serialization).
Gate BE nested-group routing/prepare behavior in VARIANT column writers based on the tablet schema flag.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
gensrc/thrift/Descriptors.thrift	Adds `variant_enable_nested_group` to `TColumn` for Thrift serialization.
gensrc/proto/olap_file.proto	Adds `variant_enable_nested_group` to `ColumnPB` for on-disk schema serialization.
fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java	Updates VARIANT type construction to pass the new nested-group flag.
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java	Adds a session variable to control the default nested-group setting.
fe/fe-core/src/main/java/org/apache/doris/nereids/types/VariantType.java	Extends Nereids `VariantType` to carry the nested-group flag.
fe/fe-core/src/main/java/org/apache/doris/nereids/types/DataType.java	Includes nested-group flag when converting from catalog VariantType to Nereids VariantType.
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java	Parses the new property and passes it into Nereids `VariantType`.
fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java	Adds property key + parsing + validation rules for `variant_enable_nested_group`.
fe/fe-core/src/main/java/org/apache/doris/catalog/Column.java	Serializes/deserializes nested-group flag to Thrift/Proto and blocks schema-change toggling.
fe/fe-common/src/main/java/org/apache/doris/catalog/VariantType.java	Adds nested-group field to catalog VariantType metadata and equality logic.
fe/fe-common/src/main/java/org/apache/doris/catalog/ScalarType.java	Adds `getVariantEnableNestedGroup()` for backward-compatible access via ScalarType.
be/src/vec/functions/array/function_array_utils.cpp	Ensures VARIANT root uses a nullable nested type when wrapping array element columns.
be/src/olap/tablet_schema.h	Adds nested-group flag to tablet schema VARIANT metadata.
be/src/olap/tablet_schema.cpp	Reads/writes the nested-group flag from/to `ColumnPB`.
be/src/olap/tablet_meta.cpp	Transfers nested-group flag from Thrift `TColumn` into tablet metadata.
be/src/olap/rowset/segment_v2/variant/variant_column_writer_impl.cpp	Conditionally builds routing plan / invokes nested-group provider based on the flag.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions

Code Review Summary: feat add variant_enable_nested_group property

This PR adds a variant_enable_nested_group property plumbed through the entire FE->BE stack (session variable, property analysis, Nereids types, catalog types, Thrift/Protobuf serialization, BE tablet schema, and writer gating). The feature is explicitly gated with a "not supported now" exception. It also includes a correctness fix in function_array_utils.cpp.

Critical Checkpoint Conclusions

Goal / Does the code accomplish it? The goal is to add the plumbing for a variant_enable_nested_group property. The plumbing is complete across all layers. However, there is a critical usability bug: the session variable defaults to true while the feature throws "not supported now" when enabled, breaking all variant<...> column definitions by default.
Is this modification minimal, clear, and focused? Mostly yes. The function_array_utils.cpp fix is unrelated to the property addition and should ideally be a separate commit/PR for clarity, but is a legitimate bug fix.
Concurrency? No new concurrency concerns. The new field is a simple boolean in immutable-after-construction structures.
Lifecycle management? No special lifecycle concerns.
Configuration items added? Yes — session variable default_variant_enable_nested_group (default true). The default is problematic (see issue #1 below).
Incompatible changes / rolling upgrade? New optional fields in Protobuf and Thrift with false defaults — backward compatible. Older BEs will simply ignore the new field.
Parallel code paths? The property is correctly gated in both VariantColumnWriterImpl::finalize() and VariantSubcolumnWriter::finalize().
Test coverage? No tests are added. At minimum, a regression test verifying the property can be set to false and a negative test for the "not supported now" error should be present.
Observability? No observability concerns for a boolean property.
Transaction/persistence? The property is correctly persisted via Protobuf (ColumnPB) and serialized via Thrift (TColumn).
FE-BE variable passing? Correctly set in Column.toThrift() and Column.toPb(), and read in TabletMeta.init_column_from_tcolumn() and TabletColumn.init_from_pb().
Performance? No performance concerns. The gating avoids calling nested group provider methods when disabled.

Issues Found

See inline comments for details.

eldenmoon · 2026-03-04T13:32:57Z

run buildall

eldenmoon · 2026-03-04T13:59:57Z

run buildall

doris-robot · 2026-03-04T14:27:58Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (1798/2266)
Line Coverage	64.52% (32203/49910)
Region Coverage	65.39% (16120/24653)
Branch Coverage	55.90% (8587/15360)

eldenmoon · 2026-03-04T14:41:24Z

run buildall

eldenmoon · 2026-03-04T15:15:40Z

run buildall

hello-stephen · 2026-03-04T15:17:13Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (1798/2266)
Line Coverage	64.55% (32216/49910)
Region Coverage	65.40% (16124/24653)
Branch Coverage	55.89% (8584/15360)

hello-stephen · 2026-03-04T16:27:18Z

FE UT Coverage Report

Increment line coverage 40.54% (15/37) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2026-03-04T16:28:25Z

TPC-H: Total hot run time: 29184 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 516cf7f418cecbd983f6184b8eb12cd453408808, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17632	4801	4408	4408
q2	q3	10717	821	524	524
q4	4707	356	253	253
q5	7892	1205	1024	1024
q6	177	171	146	146
q7	804	852	663	663
q8	10087	1486	1375	1375
q9	5777	4748	4697	4697
q10	6865	1872	1653	1653
q11	466	253	250	250
q12	738	568	465	465
q13	17793	4258	3411	3411
q14	239	232	211	211
q15	948	801	810	801
q16	762	725	683	683
q17	790	906	423	423
q18	5935	5451	5351	5351
q19	1320	976	610	610
q20	523	509	523	509
q21	4602	1978	1458	1458
q22	377	331	269	269
Total cold run time: 99151 ms
Total hot run time: 29184 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4673	4640	4695	4640
q2	q3	1822	2224	1825	1825
q4	879	1194	761	761
q5	4041	4396	4289	4289
q6	185	172	141	141
q7	1774	1634	1567	1567
q8	2558	2711	2570	2570
q9	7304	7543	7275	7275
q10	2615	2816	2389	2389
q11	498	433	406	406
q12	507	579	453	453
q13	4201	4396	3612	3612
q14	272	299	279	279
q15	863	817	854	817
q16	928	772	706	706
q17	1161	1546	1345	1345
q18	7253	6845	6526	6526
q19	898	879	933	879
q20	2109	2203	2086	2086
q21	3981	3430	3370	3370
q22	466	422	370	370
Total cold run time: 48988 ms
Total hot run time: 46306 ms

doris-robot · 2026-03-04T16:39:46Z

TPC-DS: Total hot run time: 183747 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 516cf7f418cecbd983f6184b8eb12cd453408808, data reload: false
query5	4609	631	521	521
query6	337	221	193	193
query7	4227	449	271	271
query8	334	236	230	230
query9	8748	2735	2718	2718
query10	533	361	326	326
query11	16957	17381	17431	17381
query12	195	133	130	130
query13	1450	472	351	351
query14	6932	3295	3014	3014
query14_1	2940	2916	2853	2853
query15	243	258	236	236
query16	4135	463	450	450
query17	1379	785	630	630
query18	2287	453	341	341
query19	207	207	178	178
query20	142	141	132	132
query21	218	138	122	122
query22	5404	5065	4850	4850
query23	17188	16784	16521	16521
query23_1	16619	16654	16592	16592
query24	7174	1614	1223	1223
query24_1	1226	1227	1227	1227
query25	532	459	401	401
query26	1198	255	144	144
query27	2800	466	294	294
query28	4516	1875	1859	1859
query29	777	573	494	494
query30	315	247	213	213
query31	875	728	657	657
query32	78	76	70	70
query33	546	374	279	279
query34	896	893	584	584
query35	637	686	611	611
query36	1099	1141	944	944
query37	130	92	81	81
query38	2940	2918	2835	2835
query39	893	864	855	855
query39_1	849	813	823	813
query40	227	145	135	135
query41	62	59	60	59
query42	108	102	100	100
query43	364	380	351	351
query44 query45	208	191	185	185
query46	878	985	608	608
query47	2112	2121	2061	2061
query48	305	307	232	232
query49	620	473	391	391
query50	674	264	221	221
query51	4036	4140	4076	4076
query52	105	111	98	98
query53	289	336	270	270
query54	289	267	256	256
query55	95	81	83	81
query56	297	300	324	300
query57	1366	1350	1248	1248
query58	287	274	273	273
query59	2656	2707	2586	2586
query60	333	332	326	326
query61	141	139	175	139
query62	633	599	549	549
query63	312	280	277	277
query64	4491	1220	980	980
query65 query66	1339	437	356	356
query67	16553	16312	16362	16312
query68 query69	390	306	295	295
query70	998	935	949	935
query71	359	303	299	299
query72	2837	2692	2507	2507
query73	532	538	325	325
query74	9970	9868	9795	9795
query75	2861	2756	2489	2489
query76	1866	1026	671	671
query77	371	388	305	305
query78	11234	11345	10625	10625
query79	1155	828	612	612
query80	1309	628	568	568
query81	554	294	253	253
query82	1060	151	116	116
query83	336	276	243	243
query84	263	125	98	98
query85	958	539	507	507
query86	415	318	327	318
query87	3122	3111	2977	2977
query88	3582	2642	2627	2627
query89	415	365	341	341
query90	1985	166	169	166
query91	155	155	135	135
query92	74	77	72	72
query93	913	810	496	496
query94	629	322	300	300
query95	598	396	321	321
query96	626	518	225	225
query97	2439	2478	2389	2389
query98	229	212	213	212
query99	1022	1014	916	916
Total cold run time: 256217 ms
Total hot run time: 183747 ms

doris-robot · 2026-03-04T18:00:32Z

BE UT Coverage Report

Increment line coverage 36.00% (9/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.56% (19646/37381)
Line Coverage	36.18% (183447/506983)
Region Coverage	32.48% (142312/438100)
Branch Coverage	33.43% (61740/184658)

hello-stephen · 2026-03-04T21:57:47Z

BE Regression && UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.42% (26142/36603)
Line Coverage	54.22% (274019/505430)
Region Coverage	51.45% (227520/442237)
Branch Coverage	52.85% (97890/185222)

hello-stephen · 2026-03-04T22:10:18Z

FE Regression Coverage Report

Increment line coverage 40.54% (15/37) 🎉
Increment coverage report
Complete coverage report

zclllyybb

LGTM

github-actions · 2026-03-05T02:19:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-03-05T02:19:50Z

PR approved by anyone and no changes requested.

csun5285

LGTM

eldenmoon · 2026-03-05T02:27:26Z

skip check_coverage

hello-stephen · 2026-03-05T02:31:45Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.35% (1798/2266)
Line Coverage	64.53% (32208/49910)
Region Coverage	65.44% (16132/24653)
Branch Coverage	55.96% (8596/15360)

hello-stephen · 2026-03-05T04:01:01Z

BE Regression && UT Coverage Report

Increment line coverage 56.00% (14/25) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.41% (26140/36603)
Line Coverage	54.20% (273920/505430)
Region Coverage	51.43% (227453/442237)
Branch Coverage	52.84% (97866/185222)

eldenmoon · 2026-03-05T04:03:13Z

skip check_coverage

hello-stephen · 2026-03-05T04:11:57Z

FE Regression Coverage Report

Increment line coverage 40.54% (15/37) 🎉
Increment coverage report
Complete coverage report

…ant nullable issue (apache#61047) (cherry picked from commit 405d8df)

…ith search DSL NESTED clause (#60847) (#61393) cherry-pick #60847 #60930 #61047 #61065 #61090 #61115

[feat](Variant) add variant_enable_nested_group property
a84bed7

eldenmoon requested review from dataroaring, gavinchou, yiguolei and zclllyybb as code owners March 4, 2026 13:22

CopilotAI review requested due to automatic review settings March 4, 2026 13:22

Copilotstarted reviewing on behalf of eldenmoon March 4, 2026 13:23 View session

CopilotAI reviewed Mar 4, 2026
View reviewed changes

github-actionsBot reviewed Mar 4, 2026
View reviewed changes

zclllyybb reviewed Mar 4, 2026
View reviewed changes

Comment threadbe/src/vec/functions/array/function_array_utils.cpp Outdated

eldenmoon changed the title ~~[feat](Variant) add variant_enable_nested_group property~~[feat](Variant) add variant_enable_nested_group property and fix variant nullable issueMar 4, 2026

eldenmoon force-pushed the enable_nested_group-opensrc branch from b951af9 to 2e1e4aeCompare March 4, 2026 13:59

zclllyybb reviewed Mar 4, 2026
View reviewed changes

Comment threadbe/src/vec/functions/array/function_array_utils.cpp Outdated

eldenmoon force-pushed the enable_nested_group-opensrc branch from 2e1e4ae to 06a53c9Compare March 4, 2026 14:41

fix assmue_mutable
516cf7f

eldenmoon force-pushed the enable_nested_group-opensrc branch from 06a53c9 to 516cf7fCompare March 4, 2026 15:14

zclllyybb approved these changes Mar 5, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 5, 2026

github-actionsBot added the reviewed label Mar 5, 2026

csun5285 approved these changes Mar 5, 2026
View reviewed changes

yiguolei approved these changes Mar 5, 2026
View reviewed changes

eldenmoon merged commit 405d8df into apache:masterMar 5, 2026
29 of 32 checks passed

eldenmoon added the dev/4.1.x label Mar 12, 2026

eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Mar 16, 2026
[feat](Variant) add variant_enable_nested_group property and fix vari…
e3c7986
…ant nullable issue (apache#61047) (cherry picked from commit 405d8df)

eldenmoon mentioned this pull request Mar 16, 2026
branch-4.1 :[feat](Variant) Support NestedGroup for variant columns with search DSL NESTED clause (#60847) #61393
Merged

yiguolei pushed a commit that referenced this pull request Mar 17, 2026
branch-4.1 :[feat](Variant) Support NestedGroup for variant columns w…
c26b5da
…ith search DSL NESTED clause (#60847) (#61393) cherry-pick #60847 #60930 #61047 #61065 #61090 #61115

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Mar 17, 2026

Uh oh!

Conversation

eldenmoon commented Mar 4, 2026

Uh oh!

Thearas commented Mar 4, 2026

Uh oh!

eldenmoon commented Mar 4, 2026

Uh oh!

eldenmoon commented Mar 4, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Code Review Summary: feat add variant_enable_nested_group property

Critical Checkpoint Conclusions

Issues Found

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Mar 4, 2026

Uh oh!

Uh oh!

eldenmoon commented Mar 4, 2026

Uh oh!

doris-robot commented Mar 4, 2026

Cloud UT Coverage Report

Uh oh!

Uh oh!

eldenmoon commented Mar 4, 2026

Uh oh!

eldenmoon commented Mar 4, 2026

Uh oh!

hello-stephen commented Mar 4, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 4, 2026

FE UT Coverage Report

Uh oh!

doris-robot commented Mar 4, 2026

Uh oh!

doris-robot commented Mar 4, 2026

Uh oh!

doris-robot commented Mar 4, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Mar 4, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Mar 4, 2026

FE Regression Coverage Report

Uh oh!

zclllyybb left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 5, 2026

Uh oh!

github-actionsBot commented Mar 5, 2026

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented Mar 5, 2026

Uh oh!

hello-stephen commented Mar 5, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 5, 2026