[branch-4.1][feature](variant) add variant doc mode (#59183) by csun5285 · Pull Request #61281 · apache/doris

csun5285 · 2026-03-12T11:22:55Z

pick from master #59183, #60511, #60518, #60668

hello-stephen · 2026-03-12T11:23:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

csun5285 · 2026-03-12T11:23:07Z

run buildall

csun5285 · 2026-03-12T11:27:36Z

run buildall

hello-stephen · 2026-03-12T12:02:03Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.50% (31945/49530)
Region Coverage	65.29% (15976/24468)
Branch Coverage	55.84% (8495/15214)

hello-stephen · 2026-03-12T12:40:17Z

FE UT Coverage Report

Increment line coverage 33.57% (47/140) 🎉
Increment coverage report
Complete coverage report

csun5285 · 2026-03-14T13:44:34Z

run buildall

doris-robot · 2026-03-14T14:35:46Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.48% (31938/49530)
Region Coverage	65.29% (15976/24468)
Branch Coverage	55.87% (8500/15214)

csun5285 · 2026-03-15T05:22:30Z

run buildall

doris-robot · 2026-03-15T05:59:25Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.46% (31928/49530)
Region Coverage	65.30% (15977/24468)
Branch Coverage	55.83% (8494/15214)

csun5285 · 2026-03-16T04:32:53Z

run buildall

doris-robot · 2026-03-16T04:56:33Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.49% (31941/49530)
Region Coverage	65.29% (15976/24468)
Branch Coverage	55.90% (8504/15214)

**DOC encoding**: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document. doc: apache/doris-website#3253

…hmarks (apache#60511) Improve VARIANT doc-mode ingestion and compaction performance, and add a write/compaction benchmark UT. Key changes: - Add config `enable_variant_doc_compaction_sparse_write` (default: true) to enable sparse writing for materialized subcolumns during doc-mode compaction. - Optimize doc value column generation: deduplicate paths, sort paths once, and serialize values directly into the binary column to reduce allocations/copies. - Optimize bucket sharding/stats collection with phmap and cached path->bucket mapping; remove per-row doc value sorting in writer. - Refactor VariantDocCompactWriter finalize flow with an explicit write plan; skip invalid array types and fix doc materialization offset handling. - Update variant-related UTs and add VariantDocModeCompactionTest to benchmark segment import and doc-mode compaction. UT Testing: - BE UT: VariantColumnWriterReaderTest.*, VariantUtilTest.*, ColumnVariantTest.* Benchmark Testing(Release compile): - BE UT (Release-only, slow): VariantDocModeCompactionTest.* <img width="902" height="778" alt="image" src="https://github.com/user-attachments/assets/08fb9341-2620-484d-9cf9-f3a64c95d4bd" />

…index_fault_injection (apache#60518)

…dom session variables Pin default_variant_enable_doc_mode=false and default_variant_max_subcolumns_count=2048 in search variant test suites to prevent fuzzy mode randomization from disabling subcolumn extraction, which causes inverted index search to return empty results.

Fix two test failures caused by variant doc mode cherry-picks: 1. ColumnVariantTest.test_variant_deserialize_from_sparse_column: Field element type (TYPE_INT) mismatched FieldInfo scalar_type_id (TYPE_TINYINT), causing Field::get<TINYINT>() assertion failure. Fixed by using matching TYPE_TINYINT field for first insert and separate TYPE_INT field for second insert (type promotion test). 2. NativeReaderWriterTest.read_all_types_from_pregenerated_file: Pre-generated binary test data was incompatible with updated Variant serialization format. Regenerated the test data file.

csun5285 · 2026-03-16T07:13:20Z

run buildall

hello-stephen · 2026-03-16T07:45:41Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.15% (1788/2259)
Line Coverage	64.48% (31938/49530)
Region Coverage	65.31% (15981/24468)
Branch Coverage	55.84% (8496/15214)

hello-stephen · 2026-03-16T08:32:21Z

FE UT Coverage Report

Increment line coverage 33.57% (47/140) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-03-16T10:54:34Z

BE UT Coverage Report

Increment line coverage 74.13% (1255/1693) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.93% (19292/36451)
Line Coverage	36.25% (180200/497120)
Region Coverage	32.79% (139418/425214)
Branch Coverage	33.82% (60656/179338)

eldenmoon

LGTM

csun5285 requested a review from yiguolei as a code owner March 12, 2026 11:22

csun5285 force-pushed the branch-4.1 branch from ab9ad7d to 705cf19Compare March 12, 2026 11:25

csun5285 force-pushed the branch-4.1 branch from 705cf19 to 13cd670Compare March 14, 2026 13:44

yiguolei force-pushed the branch-4.1 branch from c123b0d to 0740c34Compare March 16, 2026 04:45

csun5285and others added 7 commits March 16, 2026 14:14

[feature](variant) add variant doc mode (apache#59183)
6ad546e
**DOC encoding**: materialize all subcolumns, and additionally keep the original JSON as a “stored field” so queries can quickly return the entire JSON document. doc: apache/doris-website#3253

[regression-test](variant) disable doc mode in test_variant_count_on_…
6204b31
…index_fault_injection (apache#60518)

[opt](variant) unify doc writer pipeline and index writes (apache#60668)
d239c70

format
05e0a01

csun5285 force-pushed the branch-4.1 branch from 6ed3132 to 4a47ec7Compare March 16, 2026 06:51

eldenmoon approved these changes Mar 16, 2026
View reviewed changes

eldenmoon merged commit 8f280b3 into apache:branch-4.1Mar 16, 2026
22 of 25 checks passed

Uh oh!

Conversation

csun5285 commented Mar 12, 2026

Uh oh!

hello-stephen commented Mar 12, 2026

Uh oh!

csun5285 commented Mar 12, 2026

Uh oh!

csun5285 commented Mar 12, 2026

Uh oh!

hello-stephen commented Mar 12, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 12, 2026

FE UT Coverage Report

Uh oh!

csun5285 commented Mar 14, 2026

Uh oh!

doris-robot commented Mar 14, 2026

Cloud UT Coverage Report

Uh oh!

csun5285 commented Mar 15, 2026

Uh oh!

doris-robot commented Mar 15, 2026

Cloud UT Coverage Report

Uh oh!

csun5285 commented Mar 16, 2026

Uh oh!

doris-robot commented Mar 16, 2026

Cloud UT Coverage Report

Uh oh!

csun5285 commented Mar 16, 2026

Uh oh!

hello-stephen commented Mar 16, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 16, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Mar 16, 2026

BE UT Coverage Report

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants