[improvement](be) Use plain encoding for V3 floating-point columns by bobhan1 · Pull Request #62649 · apache/doris

bobhan1 · 2026-04-20T14:05:54Z

What problem does this PR solve?

Issue Number: None

Related PR: #63622

Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation added a tablet-schema flag and propagated it through proto, FE, Cloud, and BE writers. After #63622 split encoding selection into explicit V2 and V3 maps and persisted storage_format in TabletSchema, that propagation chain is obsolete. This PR expresses the behavior directly in the V3 default map while preserving the V2 default for compatibility.

Changes:

Use PLAIN_ENCODING as the V3 default for FLOAT and DOUBLE.
Keep the V2 default as BIT_SHUFFLE.
Update the authoritative encoding-map unit tests to lock both behaviors.
Regenerate all ten affected SegmentFlusher golden files across the two scalar and four complex Variant V3 cases.

Release note

Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged.

Check List (For Author)

Test
- Regression test
- Unit Test
  - ./run-be-ut.sh --run --filter='SegmentFlusherFormatTest.*KeepTheirSegmentBytes:SegmentFlusherTransformFormatTest.PartialUpdateAndRowBinlogPathsKeepTheirSegmentBytes' -j100 (6 tests passed in two generation runs; all 164 generated Segment files were byte-identical between runs)
  - ./run-be-ut.sh --run --filter='EncodingInfoTest.*:SegmentFlusherFormatTest.*:SegmentFlusherTransformFormatTest.*' -j100 (17 tests passed against the checked-in goldens)
- Manual test
  - build-support/clang-format.sh
  - build-support/check-format.sh
  - git diff --check
- No need to test or manual test.
Behavior changed:
- No.
- Yes. Newly written Segment V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; Segment V2 remains unchanged.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-04-20T14:06:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2026-04-21T02:07:35Z

run buildall

hello-stephen · 2026-04-21T02:51:03Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.06% (1843/2361)
Line Coverage	64.76% (32994/50947)
Region Coverage	65.24% (16365/25085)
Branch Coverage	55.81% (8736/15652)

hello-stephen · 2026-04-21T03:40:39Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

bobhan1 · 2026-04-23T15:02:42Z

run external

bobhan1 · 2026-04-23T15:02:47Z

run feut

bobhan1 · 2026-04-23T15:03:42Z

run p0

hello-stephen · 2026-04-23T16:37:31Z

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

bobhan1 · 2026-04-24T02:21:36Z

run external

bobhan1 · 2026-04-24T08:55:21Z

run buildall

hello-stephen · 2026-04-24T09:25:34Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.06% (1843/2361)
Line Coverage	64.78% (33006/50947)
Region Coverage	65.31% (16383/25085)
Branch Coverage	55.88% (8746/15652)

hello-stephen · 2026-04-24T12:29:28Z

FE Regression Coverage Report

Increment line coverage 1.01% (1/99) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-24T12:34:03Z

BE Regression && UT Coverage Report

Increment line coverage 85.71% (36/42) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.52% (26817/37498)
Line Coverage	53.87% (280093/519904)
Region Coverage	47.20% (215247/455984)
Branch Coverage	50.52% (97450/192897)

bobhan1 · 2026-08-17T11:08:51Z

/review

bobhan1 · 2026-08-17T11:08:56Z

run buildall

github-actions

Review opinion: changes requested

One blocking test-integration issue remains: the V3 FLOAT/DOUBLE default switch leaves checked-in SegmentFlusher golden segments stale. See the inline comment for the affected scope and regeneration requirement.

Critical checkpoint conclusions:

Goal and scope: The production change correctly selects PLAIN for V3 FLOAT/DOUBLE and retains BIT_SHUFFLE for V2. The code and focused map tests are small and clear, but the required golden artifacts are incomplete.
Data correctness and compatibility: Each writer persists a concrete ColumnMetaPB.encoding, and readers dispatch from that stored value rather than recomputing a default. Both FLOAT/DOUBLE PLAIN and BIT_SHUFFLE codecs already exist, so old BIT_SHUFFLE segments and new PLAIN segments can coexist across rolling upgrades.
Parallel paths: Horizontal, vertical, nested, Variant, load, compaction, and schema-change paths preserve the owning tablet storage format and route encoding selection through the same resolver. No missed production path remains.
Concurrency, lifecycle, and configuration: The resolver maps are immutable after initialization. This PR adds no lock, thread, lifecycle, static-initialization, dynamic-config, or new FE/BE variable concern.
Persistence and writes: Only newly written V3 page selection changes; transactionality, atomicity, tablet visibility, and metadata read/write protocols are otherwise unchanged.
Tests and results: The changed unit tests directly lock the V2/V3 map entries, and the regression performs V2/V3 write/read smoke coverage. However, normal SegmentFlusherFormatTest execution byte-compares against stale checked-in V3 fixtures, so six cases fail and ten .dat blobs need regeneration. The author's reported BE-UT filter covered only EncodingInfoTest.*; no builds or tests were run in this review-only runner.
Observability and protocols: No new protocol symbol, storage enum, log, metric, or observability requirement is introduced.
Performance: The change removes BIT_SHUFFLE work from new V3 FLOAT/DOUBLE pages without adding per-row work or allocations; no independent performance benchmark was run here.
User focus: No additional user-provided focus was supplied.

Review completion: the review converged after two full-coverage and risk-focused rounds. One finding is accepted inline; the weaker regression-result-oracle candidate was dismissed with concrete code evidence.

github-actions · 2026-08-17T11:48:29Z

 _set_v3_default<FieldType::OLAP_FIELD_TYPE_UNSIGNED_INT, BIT_SHUFFLE>();
- _set_v3_default<FieldType::OLAP_FIELD_TYPE_FLOAT, BIT_SHUFFLE>();
- _set_v3_default<FieldType::OLAP_FIELD_TYPE_DOUBLE, BIT_SHUFFLE>();
+ _set_v3_default<FieldType::OLAP_FIELD_TYPE_FLOAT, PLAIN_ENCODING>();


Please regenerate the affected V3 SegmentFlusher goldens with this default change. Normal SegmentFlusherFormatTest execution byte-compares generated segments with the checked-in files, and this PR updates none of them. Besides the two all_scalar_values_v3_* cases, four complex_v3_* cases materialize nested.y:3.5 as a DOUBLE subcolumn, so six case directories fail and ten .dat blobs differ. Those fixtures were generated while V3 FLOAT/DOUBLE used BIT_SHUFFLE; this line changes their stored ColumnMetaPB.encoding and page bytes to PLAIN. Please regenerate all six affected case directories and run the full SegmentFlusherFormatTest suite.

bobhan1 · 2026-08-18T02:12:07Z

run buildall

bobhan1 · 2026-08-18T02:12:16Z

/review

github-actions

Review opinion: changes requested

No new inline finding survived review on the current head. The existing blocking thread remains fully applicable: #62649 (comment). The authoritative changed-file list still contains no regenerated SegmentFlusher fixtures, so six V3 case directories and ten .dat blobs remain stale.

Critical checkpoint conclusions:

Goal and scope: The production change is small and correctly selects PLAIN for V3 FLOAT/DOUBLE while preserving V2 BIT_SHUFFLE. The focused map tests lock that split, but the checked-in physical artifacts required by the normal test suite are incomplete.
Data correctness and compatibility: Writers persist a concrete ColumnMetaPB.encoding, and readers dispatch from that stored value rather than recomputing a default. Both FLOAT/DOUBLE PLAIN and BIT_SHUFFLE codecs predate this change, so historical V3 BIT_SHUFFLE segments and new V3 PLAIN segments remain readable across rolling upgrades.
Parallel paths: Horizontal, vertical, nested ARRAY/MAP/STRUCT, Variant materialization/compaction, direct load, rowset compaction, and schema-change writers preserve the owning tablet storage format and route metadata through the same resolver. No missed production path remains.
Concurrency, lifecycle, and configuration: The resolver maps are immutable after initialization. This PR adds no thread, lock, lock-order, static-initialization, lifecycle, dynamic-configuration, or FE/BE variable concern.
Persistence and write semantics: Only the encoding chosen for newly written V3 FLOAT/DOUBLE pages changes. Transactionality, atomicity, tablet-version visibility, EditLog behavior, and crash recovery protocols are otherwise untouched.
Tests and results: The unit tests directly pin V3 PLAIN and V2 BIT_SHUFFLE, existing page tests cover both codecs, and the regression executes V2/V3 FLOAT/DOUBLE insert/read paths. However, normal SegmentFlusherFormatTest execution byte-compares against the unchanged V3 fixtures identified in the existing thread. No builds or tests were run in this review-only runner.
Observability and protocols: No new protocol symbol, storage enum, log, metric, or additional observability requirement is introduced.
Performance: The change removes BIT_SHUFFLE work from new V3 FLOAT/DOUBLE pages without adding per-row work or allocations; no independent performance benchmark was run here.
User focus: No additional user-provided focus was supplied.
Review completion: Round 1 converged. Both normal full-review subagents and the separate risk-focused subagent returned NO_NEW_VALUABLE_FINDINGS; every risk was independently resolved, the known golden issue was deduplicated against the existing thread, and the final comment set contains zero new inline comments.

hello-stephen · 2026-08-18T04:34:32Z

TPC-H: Total hot run time: 17161 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a4a7cdaf226ebcbab69073022ff1451170016927, data reload: false
------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	17583	3030	2997	2997
q2	q3	10878	867	507	507
q4	4682	271	203	203
q5	7651	583	396	396
q6	137	120	94	94
q7	535	495	390	390
q8	9269	947	865	865
q9	3471	2406	2373	2373
q10	6496	865	720	720
q11	466	270	256	256
q12	694	396	338	338
q13	17878	1526	1170	1170
q14	164	153	141	141
q15	q16	490	395	377	377
q17	845	778	792	778
q18	3236	2301	2286	2286
q19	1125	837	776	776
q20	669	517	503	503
q21	5274	1759	1933	1759
q22	330	272	232	232
Total cold run time: 91873 ms
Total hot run time: 17161 ms
----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL ============================================
q1	3368	3297	3297	3297
q2	q3	2243	2416	2212	2212
q4	1223	1226	902	902
q5	2252	2170	2127	2127
q6	168	117	92	92
q7	1022	894	868	868
q8	1605	1384	1399	1384
q9	3121	3097	3073	3073
q10	1880	1812	1620	1620
q11	358	272	256	256
q12	458	426	331	331
q13	1511	1518	1142	1142
q14	169	183	163	163
q15	q16	405	392	353	353
q17	1032	1017	1004	1004
q18	5118	4398	4700	4398
q19	855	826	830	826
q20	959	917	796	796
q21	3474	3237	3216	3216
q22	391	337	335	335
Total cold run time: 31612 ms
Total hot run time: 28395 ms

hello-stephen · 2026-08-18T04:41:04Z

TPC-DS: Total hot run time: 80785 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a4a7cdaf226ebcbab69073022ff1451170016927, data reload: false
query5	4297	422	350	350
query6	420	166	148	148
query7	4840	440	271	271
query8	300	126	121	121
query9	8728	2884	2888	2884
query10	398	257	242	242
query11	5382	1045	919	919
query12	123	71	70	70
query13	1181	431	338	338
query14	6168	2213	2099	2099
query14_1	1999	1975	1958	1958
query15	176	117	113	113
query16	946	381	351	351
query17	789	437	356	356
query18	2333	318	226	226
query19	173	135	107	107
query20	69	67	69	67
query21	218	112	97	97
query22	5439	5344	5476	5344
query23	6709	6560	6133	6133
query23_1	6158	6041	6212	6041
query24	7326	1062	768	768
query24_1	793	805	770	770
query25	438	277	241	241
query26	1266	254	162	162
query27	2727	451	291	291
query28	4614	1513	1496	1496
query29	926	421	362	362
query30	283	178	154	154
query31	856	422	355	355
query32	107	52	51	51
query33	472	223	181	181
query34	1008	802	511	511
query35	421	400	336	336
query36	568	553	515	515
query37	138	83	71	71
query38	1027	831	803	803
query39	477	518	488	488
query39_1	501	468	465	465
query40	228	122	116	116
query41	60	56	55	55
query42	83	78	79	78
query43	245	240	214	214
query44 query45	109	107	99	99
query46	788	911	562	562
query47	751	747	697	697
query48	311	321	228	228
query49	573	252	191	191
query50	855	321	279	279
query51	7917	7858	8018	7858
query52	76	76	67	67
query53	209	224	170	170
query54	262	198	187	187
query55	107	77	59	59
query56	249	247	276	247
query57	682	668	669	668
query58	242	195	187	187
query59	1211	1219	1081	1081
query60	250	200	210	200
query61	127	125	128	125
query62	384	208	178	178
query63	185	153	158	153
query64	2787	685	626	626
query65 query66	1899	321	247	247
query67	10096	9884	9858	9858
query68 query69	375	223	201	201
query70	640	625	613	613
query71	293	268	244	244
query72	2400	1746	1653	1653
query73	721	597	347	347
query74	1937	1263	1161	1161
query75	1272	1147	1009	1009
query76	2488	737	559	559
query77	284	267	220	220
query78	3909	3573	3229	3229
query79	3251	803	579	579
query80	1568	399	333	333
query81	510	199	173	173
query82	685	125	103	103
query83	337	251	231	231
query84 query85	860	423	391	391
query86	495	169	170	169
query87	978	947	883	883
query88	4454	2155	2146	2146
query89	311	226	203	203
query90	2102	142	150	142
query91	153	138	121	121
query92	70	45	41	41
query93	3033	1104	831	831
query94	699	269	232	232
query95	635	436	354	354
query96	844	574	288	288
query97	1064	1026	972	972
query98	185	133	140	133
query99	473	345	318	318
Total cold run time: 177914 ms
Total hot run time: 80785 ms

bobhan1 · 2026-08-18T10:58:11Z

/review

bobhan1 · 2026-08-18T10:58:16Z

run buildall

bobhan1 · 2026-08-18T11:36:33Z

run beut

github-actions

Review opinion: changes requested

The current head fixes the scalar portion of the existing golden-fixture thread, but the thread remains blocking: #62649 (comment). All four V3 complex cases still have stale checked-in output. Specifically, all four complex segment_0.dat files and both non-nullable complex segment_1.dat files still encode the materialized nested.y:3.5 DOUBLE subcolumn with BIT_SHUFFLE, while the current writer selects PLAIN. Those six blobs are absent from the changed-file list, so the full SegmentFlusherFormatTest producer set is still inconsistent. No new inline comment is added because the existing thread already covers the mechanism, four-case scope, and required full regeneration.

Critical checkpoint conclusions:

Goal and scope: The production change is small and correctly selects PLAIN for V3 FLOAT/DOUBLE while retaining BIT_SHUFFLE for V2. The direct map tests and regenerated scalar metadata prove that split, but the required complex golden artifacts remain incomplete.
Data correctness and write semantics: Writers persist a concrete ColumnMetaPB.encoding, and readers dispatch from that stored value. FLOAT/DOUBLE PLAIN and BIT_SHUFFLE codecs already coexist, and active V3 typed-DOUBLE regression coverage verifies exact values through the PLAIN Segment reader. Transactionality, atomicity, tablet visibility, crash recovery, and delete-bitmap behavior are otherwise untouched.
Compatibility and parallel paths: Horizontal, vertical, nullable, nested ARRAY/MAP/STRUCT, Variant materialization, load, compaction, and schema-change paths preserve the owning tablet storage format and use the common resolver. Historical BIT_SHUFFLE segments and new PLAIN segments remain readable across rolling upgrades; no parallel production path was missed.
Concurrency, lifecycle, configuration, and special conditions: The resolver maps are immutable after initialization. This PR adds no thread, lock, lock-order, lifetime, static-initialization, dynamic-config, conditional-guard, or memory-ownership concern.
Persistence, protocols, and cross-process variables: The existing Segment encoding enum is persisted per column; no EditLog path, FE/BE variable, function symbol, storage-format enum, RPC field, or mixed-version protocol contract is added.
Tests and results: The focused resolver tests lock V2 and V3 defaults, and the four regenerated scalar blobs cover horizontal/vertical plus nullable/non-nullable V3 writes. The author's reported SegmentFlusher filter exercised only the scalar producer, so it could not catch the six omitted complex blobs. The changed regression executes V2/V3 read smoke coverage; its discarded results are not a distinct defect because an active V3 typed-DOUBLE regression supplies an exact PLAIN-reader oracle through the same templated fixed-width path. No builds or tests were run in this review-only runner.
Observability: Existing writer VLOG output records the concrete encoding. No new log, metric, or tracing surface is needed for this two-entry policy change.
Performance: Selecting PLAIN removes BIT_SHUFFLE work for newly written V3 FLOAT/DOUBLE pages and adds no per-row branch, scan, or allocation. No independent performance benchmark was run here.
User focus: No additional user-provided review focus was supplied.
Review completion: The review converged after two full-coverage and risk-focused rounds. All Round 2 subagents returned NO_NEW_VALUABLE_FINDINGS; every candidate was independently verified and either dismissed with code/test evidence or deduplicated against the existing inline thread. The final comment set intentionally contains zero new inline comments.

hello-stephen · 2026-08-18T12:03:30Z

TPC-H: Total hot run time: 17250 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e92feeda1c80383d703b704ffea3727e7ca73270, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17583	3010	3009	3009
q2	1870	228	156	156
q3	10481	865	518	518
q4	4674	246	201	201
q5	7683	570	392	392
q6	139	118	94	94
q7	553	524	398	398
q8	9235	971	933	933
q9	3485	2388	2404	2388
q10	6498	856	702	702
q11	462	259	241	241
q12	683	395	350	350
q13	17886	1540	1143	1143
q14	164	145	141	141
q15	q16	427	401	366	366
q17	769	733	851	733
q18	3103	2268	2250	2250
q19	1099	829	711	711
q20	607	527	477	477
q21	4834	1817	1914	1817
q22	326	269	230	230
Total cold run time: 92561 ms
Total hot run time: 17250 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3400	3374	3341	3341
q2	205	212	162	162
q3	2238	2339	2187	2187
q4	1198	1173	904	904
q5	2189	2124	2136	2124
q6	173	121	92	92
q7	1026	942	855	855
q8	1604	1426	1437	1426
q9	3130	3103	3095	3095
q10	1862	1827	1637	1637
q11	355	271	255	255
q12	451	425	343	343
q13	1489	1543	1157	1157
q14	172	174	152	152
q15	q16	390	399	357	357
q17	1055	1030	1029	1029
q18	4980	4391	4758	4391
q19	857	858	852	852
q20	948	942	826	826
q21	3525	3308	3275	3275
q22	403	368	333	333
Total cold run time: 31650 ms
Total hot run time: 28793 ms

hello-stephen · 2026-08-18T12:10:01Z

TPC-DS: Total hot run time: 82911 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e92feeda1c80383d703b704ffea3727e7ca73270, data reload: false
query5	4240	416	342	342
query6	398	182	163	163
query7	4859	446	278	278
query8	304	120	115	115
query9	8684	2903	2901	2901
query10	415	277	213	213
query11	5380	1029	927	927
query12	117	71	72	71
query13	1189	437	331	331
query14	5990	2214	2095	2095
query14_1	1985	1978	1960	1960
query15	167	118	110	110
query16	949	365	355	355
query17	783	446	362	362
query18	2323	334	245	245
query19	168	150	114	114
query20	71	71	74	71
query21	220	117	101	101
query22	5495	5290	5225	5225
query23	6699	6156	5997	5997
query23_1	6243	6100	6001	6001
query24	7284	1132	787	787
query24_1	768	774	780	774
query25	454	306	261	261
query26	1255	275	174	174
query27	2702	455	292	292
query28	4599	1500	1521	1500
query29	924	409	350	350
query30	281	184	151	151
query31	855	425	347	347
query32	93	49	51	49
query33	458	200	175	175
query34	1001	810	472	472
query35	401	401	350	350
query36	566	548	515	515
query37	119	80	68	68
query38	996	834	819	819
query39	514	501	452	452
query39_1	455	458	453	453
query40	224	129	111	111
query41	53	52	51	51
query42	83	81	80	80
query43	241	243	210	210
query44	1016	553	549	549
query45	106	103	96	96
query46	771	830	523	523
query47	746	739	710	710
query48	323	300	234	234
query49	532	232	184	184
query50	862	345	256	256
query51	8155	8175	7969	7969
query52	75	80	66	66
query53	219	222	164	164
query54	243	199	201	199
query55	81	60	57	57
query56	257	247	230	230
query57	710	653	656	653
query58	217	187	191	187
query59	1208	1246	1074	1074
query60	281	210	199	199
query61	106	113	120	113
query62	349	287	171	171
query63	189	153	154	153
query64	2729	738	568	568
query65	1597	1573	1599	1573
query66	1914	318	241	241
query67	10217	9587	9642	9587
query68	3021	1225	794	794
query69	350	220	199	199
query70	663	636	609	609
query71	301	250	247	247
query72	2323	1752	1565	1565
query73	641	604	328	328
query74	2019	1224	1127	1127
query75	1240	1163	1028	1028
query76	2371	726	560	560
query77	250	253	213	213
query78	3840	3550	3166	3166
query79	2356	813	582	582
query80	1646	401	341	341
query81	502	199	177	177
query82	642	129	101	101
query83	380	247	240	240
query84	319	124	102	102
query85	937	463	376	376
query86	387	177	167	167
query87	1032	963	890	890
query88	2835	2139	2142	2139
query89	295	229	203	203
query90	1905	149	160	149
query91	162	145	124	124
query92	50	50	45	45
query93	1518	1134	831	831
query94	641	255	229	229
query95	636	450	358	358
query96	834	550	259	259
query97	1052	1041	986	986
query98	140	130	131	130
query99	411	352	308	308
Total cold run time: 178975 ms
Total hot run time: 82911 ms

hello-stephen · 2026-08-18T12:14:23Z

ClickBench: Total hot run time: 14.36 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e92feeda1c80383d703b704ffea3727e7ca73270, data reload: false
query1	0.01	0.00	0.00
query2	0.08	0.04	0.03
query3	0.25	0.11	0.11
query4	1.62	0.10	0.12
query5	0.18	0.16	0.17
query6	1.24	0.70	0.67
query7	0.03	0.00	0.01
query8	0.05	0.02	0.02
query9	0.28	0.21	0.22
query10	0.35	0.35	0.34
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.30	0.32	0.30
query14	0.47	0.46	0.45
query15	0.36	0.34	0.34
query16	0.22	0.23	0.22
query17	0.72	0.69	0.66
query18	0.17	0.16	0.17
query19	1.17	1.05	1.15
query20	0.01	0.02	0.02
query21	15.44	0.17	0.11
query22	5.12	0.04	0.04
query23	16.20	0.26	0.10
query24	2.98	0.33	0.27
query25	0.09	0.06	0.03
query26	0.81	0.16	0.13
query27	0.03	0.03	0.03
query28	3.69	0.59	0.25
query29	12.45	3.13	2.52
query30	0.25	0.12	0.11
query31	2.75	0.37	0.17
query32	3.55	0.32	0.22
query33	1.34	1.40	1.41
query34	15.35	2.14	1.72
query35	1.73	1.73	1.70
query36	0.47	0.29	0.28
query37	0.06	0.03	0.04
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.07	0.02	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.02
Total cold run time: 90.45 s
Total hot run time: 14.36 s

### What problem does this PR solve? Issue Number: None Related PR: apache#63622 Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation required a tablet-schema flag propagated through proto, FE, Cloud, and BE writers. After apache#63622 split encoding selection into explicit V2 and V3 maps and persisted storage_format in TabletSchema, that propagation chain is obsolete. Express the behavior directly in the V3 default map, preserve the V2 BIT_SHUFFLE default for compatibility, and refresh all affected scalar and materialized Variant Segment goldens. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter='SegmentFlusherFormatTest.*KeepTheirSegmentBytes:SegmentFlusherTransformFormatTest.PartialUpdateAndRowBinlogPathsKeepTheirSegmentBytes' -j100` (6 tests passed in two generation runs; all 164 generated Segment files were byte-identical between runs) - `./run-be-ut.sh --run --filter='EncodingInfoTest.*:SegmentFlusherFormatTest.*:SegmentFlusherTransformFormatTest.*' -j100` (17 tests passed against the checked-in goldens) - `build-support/clang-format.sh` - `build-support/check-format.sh` - `git diff --check` - Behavior changed: Yes. Newly written V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; V2 remains BIT_SHUFFLE. - Does this need documentation: No

bobhan1 · 2026-08-26T07:15:13Z

run buildall

bobhan1 · 2026-08-26T07:15:19Z

/review

github-actions

Review opinion: no new blocking issue found; the current head is ready from the code-review perspective.

Critical checkpoint conclusions:

Goal and proof: the V3 default map now selects numeric PLAIN_ENCODING for FLOAT and DOUBLE, while V2 remains BIT_SHUFFLE. The focused assertions and exhaustive V2/V3 expectation tables lock both behaviors.
Scope and clarity: the source change is the minimal two-entry policy update enabled by the existing split default maps; it does not reintroduce schema flags or cross-layer propagation.
Data correctness and compatibility: every writer persists the resolved enum in ColumnMetaPB.encoding, and readers dispatch from that stored enum. Both encodings were already registered at the merge base, so old readers accept new PLAIN V3 pages and new readers accept existing BIT_SHUFFLE V3 pages during rolling upgrades.
Parallel paths: horizontal, vertical, nested complex-child, auxiliary child, and materialized Variant subcolumn writers all propagate the tablet storage format through the same resolver. The scalar and Variant fixtures exercise both writer families, nullability, Bloom/no-Bloom, and compressed/uncompressed cases.
Floating-point and index semantics: numeric PLAIN copies fixed-width bytes verbatim, preserving finite values, NaN payloads, infinities, and signed zero. Nullmaps, logical page counts, zone maps, Bloom filters, ordinals, and page pointers remain independent of the physical transform.
Concurrency, lifecycle, configuration, protocol, transaction, and persistence: no mutable runtime state, lock, ownership, dynamic config, FE/BE variable, transaction, visibility, delete-bitmap, or crash-recovery path is changed.
Error handling, memory, performance, and observability: unsupported encodings still fail through checked Status; no allocation ownership or tracking changes. PLAIN removes the bitshuffle transform and the regenerated samples are smaller. Broader performance benchmarking is a product tradeoff, but no correctness or severe performance regression was substantiated. Existing VLOG context is sufficient and no new metric is needed.
Tests and results: the current head includes exactly the ten regenerated blobs requested by the existing inline thread. Independent head/base decoding found only the intended four scalar FLOAT/DOUBLE pairs and six Variant DOUBLE pages recoded from 6 to 2; all other semantic metadata and referenced pages remain unchanged apart from expected size/offset relocation. The prior concern is therefore addressed. The PR reports deterministic two-pass generation of all 164 segments and a 17-test EncodingInfo/SegmentFlusher run; I did not rerun builds or tests because this review environment explicitly forbids builds.
User focus: no additional focus was supplied; the full PR was reviewed.

No inline comments are proposed.

dismiss

hello-stephen · 2026-08-26T08:39:17Z

TPC-H: Total hot run time: 17036 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 276cfc47bc47047868624efa70628ce6dc6158ca, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17589	3062	3026	3026
q2	2277	259	225	225
q3	10055	876	518	518
q4	4676	250	206	206
q5	7666	576	390	390
q6	145	119	97	97
q7	537	532	392	392
q8	9259	928	930	928
q9	3533	2452	2441	2441
q10	6512	868	715	715
q11	402	198	187	187
q12	620	258	196	196
q13	18138	1585	1192	1192
q14	168	160	149	149
q15	q16	459	398	371	371
q17	1402	854	718	718
q18	3184	2447	2389	2389
q19	1121	876	781	781
q20	392	283	205	205
q21	5184	1682	1854	1682
q22	331	273	228	228
Total cold run time: 93650 ms
Total hot run time: 17036 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3441	3381	3357	3357
q2	526	408	384	384
q3	2364	2531	2336	2336
q4	1261	1257	908	908
q5	2299	2224	2227	2224
q6	176	129	88	88
q7	1076	989	899	899
q8	1625	1443	1432	1432
q9	3329	3287	3285	3285
q10	2012	1942	1764	1764
q11	374	274	263	263
q12	472	442	347	347
q13	1537	1593	1214	1214
q14	187	198	168	168
q15	q16	407	423	374	374
q17	3887	3555	3473	3473
q18	5354	4901	5615	4901
q19	1776	885	877	877
q20	1061	1053	946	946
q21	3872	3141	3303	3141
q22	402	352	316	316
Total cold run time: 37438 ms
Total hot run time: 32697 ms

hello-stephen · 2026-08-26T08:45:55Z

TPC-DS: Total hot run time: 85479 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 276cfc47bc47047868624efa70628ce6dc6158ca, data reload: false
query5	4302	426	347	347
query6	410	142	125	125
query7	4895	389	247	247
query8	315	126	120	120
query9	8702	2930	2934	2930
query10	394	225	190	190
query11	5469	1079	960	960
query12	128	74	75	74
query13	1215	458	354	354
query14	6059	2314	2226	2226
query14_1	2116	2094	2099	2094
query15	175	123	119	119
query16	937	385	370	370
query17	811	465	378	378
query18	2345	338	247	247
query19	188	138	110	110
query20	80	75	73	73
query21	209	103	89	89
query22	5887	5868	5783	5783
query23	7177	6873	6598	6598
query23_1	6443	6699	6527	6527
query24	7373	1087	817	817
query24_1	800	824	802	802
query25	448	325	266	266
query26	1228	233	133	133
query27	2786	435	264	264
query28	4663	1535	1515	1515
query29	977	440	369	369
query30	269	158	133	133
query31	836	417	339	339
query32	126	83	76	76
query33	472	233	182	182
query34	1030	872	502	502
query35	416	409	362	362
query36	589	590	555	555
query37	128	85	77	77
query38	1018	903	896	896
query39	519	505	507	505
query39_1	488	472	455	455
query40	209	93	81	81
query41	59	57	57	57
query42	76	74	74	74
query43	246	244	209	209
query44	1055	572	590	572
query45	111	103	100	100
query46	774	883	530	530
query47	793	808	762	762
query48	317	331	236	236
query49	540	253	180	180
query50	757	263	199	199
query51	7985	8134	8018	8018
query52	74	72	58	58
query53	195	199	157	157
query54	226	163	176	163
query55	76	63	59	59
query56	190	177	163	163
query57	664	670	665	665
query58	213	170	158	158
query59	1176	1210	1070	1070
query60	245	203	170	170
query61	123	122	119	119
query62	363	200	186	186
query63	181	141	137	137
query64	2599	669	535	535
query65	1611	1597	1707	1597
query66	1785	276	223	223
query67	10255	10122	10281	10122
query68	2779	1223	697	697
query69	352	220	196	196
query70	684	597	604	597
query71	250	176	169	169
query72	2370	1736	1564	1564
query73	699	579	346	346
query74	1572	1235	1167	1167
query75	1174	1141	1006	1006
query76	2310	749	547	547
query77	256	286	216	216
query78	4054	4077	3520	3520
query79	2138	916	579	579
query80	1575	346	281	281
query81	493	156	135	135
query82	654	123	96	96
query83	278	209	200	200
query84	300	109	93	93
query85	804	356	300	300
query86	428	182	172	172
query87	1023	975	896	896
query88	2825	2156	2118	2118
query89	296	196	177	177
query90	1984	130	130	130
query91	134	121	103	103
query92	85	71	73	71
query93	1475	1074	681	681
query94	668	256	225	225
query95	535	257	229	229
query96	789	617	277	277
query97	1054	1088	1044	1044
query98	154	141	134	134
query99	448	365	320	320
Total cold run time: 179430 ms
Total hot run time: 85479 ms

hello-stephen · 2026-08-26T08:50:18Z

ClickBench: Total hot run time: 14.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 276cfc47bc47047868624efa70628ce6dc6158ca, data reload: false
query1	0.01	0.00	0.00
query2	0.08	0.03	0.03
query3	0.24	0.10	0.11
query4	1.61	0.11	0.10
query5	0.18	0.16	0.16
query6	1.22	0.70	0.66
query7	0.04	0.01	0.00
query8	0.04	0.03	0.02
query9	0.30	0.23	0.23
query10	0.35	0.35	0.36
query11	0.17	0.11	0.13
query12	0.15	0.12	0.12
query13	0.30	0.31	0.31
query14	0.48	0.47	0.48
query15	0.38	0.34	0.35
query16	0.22	0.24	0.22
query17	0.70	0.73	0.73
query18	0.19	0.17	0.17
query19	1.15	1.21	1.24
query20	0.01	0.01	0.01
query21	15.44	0.15	0.11
query22	5.08	0.05	0.04
query23	16.21	0.24	0.10
query24	3.30	0.33	0.27
query25	0.12	0.04	0.03
query26	0.72	0.17	0.13
query27	0.05	0.03	0.04
query28	3.62	0.55	0.28
query29	12.48	3.11	2.57
query30	0.26	0.12	0.12
query31	2.76	0.39	0.19
query32	3.50	0.32	0.22
query33	1.38	1.38	1.44
query34	15.37	2.19	1.75
query35	1.78	1.80	1.73
query36	0.49	0.31	0.30
query37	0.07	0.04	0.04
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.12	0.09	0.08
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 90.81 s
Total hot run time: 14.83 s

hello-stephen · 2026-08-26T09:50:58Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.51% (29187/46688)
Line Coverage	47.54% (305465/642530)
Region Coverage	43.20% (246682/570969)
Branch Coverage	44.78% (114875/256529)

hello-stephen · 2026-08-26T11:07:06Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.94% (34339/45221)
Line Coverage	60.89% (386790/635185)
Region Coverage	57.04% (324447/568805)
Branch Coverage	57.85% (147991/255804)

bobhan1 requested review from dataroaring, gavinchou and yiguolei as code owners April 20, 2026 14:05

bobhan1 changed the title ~~[Enhancement] Segment V3: disable bitshuffle encoding for float/double types~~[Enhancement](storage) Segment V3: disable bitshuffle encoding for float/double typesApr 20, 2026

bobhan1 requested a review from w41ter as a code owner April 20, 2026 14:30

bobhan1 mentioned this pull request Apr 21, 2026
[Feature](encoding) Add float_type_default_use_plain_encoding table property #62085
Closed
16 tasks

bobhan1 force-pushed the segment-v3-control-double branch from a6f0314 to 5275234Compare April 24, 2026 08:55

bobhan1 force-pushed the segment-v3-control-double branch from 5275234 to 5394bd3Compare August 14, 2026 09:17

bobhan1 requested review from airborne12, csun5285, eldenmoon and liaoxin01 as code owners August 14, 2026 09:17

bobhan1 changed the title ~~[Enhancement](storage) Segment V3: disable bitshuffle encoding for float/double types~~[improvement](be) Use plain encoding for V3 floating-point columnsAug 14, 2026

github-actionsBot requested changes Aug 17, 2026
View reviewed changes

bobhan1 force-pushed the segment-v3-control-double branch from 5394bd3 to a4a7cdaCompare August 18, 2026 02:11

github-actionsBot requested changes Aug 18, 2026
View reviewed changes

bobhan1 force-pushed the segment-v3-control-double branch from a4a7cda to e92feedCompare August 18, 2026 09:12

github-actionsBot previously requested changes Aug 18, 2026
View reviewed changes

bobhan1 force-pushed the segment-v3-control-double branch from e92feed to 24943ccCompare August 26, 2026 07:10

bobhan1 force-pushed the segment-v3-control-double branch from 24943cc to 276cfc4Compare August 26, 2026 07:13

github-actionsBot reviewed Aug 26, 2026
View reviewed changes

csun5285 approved these changes Aug 26, 2026
View reviewed changes

gavinchou approved these changes Aug 27, 2026
View reviewed changes

gavinchou merged commit 3029e6e into apache:masterAug 27, 2026
33 checks passed

Uh oh!

Conversation

bobhan1 commented Apr 20, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 20, 2026

Uh oh!

bobhan1 commented Apr 21, 2026

Uh oh!

hello-stephen commented Apr 21, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 21, 2026

FE UT Coverage Report

Uh oh!

bobhan1 commented Apr 23, 2026

Uh oh!

bobhan1 commented Apr 23, 2026

Uh oh!

bobhan1 commented Apr 23, 2026

Uh oh!

hello-stephen commented Apr 23, 2026

FE UT Coverage Report

Uh oh!

bobhan1 commented Apr 24, 2026

Uh oh!

bobhan1 commented Apr 24, 2026

Uh oh!

hello-stephen commented Apr 24, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 24, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Apr 24, 2026

BE Regression && UT Coverage Report

Uh oh!

bobhan1 commented Aug 17, 2026

Uh oh!

bobhan1 commented Aug 17, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotAug 17, 2026

Choose a reason for hiding this comment

Uh oh!

bobhan1 commented Aug 18, 2026

Uh oh!

bobhan1 commented Aug 18, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

bobhan1 commented Aug 18, 2026

Uh oh!

bobhan1 commented Aug 18, 2026

Uh oh!

bobhan1 commented Aug 18, 2026

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

hello-stephen commented Aug 18, 2026

Uh oh!

bobhan1 commented Aug 26, 2026

Uh oh!

bobhan1 commented Aug 26, 2026

bobhan1 commented Apr 20, 2026•
edited
Loading