Uh oh!
There was an error while loading. Please reload this page.
[refactor](BE) split EncodingInfo defaults into 4 explicit maps - #63622
Merged
Conversation
hello-stephen
commented
May 25, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
csun5285
commented
May 25, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
May 25, 2026
Contributor
TPC-H: Total hot run time: 32287 ms |
hello-stephen
commented
May 25, 2026
Contributor
TPC-DS: Total hot run time: 173107 ms |
csun5285
commented
May 25, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
May 25, 2026
Contributor
TPC-H: Total hot run time: 31498 ms |
hello-stephen
commented
May 25, 2026
Contributor
TPC-DS: Total hot run time: 173081 ms |
hello-stephen
commented
May 25, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
csun5285
commented
May 26, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
May 26, 2026
Contributor
TPC-H: Total hot run time: 31135 ms |
hello-stephen
commented
May 26, 2026
Contributor
TPC-DS: Total hot run time: 76522 ms |
csun5285force-pushed
the
dev/encoding-info-refactor
branch
from
May 27, 2026 01:09
a91a4b5 to
b5923d7Comparecsun5285
commented
May 27, 2026
ContributorAuthor
run buildall |
hello-stephen
commented
May 27, 2026
Contributor
TPC-H: Total hot run time: 32014 ms |
hello-stephen
commented
May 27, 2026
Contributor
TPC-DS: Total hot run time: 173506 ms |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
yiguolei
commented
May 27, 2026
Contributor
/review |
Contributor
There was a problem hiding this comment.
Review result: request changes.
Critical checkpoint conclusions:
- Goal/test: the PR centralizes tablet storage format and encoding defaults for V2/V3 and adds/updates BE unit coverage, but one compatibility path is not proven safe.
- Scope/clarity: the refactor is mostly focused around segment encoding defaults and tablet schema persistence.
- Concurrency/lifecycle: no new shared mutable runtime state or locking path was found in the changed write/read paths; the static resolver remains the existing process-lifetime singleton pattern.
- Configuration/dynamic behavior: no new dynamic config item is introduced.
- Compatibility/storage format: blocking issue found. V3 tablet schema persistence now relies only on the newly added protobuf field in the non-cloud BE create path, which old BEs ignore during rolling upgrade/downgrade scenarios.
- Parallel paths: cloud FE creation writes both the new field and old flags, but the BE non-cloud creation path no longer does; these paths are inconsistent.
- Tests: tests cover the new resolver behavior, but I did not see coverage for mixed-version/legacy-proto compatibility of newly created V3 tablet schemas.
- Observability/transactions/data writes: no additional observability issue found; the concern is schema metadata compatibility before any data-page correctness can be relied on.
- User focus: no additional user-provided review focus was supplied.
Please keep the legacy V3 flags populated together with storage_format, or otherwise document/prove that old BEs cannot consume these tablet schema PBs during supported upgrade/rollback flows.
Uh oh!
There was an error while loading. Please reload this page.
hello-stephen
commented
May 28, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
bobhan1
approved these changes
May 28, 2026
gavinchou
approved these changes
May 28, 2026
yiguolei
approved these changes
May 29, 2026
yiguolei
commented
May 29, 2026
Contributor
/review |
Contributor
There was a problem hiding this comment.
Automated review summary for PR 63622 at 8eff36a.
No additional blocking issues were found beyond the already-existing inline review threads. I did not duplicate those known comments.
Critical checkpoint conclusions:
- Goal and tests: The PR centralizes tablet storage-format-driven encoding behavior and persists storage_format in tablet schema metadata. The modified BE unit/regression tests cover schema metadata conversion and encoding selection paths.
- Scope/focus: The change is focused on storage format metadata, default encoding resolution, and related writer/reader plumbing.
- Concurrency/lifecycle: No new shared mutable concurrency path or special lifecycle hazard was identified in the reviewed diff.
- Configuration: No new dynamic config item is added; the behavior is driven by persisted tablet storage format.
- Compatibility: The new storage_format field is converted in cloud/non-cloud schema PB paths, with legacy V3-flavor fields retained on V3 schema serialization for downgrade compatibility. Existing inline compatibility discussion was treated as known context.
- Parallel paths: SegmentWriter, VerticalSegmentWriter, complex child writers, variant writers, indexed columns, and cloud PB conversion paths were checked for propagation.
- Error handling/memory: No ignored Status or untracked large allocation issue was found in the changed code paths.
- Data correctness: Encoding selection is resolved before writer init, and read-side PageIO predecode paths were checked for PLAIN_ENCODING_V2/dictionary handling.
- Test results: I did not run builds or tests in this review pass.
- Observability/performance: No required additional logging/metrics or obvious hot-path regression was identified.
User focus: No additional user-provided review focus was present.
Uh oh!
There was an error while loading. Please reload this page.
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 1, 2026
…ta + lengths trailer V3 layout: |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)| Compared to V2 (length and data interleaved per entry), V3 lets the pre-decoder memcpy the entire binary payload in a single shot and walk the contiguous varuint length block once to fill the V1 offsets array, with no data-pointer-vs-length-pointer dependency between the two passes. This commit introduces PLAIN_ENCODING_V3 as a registered, available binary-plain encoding; it does not change any default encoding. The on-disk format is a new EncodingTypePB (= 9) and BinaryPlainEncodingTypePB (= 3), so existing V1/V2 segments keep dispatching to their own pre-decoders. Wired into the (post-apache#63622) EncodingInfoResolver: - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> for the CppType and Slice specializations, _register_supported_encoding<..., PLAIN_ENCODING_V3>() for every binary type (CHAR/VARCHAR/STRING/JSONB/VARIANT/HLL/BITMAP/ QUANTILE_STATE/AGG_STATE), and a BinaryPlainPageV3PreDecoder hook in the EncodingInfo constructor (Slice types only). - gensrc protos: new EncodingTypePB and BinaryPlainEncodingTypePB enum values. Note: the original change also flipped several V3 default encodings and the row store column to V3 and edited segment_writer/vertical_segment_writer/ binary_dict_page/tablet_meta. Those paths were rewritten upstream by apache#63622 (EncodingPreference -> explicit default maps), which now owns the V3 defaults; this rebase keeps those defaults unchanged and leaves PLAIN_ENCODING_V3 opt-in. Tests: 15 BinaryPlainPageV3Test cases covering encode/decode roundtrip, seek, read_by_rowids, empty page, page_full, large N, mixed lengths (including unicode), reset, varint length boundaries (127/128/16383/16384 byte values across 1/2/3-byte varint bands), raw trailer layout assertions, and two corruption-rejection cases. Benchmark (15-rep x 2s median, V3 / V2 speedup at 256 KiB page): 8B: 3.56x 16B: 3.07x 32B: 2.46x 64B: 2.63x 128B: 2.25x 256B: 1.39x 512B: 1.22x 1024B: 1.11x 4096B: 1.01x V3 strictly does not lose to V2 across the tested grid. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 1, 2026
… segments Introduce PLAIN_ENCODING_V3, a binary plain page whose on-disk layout is |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|. Unlike V2 (length and data interleaved per entry), the contiguous layout lets the pre-decoder memcpy the whole data block in one shot and walk the varint lengths once to fill the V1 offsets array, with no data-vs-length pointer dependency between the two passes. V3 stores exactly the same bytes as V1/V2 — only the layout differs. In particular CHAR keeps its trailing '\0' padding on disk (written by OlapColumnDataConvertorChar), VARCHAR does not; the CHAR padding is stripped on read by BinaryPlainPageV3PreDecoder <true>, mirroring PLAIN_ENCODING_V2. Write/read wiring (post-apache#63622 EncodingInfoResolver): - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> (CppType + Slice), _register_supported_encoding for every binary type, and the IS_CHAR-aware V3 pre-decoder dispatch in the EncodingInfo constructor. - binary_plain_page_v3.h / binary_plain_page_v3_pre_decoder.h: builder and the template<bool IS_CHAR> pre-decoder (CHAR strnlen-strips on read; non-CHAR takes the single-memcpy fast path). - gensrc protos: new EncodingTypePB (=9) and BinaryPlainEncodingTypePB (=3). V3 segment defaults — every binary-plain default now uses the V3 layout: - CHAR/VARCHAR/STRING/JSONB/VARIANT keep DICT_ENCODING, but the dictionary word page (and the dict-overflow fallback plain page) use V3: ScalarColumnWriter::init sets dict_binary_plain_encoding to V3, binary_dict_page.cpp maps it and dispatches the V3 inner encoding, and binary_dict_page_pre_decoder.h rewrites the inner V3 page to V1. CHAR dictionary words are unpadded on read via the (CHAR, V3) IS_CHAR pre-decoder. - HLL/BITMAP/QUANTILE_STATE/AGG_STATE -> PLAIN_ENCODING_V3 (_set_v3_default). - the hidden row store column -> PLAIN_ENCODING_V3 (resolve_default_encoding). Benchmarks (be/benchmark/benchmark_binary_plain_page_v2.hpp): - V2 vs V3 pre-decode speed at fixed page sizes; V3 is faster for short/medium values (~2.4-2.9x at <=64B) and never slower (1.0x at 4KB). - V2 vs V3 on-disk size after ZSTD compression; same bytes, the V3 layout compresses slightly better (smaller for short values). - benchmark_main.cpp uses a custom main (DataPage allocation needs a ThreadContext + mem tracker) and re-enables binary_cast_benchmark (it was wrongly disabled; the real issue was include order, fixed by keeping our hpp last). Tests: BinaryPlainPageV3Test (encode/decode/seek/read_by_rowids/empty/page_full/varint boundaries/corruption, plus padded-CHAR round-trips that exercise the IS_CHAR read-side strip), BinaryDictPageTest V3 round-trips (word page + dict-overflow fallback), EncodingInfoTest defaults + pre-decoder dispatch, and column_meta_accessor / variant / encoding_info expectation updates. All pass under ASAN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 1, 2026
… segments Introduce PLAIN_ENCODING_V3, a binary plain page whose on-disk layout is |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|. Unlike V2 (length and data interleaved per entry), the contiguous layout lets the pre-decoder memcpy the whole data block in one shot and walk the varint lengths once to fill the V1 offsets array, with no data-vs-length pointer dependency between the two passes. V3 stores exactly the same bytes as V1/V2 — only the layout differs. In particular CHAR keeps its trailing '\0' padding on disk (written by OlapColumnDataConvertorChar), VARCHAR does not; the CHAR padding is stripped on read by BinaryPlainPageV3PreDecoder <true>, mirroring PLAIN_ENCODING_V2. Write/read wiring (post-apache#63622 EncodingInfoResolver): - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> (CppType + Slice), _register_supported_encoding for every binary type, and the IS_CHAR-aware V3 pre-decoder dispatch in the EncodingInfo constructor. - binary_plain_page_v3.h / binary_plain_page_v3_pre_decoder.h: builder and the template<bool IS_CHAR> pre-decoder (CHAR strnlen-strips on read; non-CHAR takes the single-memcpy fast path). - gensrc protos: new EncodingTypePB (=9) and BinaryPlainEncodingTypePB (=3). V3 segment defaults — every binary-plain default now uses the V3 layout: - CHAR/VARCHAR/STRING/JSONB/VARIANT keep DICT_ENCODING, but the dictionary word page (and the dict-overflow fallback plain page) use V3: ScalarColumnWriter::init sets dict_binary_plain_encoding to V3, binary_dict_page.cpp maps it and dispatches the V3 inner encoding, and binary_dict_page_pre_decoder.h rewrites the inner V3 page to V1. CHAR dictionary words are unpadded on read via the (CHAR, V3) IS_CHAR pre-decoder. - HLL/BITMAP/QUANTILE_STATE/AGG_STATE -> PLAIN_ENCODING_V3 (_set_v3_default). - the hidden row store column -> PLAIN_ENCODING_V3 (resolve_default_encoding). Benchmarks (be/benchmark/benchmark_binary_plain_page_v2.hpp): - V2 vs V3 pre-decode speed at fixed page sizes; V3 is faster for short/medium values (~2.4-2.9x at <=64B) and never slower (1.0x at 4KB). - V2 vs V3 on-disk size after ZSTD compression; same bytes, the V3 layout compresses slightly better (smaller for short values). - benchmark_main.cpp uses a custom main (DataPage allocation needs a ThreadContext + mem tracker) and re-enables binary_cast_benchmark (it was wrongly disabled; the real issue was include order, fixed by keeping our hpp last). Tests: BinaryPlainPageV3Test (encode/decode/seek/read_by_rowids/empty/page_full/varint boundaries/corruption, plus padded-CHAR round-trips that exercise the IS_CHAR read-side strip), BinaryDictPageTest V3 round-trips (word page + dict-overflow fallback), EncodingInfoTest defaults + pre-decoder dispatch, and column_meta_accessor / variant / encoding_info expectation updates. All pass under ASAN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 1, 2026
… segments Introduce PLAIN_ENCODING_V3, a binary plain page whose on-disk layout is |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|. Unlike V2 (length and data interleaved per entry), the contiguous layout lets the pre-decoder memcpy the whole data block in one shot and walk the varint lengths once to fill the V1 offsets array, with no data-vs-length pointer dependency between the two passes. V3 stores exactly the same bytes as V1/V2 — only the layout differs. In particular CHAR keeps its trailing '\0' padding on disk (written by OlapColumnDataConvertorChar), VARCHAR does not; the CHAR padding is stripped on read by BinaryPlainPageV3PreDecoder <true>, mirroring PLAIN_ENCODING_V2. Write/read wiring (post-apache#63622 EncodingInfoResolver): - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> (CppType + Slice), _register_supported_encoding for every binary type, and the IS_CHAR-aware V3 pre-decoder dispatch in the EncodingInfo constructor. - binary_plain_page_v3.h / binary_plain_page_v3_pre_decoder.h: builder and the template<bool IS_CHAR> pre-decoder (CHAR strnlen-strips on read; non-CHAR takes the single-memcpy fast path). - gensrc protos: new EncodingTypePB (=9) and BinaryPlainEncodingTypePB (=3). V3 segment defaults — every binary-plain default now uses the V3 layout: - CHAR/VARCHAR/STRING/JSONB/VARIANT keep DICT_ENCODING, but the dictionary word page (and the dict-overflow fallback plain page) use V3: ScalarColumnWriter::init sets dict_binary_plain_encoding to V3, binary_dict_page.cpp maps it and dispatches the V3 inner encoding, and binary_dict_page_pre_decoder.h rewrites the inner V3 page to V1. CHAR dictionary words are unpadded on read via the (CHAR, V3) IS_CHAR pre-decoder. - HLL/BITMAP/QUANTILE_STATE/AGG_STATE -> PLAIN_ENCODING_V3 (_set_v3_default). - the hidden row store column -> PLAIN_ENCODING_V3 (resolve_default_encoding). Benchmarks (be/benchmark/benchmark_binary_plain_page_v2.hpp): - V2 vs V3 pre-decode speed at fixed page sizes; V3 is faster for short/medium values (~2.4-2.9x at <=64B) and never slower (1.0x at 4KB). - V2 vs V3 on-disk size after ZSTD compression; same bytes, the V3 layout compresses slightly better (smaller for short values). - benchmark_main.cpp uses a custom main (DataPage allocation needs a ThreadContext + mem tracker) and re-enables binary_cast_benchmark (it was wrongly disabled; the real issue was include order, fixed by keeping our hpp last). Tests: BinaryPlainPageV3Test (encode/decode/seek/read_by_rowids/empty/page_full/varint boundaries/corruption, plus padded-CHAR round-trips that exercise the IS_CHAR read-side strip), BinaryDictPageTest V3 round-trips (word page + dict-overflow fallback), EncodingInfoTest defaults + pre-decoder dispatch, and column_meta_accessor / variant / encoding_info expectation updates. All pass under ASAN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 1, 2026
… segments Introduce PLAIN_ENCODING_V3, a binary plain page whose on-disk layout is |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|. Unlike V2 (length and data interleaved per entry), the contiguous layout lets the pre-decoder memcpy the whole data block in one shot and walk the varint lengths once to fill the V1 offsets array, with no data-vs-length pointer dependency between the two passes. V3 stores exactly the same bytes as V1/V2 — only the layout differs. In particular CHAR keeps its trailing '\0' padding on disk (written by OlapColumnDataConvertorChar), VARCHAR does not; the CHAR padding is stripped on read by BinaryPlainPageV3PreDecoder <true>, mirroring PLAIN_ENCODING_V2. Write/read wiring (post-apache#63622 EncodingInfoResolver): - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> (CppType + Slice), _register_supported_encoding for every binary type, and the IS_CHAR-aware V3 pre-decoder dispatch in the EncodingInfo constructor. - binary_plain_page_v3.h / binary_plain_page_v3_pre_decoder.h: builder and the template<bool IS_CHAR> pre-decoder (CHAR strnlen-strips on read; non-CHAR takes the single-memcpy fast path). - gensrc protos: new EncodingTypePB (=9) and BinaryPlainEncodingTypePB (=3). V3 segment defaults — every binary-plain default now uses the V3 layout: - CHAR/VARCHAR/STRING/JSONB/VARIANT keep DICT_ENCODING, but the dictionary word page (and the dict-overflow fallback plain page) use V3: ScalarColumnWriter::init sets dict_binary_plain_encoding to V3, binary_dict_page.cpp maps it and dispatches the V3 inner encoding, and binary_dict_page_pre_decoder.h rewrites the inner V3 page to V1. CHAR dictionary words are unpadded on read via the (CHAR, V3) IS_CHAR pre-decoder. - HLL/BITMAP/QUANTILE_STATE/AGG_STATE -> PLAIN_ENCODING_V3 (_set_v3_default). - the hidden row store column -> PLAIN_ENCODING_V3 (resolve_default_encoding). Benchmarks (be/benchmark/benchmark_binary_plain_page_v2.hpp): - V2 vs V3 pre-decode speed at fixed page sizes; V3 is faster for short/medium values (~2.4-2.9x at <=64B) and never slower (1.0x at 4KB). - V2 vs V3 on-disk size after ZSTD compression; same bytes, the V3 layout compresses slightly better (smaller for short values). - benchmark_main.cpp uses a custom main (DataPage allocation needs a ThreadContext + mem tracker) and re-enables binary_cast_benchmark (it was wrongly disabled; the real issue was include order, fixed by keeping our hpp last). Tests: BinaryPlainPageV3Test (encode/decode/seek/read_by_rowids/empty/page_full/varint boundaries/corruption, plus padded-CHAR round-trips that exercise the IS_CHAR read-side strip), BinaryDictPageTest V3 round-trips (word page + dict-overflow fallback), EncodingInfoTest defaults + pre-decoder dispatch, and column_meta_accessor / variant / encoding_info expectation updates. All pass under ASAN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 2, 2026
… segments Introduce PLAIN_ENCODING_V3, a binary plain page whose on-disk layout is |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|. Unlike V2 (length and data interleaved per entry), the contiguous layout lets the pre-decoder memcpy the whole data block in one shot and walk the varint lengths once to fill the V1 offsets array, with no data-vs-length pointer dependency between the two passes. V3 stores exactly the same bytes as V1/V2 — only the layout differs. In particular CHAR keeps its trailing '\0' padding on disk (written by OlapColumnDataConvertorChar), VARCHAR does not; the CHAR padding is stripped on read by BinaryPlainPageV3PreDecoder <true>, mirroring PLAIN_ENCODING_V2. Write/read wiring (post-apache#63622 EncodingInfoResolver): - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> (CppType + Slice), _register_supported_encoding for every binary type, and the IS_CHAR-aware V3 pre-decoder dispatch in the EncodingInfo constructor. - binary_plain_page_v3.h / binary_plain_page_v3_pre_decoder.h: builder and the template<bool IS_CHAR> pre-decoder (CHAR strnlen-strips on read; non-CHAR takes the single-memcpy fast path). - gensrc protos: new EncodingTypePB (=9) and BinaryPlainEncodingTypePB (=3). V3 segment defaults — every binary-plain default now uses the V3 layout: - CHAR/VARCHAR/STRING/JSONB/VARIANT keep DICT_ENCODING, but the dictionary word page (and the dict-overflow fallback plain page) use V3: ScalarColumnWriter::init sets dict_binary_plain_encoding to V3, binary_dict_page.cpp maps it and dispatches the V3 inner encoding, and binary_dict_page_pre_decoder.h rewrites the inner V3 page to V1. CHAR dictionary words are unpadded on read via the (CHAR, V3) IS_CHAR pre-decoder. - HLL/BITMAP/QUANTILE_STATE/AGG_STATE -> PLAIN_ENCODING_V3 (_set_v3_default). - the hidden row store column -> PLAIN_ENCODING_V3 (resolve_default_encoding). Benchmarks (be/benchmark/benchmark_binary_plain_page_v2.hpp): - V2 vs V3 pre-decode speed at fixed page sizes; V3 is faster for short/medium values (~2.4-2.9x at <=64B) and never slower (1.0x at 4KB). - V2 vs V3 on-disk size after ZSTD compression; same bytes, the V3 layout compresses slightly better (smaller for short values). - benchmark_main.cpp uses a custom main (DataPage allocation needs a ThreadContext + mem tracker) and re-enables binary_cast_benchmark (it was wrongly disabled; the real issue was include order, fixed by keeping our hpp last). Tests: BinaryPlainPageV3Test (encode/decode/seek/read_by_rowids/empty/page_full/varint boundaries/corruption, plus padded-CHAR round-trips that exercise the IS_CHAR read-side strip), BinaryDictPageTest V3 round-trips (word page + dict-overflow fallback), EncodingInfoTest defaults + pre-decoder dispatch, and column_meta_accessor / variant / encoding_info expectation updates. All pass under ASAN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 2, 2026
… segments Introduce PLAIN_ENCODING_V3, a binary plain page whose on-disk layout is |data1..dataN|varuint_len1..varuint_lenN|data_block_size(u32)|num_elems(u32)|. Unlike V2 (length and data interleaved per entry), the contiguous layout lets the pre-decoder memcpy the whole data block in one shot and walk the varint lengths once to fill the V1 offsets array, with no data-vs-length pointer dependency between the two passes. V3 stores exactly the same bytes as V1/V2 — only the layout differs. In particular CHAR keeps its trailing '\0' padding on disk (written by OlapColumnDataConvertorChar), VARCHAR does not; the CHAR padding is stripped on read by BinaryPlainPageV3PreDecoder <true>, mirroring PLAIN_ENCODING_V2. Write/read wiring (post-apache#63622 EncodingInfoResolver): - encoding_info.cpp: TypeEncodingTraits<PLAIN_ENCODING_V3> (CppType + Slice), _register_supported_encoding for every binary type, and the IS_CHAR-aware V3 pre-decoder dispatch in the EncodingInfo constructor. - binary_plain_page_v3.h / binary_plain_page_v3_pre_decoder.h: builder and the template<bool IS_CHAR> pre-decoder (CHAR strnlen-strips on read; non-CHAR takes the single-memcpy fast path). - gensrc protos: new EncodingTypePB (=9) and BinaryPlainEncodingTypePB (=3). V3 segment defaults — every binary-plain default now uses the V3 layout: - CHAR/VARCHAR/STRING/JSONB/VARIANT keep DICT_ENCODING, but the dictionary word page (and the dict-overflow fallback plain page) use V3: ScalarColumnWriter::init sets dict_binary_plain_encoding to V3, binary_dict_page.cpp maps it and dispatches the V3 inner encoding, and binary_dict_page_pre_decoder.h rewrites the inner V3 page to V1. CHAR dictionary words are unpadded on read via the (CHAR, V3) IS_CHAR pre-decoder. - HLL/BITMAP/QUANTILE_STATE/AGG_STATE -> PLAIN_ENCODING_V3 (_set_v3_default). - the hidden row store column -> PLAIN_ENCODING_V3 (resolve_default_encoding). Benchmarks (be/benchmark/benchmark_binary_plain_page_v2.hpp): - V2 vs V3 pre-decode speed at fixed page sizes; V3 is faster for short/medium values (~2.4-2.9x at <=64B) and never slower (1.0x at 4KB). - V2 vs V3 on-disk size after ZSTD compression; same bytes, the V3 layout compresses slightly better (smaller for short values). - benchmark_main.cpp uses a custom main (DataPage allocation needs a ThreadContext + mem tracker) and re-enables binary_cast_benchmark (it was wrongly disabled; the real issue was include order, fixed by keeping our hpp last). Tests: BinaryPlainPageV3Test (encode/decode/seek/read_by_rowids/empty/page_full/varint boundaries/corruption, plus padded-CHAR round-trips that exercise the IS_CHAR read-side strip), BinaryDictPageTest V3 round-trips (word page + dict-overflow fallback), EncodingInfoTest defaults + pre-decoder dispatch, and column_meta_accessor / variant / encoding_info expectation updates. All pass under ASAN. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zhaorongsheng pushed a commit
to zhaorongsheng/doris
that referenced
this pull request
Jun 4, 2026
…he#63622) Replace the EncodingPreference + runtime hook machinery in EncodingInfoResolver with four explicit maps and four matching get methods: - _v2_default_map -> get_v2_default_encoding(type) - _v3_default_map -> get_v3_default_encoding(type) - _index_column_default_map -> get_index_column_encoding(type) - _encoding_map -> get(type, encoding, out) No on-disk format change; the resolved encodings written into ColumnMetaPB match the pre-refactor outputs for both v2 and V3 tablets. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [x] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
csun5285 added a commit
to csun5285/doris
that referenced
this pull request
Jun 16, 2026
…he#63622) Replace the EncodingPreference + runtime hook machinery in EncodingInfoResolver with four explicit maps and four matching get methods: - _v2_default_map -> get_v2_default_encoding(type) - _v3_default_map -> get_v3_default_encoding(type) - _index_column_default_map -> get_index_column_encoding(type) - _encoding_map -> get(type, encoding, out) No on-disk format change; the resolved encodings written into ColumnMetaPB match the pre-refactor outputs for both v2 and V3 tablets. Issue Number: close #xxx Related PR: #xxx Problem Summary: None - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [x] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> (cherry picked from commit 4f1dcdf)
yiguolei pushed a commit
that referenced
this pull request
Jun 17, 2026
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Aug 14, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63622 Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation required a tablet-schema flag propagated through proto, FE, Cloud, and BE writers. After apache#63622 split encoding selection into explicit V2 and V3 maps, that propagation chain is obsolete. Express the behavior directly in the V3 default map, preserve the V2 BIT_SHUFFLE default for compatibility, and lock the distinction with unit and end-to-end regression coverage. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged. ### Check List (For Author) - Test: Unit Test / Regression test / Build - `./build.sh --be -j100` - `./build.sh --fe -j100` - `./run-be-ut.sh --run --filter=EncodingInfoTest.* -j100` (8 tests passed) - `./run-regression-test.sh --run -d table_p0 -s test_storage_format_controls_encoding` (1 suite passed) - `build-support/clang-format.sh` and `build-support/check-format.sh` - Behavior changed: Yes. Newly written V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; V2 remains BIT_SHUFFLE. - Does this need documentation: No
12 tasks
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Aug 18, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63622 Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation required a tablet-schema flag propagated through proto, FE, Cloud, and BE writers. After apache#63622 split encoding selection into explicit V2 and V3 maps, that propagation chain is obsolete. Express the behavior directly in the V3 default map, preserve the V2 BIT_SHUFFLE default for compatibility, and lock the distinction with unit and end-to-end regression coverage. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged. ### Check List (For Author) - Test: Unit Test / Regression test / Build - `./build.sh --be -j100` - `./build.sh --fe -j100` - `./run-be-ut.sh --run --filter=EncodingInfoTest.* -j100` (8 tests passed) - `./run-regression-test.sh --run -d table_p0 -s test_storage_format_controls_encoding` (1 suite passed) - `build-support/clang-format.sh` and `build-support/check-format.sh` - Behavior changed: Yes. Newly written V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; V2 remains BIT_SHUFFLE. - Does this need documentation: No
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Aug 18, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63622 Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation required a tablet-schema flag propagated through proto, FE, Cloud, and BE writers. After apache#63622 split encoding selection into explicit V2 and V3 maps, that propagation chain is obsolete. Express the behavior directly in the V3 default map, preserve the V2 BIT_SHUFFLE default for compatibility, and lock the distinction with unit, golden-format, and end-to-end regression coverage. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged. ### Check List (For Author) - Test: Unit Test / Regression test / Build - `./build.sh --be -j100` - `./build.sh --fe -j100` - `./run-be-ut.sh --run '--filter=EncodingInfoTest.*:SegmentFlusherFormatTest.AllSupportedScalarValueTypesKeepTheirSegmentBytes' -j100` (9 tests passed) - `./run-regression-test.sh --run -d table_p0 -s test_storage_format_controls_encoding` (1 suite passed) - `build-support/clang-format.sh` and `build-support/check-format.sh` - Behavior changed: Yes. Newly written V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; V2 remains BIT_SHUFFLE. - Does this need documentation: No
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Aug 26, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63622 Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation required a tablet-schema flag propagated through proto, FE, Cloud, and BE writers. After apache#63622 split encoding selection into explicit V2 and V3 maps and persisted storage_format in TabletSchema, that propagation chain is obsolete. Express the behavior directly in the V3 default map, preserve the V2 BIT_SHUFFLE default for compatibility, and refresh all affected scalar and materialized Variant Segment goldens. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter='SegmentFlusherFormatTest.*KeepTheirSegmentBytes:SegmentFlusherTransformFormatTest.PartialUpdateAndRowBinlogPathsKeepTheirSegmentBytes' -j100` (6 tests passed in two generation runs; all 164 generated Segment files were byte-identical between runs) - `./run-be-ut.sh --run --filter='EncodingInfoTest.*:SegmentFlusherFormatTest.*:SegmentFlusherTransformFormatTest.*' -j100` (17 tests passed against the checked-in goldens) - `build-support/clang-format.sh` - `build-support/check-format.sh` - `git diff --check` - Behavior changed: Yes. Newly written V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; V2 remains BIT_SHUFFLE. - Does this need documentation: No
bobhan1 added a commit
to bobhan1/doris
that referenced
this pull request
Aug 26, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63622 Problem Summary: Segment V3 continued to select BIT_SHUFFLE for FLOAT and DOUBLE columns. The original implementation required a tablet-schema flag propagated through proto, FE, Cloud, and BE writers. After apache#63622 split encoding selection into explicit V2 and V3 maps and persisted storage_format in TabletSchema, that propagation chain is obsolete. Express the behavior directly in the V3 default map, preserve the V2 BIT_SHUFFLE default for compatibility, and refresh all affected scalar and materialized Variant Segment goldens. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged. ### Check List (For Author) - Test: Unit Test - `./run-be-ut.sh --run --filter='SegmentFlusherFormatTest.*KeepTheirSegmentBytes:SegmentFlusherTransformFormatTest.PartialUpdateAndRowBinlogPathsKeepTheirSegmentBytes' -j100` (6 tests passed in two generation runs; all 164 generated Segment files were byte-identical between runs) - `./run-be-ut.sh --run --filter='EncodingInfoTest.*:SegmentFlusherFormatTest.*:SegmentFlusherTransformFormatTest.*' -j100` (17 tests passed against the checked-in goldens) - `build-support/clang-format.sh` - `build-support/check-format.sh` - `git diff --check` - Behavior changed: Yes. Newly written V3 FLOAT and DOUBLE pages use PLAIN instead of BIT_SHUFFLE; V2 remains BIT_SHUFFLE. - Does this need documentation: No
gavinchou pushed a commit
that referenced
this pull request
Aug 27, 2026
…62649) Related PR: #63622 Problem Summary: Segment V3 continued to select `BIT_SHUFFLE` for `FLOAT` and `DOUBLE` columns. The original implementation added a tablet-schema flag and propagated it through proto, FE, Cloud, and BE writers. After #63622 split encoding selection into explicit V2 and V3 maps and persisted `storage_format` in `TabletSchema`, that propagation chain is obsolete. This PR expresses the behavior directly in the V3 default map while preserving the V2 default for compatibility. Changes: - Use `PLAIN_ENCODING` as the V3 default for `FLOAT` and `DOUBLE`. - Keep the V2 default as `BIT_SHUFFLE`. - Update the authoritative encoding-map unit tests to lock both behaviors. - Regenerate all ten affected SegmentFlusher golden files across the two scalar and four complex Variant V3 cases. ### Release note Segment V3 now uses PLAIN encoding by default for FLOAT and DOUBLE columns. Segment V2 behavior is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the EncodingPreference + runtime hook machinery in EncodingInfoResolver with four explicit maps and four matching get methods:
No on-disk format change; the resolved encodings written into ColumnMetaPB match the pre-refactor outputs for both v2 and V3 tablets.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)