Uh oh!
There was an error while loading. Please reload this page.
[refactor](index) Remove CommonGrams and realign the SNII on-disk layout with the released format - #67537
Draft
airborne12 wants to merge 7 commits into
Draft
[refactor](index) Remove CommonGrams and realign the SNII on-disk layout with the released format#67537airborne12 wants to merge 7 commits into
airborne12 wants to merge 7 commits into
Conversation
用生产写入栈写出一组普通 SNII 段（keyword/english/unicode、标量与 ARRAY、 含 NULL、空串、控制字节 term、超长文档），把每条查询的 docid 集合、null bitmap 与状态码记成期望文件；之后的任何改动都必须原样回读并逐条一致。 校验时每条查询跑三遍（结果缓存关闭、开启冷、开启热）。 SNII_GOLDEN_DIR 未设置时跳过。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ
- IndexPolicyMgr：删 validateAnalyzerUsesCommonGrams、analyzer 图的 unit-position 约束（UNIT_POSITION_* / validateAnalyzerGraphLocked / requireAnalyzerLocked / resolveAnalyzerComponentLocked）与 common_grams token filter 类型；新增 validateReferencedTokenFiltersUsableLocked 与加载/回放时 warnIfUnsupported： 老镜像里残留的 common_grams 策略照常加载但标记无效，引用它的 analyzer 在 建索引时被明确拒绝，FE 不会因此起不来。 - IndexPolicy：删 COMMON_GRAMS_TYPE / isCommonGramsPolicy；isInvalid 识别 LEGACY_UNSUPPORTED_TOKEN_FILTER_TYPES。 - InvertedIndexUtil.checkAnalyzerName 恢复为 CommonGrams 之前的形态 （validateAnalyzerExists），去掉 SNII/ARRAY/support_phrase 的 CG 专属限制。 - CheckScoreUsage / PushDownScoreTopNIntoOlapScan：删整个 score() 策略准入 （它只为 CommonGrams 存在）。 - 删 CheckScoreUsageTest、CommonGramsDdlValidationTest（均随 CG 引入）， 修 InvertedIndexPropertiesTest；删回归 test_common_grams_snii 及数据。 验证：fe-core 主代码与全部测试编译通过，InvertedIndexPropertiesTest 49/49， checkstyle 0 违规。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ
已上线的 3.1 系生产 writer不写 SniiStatsPB.sum_total_term_freq(5) 与 SniiSectionRefsPB.norms(5)，而 master 的 decode 把它们当必填，导致 master 打开 生产段直接报 corrupted（真实二进制互读：8/8 样本 "missing statistics field"）。 缺失时按 ttf=0、norms=空 region 处理，只影响 BM25 打分是否可用；encode 在 norms 为空时不再写字段 5，与生产 writer 的字节形态一致。 新增用例：AcceptsProductionShapeWithoutNormsAndTotalTermFreq、OmitsEmptyNormsRefOnEncode。 验证：格式单测 20/20；master 黄金样本回读 513/0；生产样本回读的失败点从 core 元数据后移到 dict entry 解码（布局分叉，另行处理）。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ
- SniiIndexReader：结果缓存/single-flight 只以 (索引文件, 列, 查询类型, 原始查询字节) 为键，打开 segment 之前即可判定，只有打分查询不进缓存；删 analyzer 上下文重建 (segment_analyzer_context)、force_plain / kill switch / 代价模型、CG 计划入口、 has_common_gram bypass；SniiQueryBitmapRequest 去掉 CG 字段。 - 缓存键去掉 enable_common_grams_query_plan 位，INVERTED_INDEX_QUERY_CACHE_SEMANTICS_VERSION bump 到 2。 - 短语查询：删 planned_exact_phrase_query / planned_phrase_prefix_query 及其 hybrid 计划、 PhysicalPhrasePlan、代价模型（phrase_cost.cpp 整删）、CommonGramsPlanningTimer、 PlanKind / DebugOverride；只保留 plain 的 phrase_query_impl / phrase_prefix_query_impl。 - profile：删 17 个 common_grams_* 计数器（SniiQueryStats / PhraseQueryExecutionStats / SniiPrxProfile）。 - core 元数据：条件式墓碑——载体带 gram 覆盖、键转义或混合 posting 策略的段 Unsupported（必须重建索引）；只借载体存打分统计的 kNone 段照常读取。 - 测试：删 common_grams_namespace_query_test / common_grams_dict_plan_test； count_fallback / query_profile / analysis_purpose / index_policy_mgr / scanner_context / snii_vs_v3_benchmark 去 CG 用例与计数器；analysis_purpose 期望的 purpose 改由 select_analysis_purpose 决定；golden bytes 摘要因阶段 A 不再写空 norms 引用而更新。 验证：定向 348 条测试全过；master 黄金样本回读 513/0。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ
…alyzer framework (phase C)
Writer side: SniiIndexWriter no longer seeds CommonGrams metadata; an analyzed
index with positions always writes BM25 norms (A2: norms are an optional T2
region, no new index_config). IndexTier::kT3 / kDocsPositionsScoring /
has_scoring() are gone: scoring capability is "has positions + has norms".
Compaction: SniiStreamedMergeKind and the CommonGrams T3 merge plan are gone.
The destination writes norms exactly when its index is analyzed, and the
posting merger rebuilds each document's norm as the u8-saturating sum of term
frequencies (encode_norm maps empty documents to 1), so legacy segments
without norms gain scoring after a normal compaction -- verified byte-identical
against a rebuilt index. Streamed sessions late-bind norms exactly once via
set_encoded_norms.
Scoring statistics are physical again: SniiStatsProvider reads the stats
block, resolve_snii_scoring_segment needs positions + norms, analyzer
fingerprints are deleted. Custom analyzers build one analyzer regardless of
purpose; the CommonGrams token filter, word set query cost, and the four
enable_common_grams_* configs are deleted. common_word_set.{h,cpp} stay until
phase D removes the SPIMI pair-key machinery that still includes them.
Tests: CommonGrams-specific suites deleted, norms reconstruction and late
binding covered by new cases; golden corpus verify 513/0.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ…SPIMI pair-key path (phase D)
Term keys are the analyzed bytes, full stop. PlainTermKeyVersion, the
escaped-v1 codec, CommonGramsSegmentMetadata and the per-segment key routing
are gone: every segment -- production ones included -- is read the way
production segments always were (raw key, bypass on overlap with the
\x1F phrase-bigram namespace). Prefix expansion no longer splits the dictionary
around an escape prefix.
Core metadata keeps fields 4/5 of SniiCoreMetadataPB as opaque legacy fields:
their mere presence marks a segment written with CommonGrams and is rejected
as NOT_SUPPORTED ("rebuild the index"). Production writers never emitted them.
SPIMI: the pair-key fast path (10-byte transient keys, L0 caches, native pair
interner, common-word classification, escaped-key logical ordering) and the
freq-less "statless" posting shape are deleted from the term buffer and the
spill-run codec; run records are always docs+freqs or positioned.
AnalysisPurpose is collapsed: an analyzer provider hands out one analyzer for
indexing and every query type, so select_analysis_purpose and the purpose
overloads of create_analyzer/get_analyse_result disappear.
Deleted: common_grams_key_codec.{h,cpp}, common_grams_segment_metadata.{h,cpp},
common_word_set.{h,cpp} and their tests; a missing declaration of
append_resolved_phrase_clause that only unity builds hid is fixed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJkFormatVersion 保持 1。master 在 CommonGrams/打分演进期加到 v1 物理布局里的所有 freq 相关字段全部删除，dict entry / dict block / frq prelude / frq pod 的编解码直接 采用生产分支（3.1 系）的实现，使新 BE 能读生产段、回滚后的旧 BE 也能读新 BE 写出的段。 删除的布局要素： - prelude flags：bit0 = has_prx（原 bit0=has_freq/bit1=has_prx）；未知 flag 拒绝 - 窗口行：freq 区（freq_off/disk_len/uncomp_len/crc_freq）、max_freq、max_norm - win_mode：仅 kDdZstd，kFreqZstd 删除 - dict entry：ttf_delta / max_freq / term_stats_present / frq_docs_len / inline_dd_disk_len / freq_meta；dict block 的 kNoTermStats 标志 - freq 区编解码（build_freq_region / decode_freq_region）与 WindowEmitter 的 freq 分级/暂存路径 - BE 开关 snii_positions_index_write_freq 与 index_file_writer 的 write_freq 解析 打分能力保留，扩展只放在老 reader 忽略的 proto 引用 region： - norms = core 元数据字段 5（可选），sum_total_term_freq = stats 字段 5（可选） - BM25 的 tf = 该 doc 在 prx 里的位置数；writer 的 total_freq 只喂 sum_total_term_freq - WAND / block-max 路径（scoring_query_wand / _wand_selective、bm25 max_score）删除， 保留 candidates / exhaustive 两条打分入口 - compaction 的 norms 重建改为从位置计数得到词频 测试： - 格式层四个测试（dict_entry/dict_block/frq_prelude/frq_pod）直接采用生产版本 - 删除 scoring_wand_selective_test；其余 freq 相关用例按新布局改写或删除 - 黄金摘要 12 个常量重采 - 三个 cursor 夹具的 slim 项改为不规则 docid 间距（去掉 freq 区后原夹具落进 inline 阈值） - spill run 测试改为显式给 freqs（无频次形状只有 CommonGrams 用过，生产 SPIMI 总会填充） - 短语位置工作量估算只读 prx 帧头不校验 crc（read_prx_frame 新增 verify_crc）， 避免每次短语查询多算一遍 crc32c；坏帧仍在解码时被拦下 验证（真实二进制，黄金语料 8 样本 / 179 条期望 / 513 项校验）： - 定向 247 套件 1828 条：1825 过、5 跳过、0 失败 - master(S1) 读生产 writer 写出的段：8/8 打开，513/0 - S1 写出的段 + 删除前 master 的期望：513/0（语义等价） - 生产二进制读 S1 写出的段：8/8 打开，513/0（回滚方向兼容） Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ
hello-stephen
commented
Sep 4, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
CommonGrams(the token filter, its word set, the term-key escaping codec, the per-segment metadata, the hybrid docs-only/positioned query planner and its result-cache gate, the SPIMI pair-key fast path, 17 profile counters, 4 BE switches, the FE DDL validation and thescore()admission rule) was an experiment that never reached a released build, yet it had grown roots through the whole SNII stack and had diverged the v1 on-disk layout from what the released 3.1-based writer produces. This PR removes it end-to-end and realigns the physical format so that a node running this code can read segments written by the released writer, and vice versa, without a format version bump.What changes:
be/src/storage/index/inverted/common_grams/, thecommon_gramstoken filter, the SNII term-key escaping codec,SniiCommonGramsMetadataPB, the hybrid query planner and its cost model, the SPIMI pair-key path, the frequency-less spill-run shape) and in FE (thecommon_gramstoken filter type, the CommonGrams policy validation, thescore()admission rule). Loading an existing policy of that type marks it invalid with an error log instead of crashing FE.\x1Fphrase-bigram namespace bypass the index (the same rule the released writer already implied).index_config).tfis the number of positions the term has in the document. Older segments without norms answerNOT_SUPPORTEDfor scoring only; a normal compaction reconstructs norms from the postings (byte-identical to a rebuild), so they become scorable after one compaction. The WAND / block-max paths are removed together with the per-window max statistics they depended on.kFormatVersionstays 1. The dict entry / dict block / frq prelude / frq pod encoders and decoders are byte-for-byte the released layout again: prelude flag bit0 =has_prx, no freq regions, no per-windowmax_freq/max_norm, no per-entry term statistics, nofrq_docs_len/inline_dd_disk_len/freq_meta, nokNoTermStatsdict-block flag.sum_total_term_freqand the norms reference live in optional proto fields the released reader ignores. Segments carrying the experimental CommonGrams metadata fields (or the experimental scoringindex_config) are rejected asNOT_SUPPORTEDwith a rebuild hint.read_prx_frame(..., verify_crc=false)), so a phrase query no longer checksums every candidate prx window twice; and the frequency-less spill-run record shape is gone (the production SPIMI path always materializes frequencies).Compatibility evidence (real binaries, golden corpus of 8 sample indexes / 179 query expectations / 513 checks across cache-off, cache-cold and cache-warm):
Size: 178 files, +2321 / −20981 (be/src −8184, be/test −10066, fe −946).
Release note
The experimental
common_gramstoken filter and its scoring tier are removed. BM25 scoring on a tokenized SNII index now requires the norms region, which the writer always emits for tokenized indexes with positions; segments written before this change become scorable after one compaction.Check List (For Author)
Test
A real in-place cluster upgrade rehearsal was run on a mini cluster (details below), plus the unit and golden-corpus evidence.
Cluster upgrade rehearsal. A mini cluster first ran the released 3.1-based build (which has SNII and the CommonGrams experiment, and no scoring at all), loaded 20k rows into a single tablet with a tokenized
support_phraseSNII index, then the binaries were swapped in place (FE metadata and BE storage untouched) for this branch. Results: (a) twelve queries over the pre-upgrade data — counts, MATCH_ANY/MATCH_ALL, phrase, phrase-prefix, keyword equality, id lists, sums — return byte-identical results before the upgrade, after the upgrade, after a full compaction, and after rolling back; (b) a scoring query over the pre-upgrade segments fails cleanly withSNII scoring requires positions and norms; this segment was written without norms, and returns results after one full compaction, which is the documented path; (c) 2k rows loaded after the upgrade mix correctly with the pre-upgrade data; (d) a table built with acommon_gramsanalyzer on the old version: the new FE starts normally and logs that the policy's token filter type is no longer supported, a new table referencing that analyzer is rejected with a clear message, and querying the old table fails cleanly. Rolling back needed two steps unrelated to this PR — restoring the pre-upgrade FE metadata backup (a new FE image is not readable by the old FE, a general Doris property) and removing the stream-load record RocksDB (the newer BE adds a column family there) — after which the released binary reads the segments this branch wrote, including the compaction-rewritten ones, with identical results.Full BE UT after the change: 13158 passed, 1 failed (
TableReaderTest.CountStarFallbackKeepsLateRuntimeFilterCarrierValues, fails identically on the base commit: a shared/tmppermission issue). Compared name-by-name with the base commit's run (13448 tests): 274 tests removed or renamed by this PR, 61 environment-gated tests skipped in both runs, 48 tests added or renamed. Targeted SNII suites: 247 suites / 1828 tests green. FE: fe-core 10442 tests, 3 failures + 5 errors — all eight reproduce identically (same tests, same per-class counts) when the same seven test classes run on the base commit in a clean worktree, so they are pre-existing: four are the test environment's mocked backend not coming up ("available backend num is 0"), one is a JSON cast in an HTTP response test, two are nereids rewrite-rule plan pattern mismatches, one is a cloud-mode assertion inPropertyAnalyzerTest. None of those test files or the code they exercise is touched by this PR. Manual: the cross-read matrix above, run with the golden-corpus test (SNII_GOLDEN_DIR/SNII_GOLDEN_MODE).Behavior changed:
common_gramsis no longer a valid token filter type; scoring requires norms (see release note); segments written with the experimental CommonGrams metadata or scoringindex_configare rejected asNOT_SUPPORTED; one analyzer instance now serves indexing and every query type.Does this need documentation?
common_gramstoken filter documentation needs to be removed (doc PR to follow).Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code
https://claude.ai/code/session_01YbPyc5mrQswEfVZXWi7nyJ