Uh oh!
There was an error while loading. Please reload this page.
[Feature](partition) Support new partition recycle mechanism - #57013
Merged
Conversation
zclllyybb
requested review from
CalvinKirs, dataroaring, gavinchou, morningman and w41ter
as code ownersOctober 15, 2025 13:52
Thearas
commented
Oct 15, 2025
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
zclllyybb
commented
Oct 15, 2025
ContributorAuthor
run buildall |
doris-robot
commented
Oct 15, 2025
TPC-DS: Total hot run time: 190623 ms |
doris-robot
commented
Oct 15, 2025
ClickBench: Total hot run time: 30.64 s |
zclllyybb
commented
Oct 15, 2025
ContributorAuthor
run buildall |
doris-robot
commented
Oct 15, 2025
TPC-DS: Total hot run time: 189945 ms |
doris-robot
commented
Oct 15, 2025
ClickBench: Total hot run time: 30.08 s |
hello-stephen
commented
Oct 15, 2025
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Oct 15, 2025
Contributor
FE Regression Coverage ReportIncrement line coverage |
Uh oh!
There was an error while loading. Please reload this page.
dataroaring
commented
Oct 16, 2025
Contributor
what about using retention_count, e.g. "partition.retention_count" = "3"? |
Uh oh!
There was an error while loading. Please reload this page.
zclllyybb
commented
Oct 16, 2025
ContributorAuthor
run buildall |
doris-robot
commented
Oct 16, 2025
TPC-DS: Total hot run time: 190908 ms |
doris-robot
commented
Oct 16, 2025
ClickBench: Total hot run time: 30.38 s |
hello-stephen
commented
Oct 16, 2025
Contributor
FE UT Coverage ReportIncrement line coverage |
hello-stephen
commented
Oct 16, 2025
Contributor
FE Regression Coverage ReportIncrement line coverage |
8 tasks
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
github-actionsBot
pushed a commit
that referenced
this pull request
Nov 12, 2025
now support: ```sql create table auto_recycle( k0 datetime(6) not null ) auto partition by range (date_trunc(k0, 'day')) () DISTRIBUTED BY HASH(`k0`) BUCKETS 1 properties( "replication_num" = "1", "partition.retention_count" = "3" ); ``` which means only keep the latest 3 partition in history partitions. ccr will be tested in selectdb/ccr-syncer#646 ### Release note Resolve the problem of different name format when need to recycle auto partition
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Nov 13, 2025
…57013) now support: ```sql create table auto_recycle( k0 datetime(6) not null ) auto partition by range (date_trunc(k0, 'day')) () DISTRIBUTED BY HASH(`k0`) BUCKETS 1 properties( "replication_num" = "1", "partition.retention_count" = "3" ); ``` which means only keep the latest 3 partition in history partitions. ccr will be tested in selectdb/ccr-syncer#646 ### Release note Resolve the problem of different name format when need to recycle auto partition
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Nov 18, 2025
…57013) now support: ```sql create table auto_recycle( k0 datetime(6) not null ) auto partition by range (date_trunc(k0, 'day')) () DISTRIBUTED BY HASH(`k0`) BUCKETS 1 properties( "replication_num" = "1", "partition.retention_count" = "3" ); ``` which means only keep the latest 3 partition in history partitions. ccr will be tested in selectdb/ccr-syncer#646 ### Release note Resolve the problem of different name format when need to recycle auto partition
16 tasks
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 25, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 26, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud tests were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 26, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud regression test test_nereids_showpartitionid passed against the persistent Cloud runtime; Docker suites were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 26, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: A table can change its inverted-index storage format, but existing partitions must retain the format used when their tablets were created. Previously, there was no partition-scoped persistent format to distinguish historical tablets from tablets created after a format rollout. This change adds `partition.inverted_index_storage_format` for future logical partitions. The resolved V2 or V3 format is persisted in partition metadata and is passed to tablet and replica creation. Existing partitions are not rewritten, allowing V2 and V3 partitions to coexist while a table rolls forward. The implementation preserves this value through partition creation, auto and dynamic partitioning, edit-log replay, restore, recycle-bin recovery, truncate, insert overwrite, and replica repair. Observability: - `SHOW PARTITIONS`, `SHOW PARTITION <partition_id>`, and `partitions(...)` expose `InvertedIndexStorageFormat`. - `SHOW CREATE TABLE` includes `partition.inverted_index_storage_format` when configured. - `SHOW TABLETS FROM <table> PARTITION(<partition>)` returns each tablet MetaUrl. Requesting `GET <MetaUrl>` returns `schema.inverted_index_storage_format`, which verifies the actual BE tablet metadata. Example SQL: ```sql CREATE TABLE db.t ( k DATE NOT NULL, v VARCHAR(100), INDEX idx_v (v) USING INVERTED PROPERTIES("parser" = "english") ) ENGINE=OLAP DUPLICATE KEY(k) PARTITION BY RANGE(k) ( PARTITION p_old VALUES LESS THAN ("2024-01-01") ) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ( "replication_num" = "1", "inverted_index_storage_format" = "V2" ); ALTER TABLE db.t SET ("partition.inverted_index_storage_format" = "V3"); ALTER TABLE db.t ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01")); SHOW PARTITIONS FROM db.t; SHOW TABLETS FROM db.t PARTITION(p_new); ``` Expected observability result: `p_old` reports V2 and its tablet meta header contains `schema.inverted_index_storage_format: "V2"`; `p_new` reports V3 and its tablet meta header contains `schema.inverted_index_storage_format: "V3"`. Users can roll out V3 inverted-index storage format to newly created partitions without changing historical partitions. - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Test summary: - `./run-fe-ut.sh --run org.apache.doris.catalog.TablePropertyTest,org.apache.doris.common.PropertyAnalyzerTest` passed. - `DISABLE_BUILD_UI=ON ./build.sh --fe` passed, including FE Checkstyle and packaging. - `./run-regression-test.sh --run -d inverted_index_p0 -s test_partition_inverted_index_storage_format_rollout` passed. It covers initial, manual, auto, and dynamic partitions, `SHOW PARTITION`, and actual BE tablet meta headers. - `./run-regression-test.sh --run -d inverted_index_p2 -s test_partition_inverted_index_storage_format_replica_recovery` passed. It verifies that recovery recreates a removed V2 replica as V2 while a V3 partition remains V3. - Cloud regression test test_nereids_showpartitionid passed against the persistent Cloud runtime; Docker suites were not run. - Behavior changed: - [ ] No. - [x] Yes. Future partitions can use a different inverted-index storage format from historical partitions. - Does this need documentation? - [x] No. - [ ] Yes. - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 30, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. ### Release note Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. ### Check List (For Author) - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
hoshinojyunn added a commit
to hoshinojyunn/doris
that referenced
this pull request
Jul 31, 2026
Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition while keeping existing partitions unchanged. Persist an optional partition-level format through partition creation, replay, replacement, recycle, recovery, restore, and schema changes; a missing entry resolves to the legacy table-level format. Restore and truncate retain their source partition format. Updating the partition format default no longer synchronously fetches one tablet from every partition while holding the table write lock, eliminating the O(partition_count) Meta Service RPC scan and its retry failure mode. Cloud tables support partition-level V2/V3 inverted-index format rollout while retaining the format of existing partitions. - Test: Regression test / Unit Test - Unit Test: 13 HEAD/HEAD~1 related FE test classes, 106 passed - Unit Test: CloudSchemaChangeHandlerTest, OlapTableTest, CloudSchemaChangeJobV2Test, ModifyDynamicPartitionInfoTest, 22 passed - Regression test: external_table_p0/tvf/test_partitions_tvf passed - Regression test: Cloud suites test_cloud_show_inverted_index_storage_format, test_partition_cloud_inverted_index_storage_format_rollout, and test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Behavior changed: Yes. Cloud partitions retain their creation-time inverted-index format while new partitions use the current configured default, and property updates no longer scan every partition's tablet metadata. - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
now support:
which means only keep the latest 3 partition in history partitions.
ccr will be tested in selectdb/ccr-syncer#646
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)