[Feature] Lightweight schema change of add/drop column by Lchangliang · Pull Request #10136 · apache/doris

Lchangliang · 2022-06-14T12:26:01Z

Proposed changes

Issue Number: close#10135

Problem Summary:

Optimize ideas for writing in associated issue

Checklist(Required)

Does it affect the original behavior: (Yes/No/I Don't know)
Has unit tests been added: (Yes/No/No Need)
Has document been added or modified: (Yes/No/No Need)
Does it need to update dependencies: (Yes/No)
Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Gabriel39 · 2022-06-14T12:35:15Z

What a fancy improvement! And after reading your issue, I'm curious will it increase the pressure on metadata storage because we persist all schema of all rowsets?

yixiutt · 2022-06-15T06:15:42Z

What a fancy improvement! And after reading your issue, I'm curious will it increase the pressure on metadata storage because we persist all schema of all rowsets?

schema is really small, and one schema per rowset is really controllable.

yiguolei · 2022-07-01T10:08:50Z

There is already a schema info for every segment. I think you could use it.

* [feature](schema-change) support fast schema change. coauthor: yixiutt * [schema change] Using columns desc from fe to read data. coauthor: Lchangliang * [feature](schema change) schema change optimize for add/drop columns. 1.add uniqueId field for class column. 2.schema change for add/drop columns directly update schema meta Co-authored-by: yixiutt <yixiu@selectdb.com> Co-authored-by: SWJTU-ZhangLei <1091517373@qq.com> [Feature](schema change) fix write and add regression test (apache#69) Co-authored-by: yixiutt <yixiu@selectdb.com> [schema change] be ssupport that delete use newest schema add delete regression test fix regression case (apache#107) tmp [feature](schema change) light schema change exclude rollup and agg/uniq/dup key type. [feature](schema change) fe olapTable maxUniqueId write in disk. [feature](schema change) add rpc iface for sc add column. [feature](schema change) add columnsDesc to TPushReq for ligtht sc. resolve the deadlock when schema change (apache#124) fix columns from fe don't has bitmap_index flag (apache#134) add update/delete case construct MATERIALIZED schema from origin schema when insert fix not vectorized compaction coredump use segment cache choose newest schema by schema version when compaction (apache#182) [bugfix](schema change) fix ligth schema change problem. [feature](schema change) light schema change add alter job. (#1) fix be ut [bug] (schema change) unique drop key column should not light schema change [feature](schema change) add schema change regression-test. fix regression test [bugfix](schema change) fix multi alter clauses for light schema change. (#2) [bugfix](schema change) fix multi clauses calculate column unique id (#3) modify PushTask process (apache#217) [Bugfix](schema change) fix jobId replay cause bdbje exception. [bug](schema change) fix max col unique id repeatitive. (apache#232) [optimize](schema change) modify pendingMaxColUniqueId generate rule. fix compaction error * fix be ut * fix snapshot load core fix unique_id error (apache#278) [refact](fe) remove redundant code for light schema change. (#4) [refact](fe) remove redundant code for light schema change. (#4) format fe core format be core fix be ut modify fe meta version fix rebase error flush schema into rowset_meta in old table [refactor](schema change) refact fe light schema change. (#5) delete the change of schemahash and support get max version schema

github-actions · 2022-07-12T09:54:05Z

PR approved by at least one committer and no changes requested.

github-actions · 2022-07-12T09:54:09Z

PR approved by anyone and no changes requested.

* [Schema Change] support fast add/drop column (apache#49) * [feature](schema-change) support fast schema change. coauthor: yixiutt * [schema change] Using columns desc from fe to read data. coauthor: Lchangliang * [feature](schema change) schema change optimize for add/drop columns. 1.add uniqueId field for class column. 2.schema change for add/drop columns directly update schema meta Co-authored-by: yixiutt <yixiu@selectdb.com> Co-authored-by: SWJTU-ZhangLei <1091517373@qq.com> [Feature](schema change) fix write and add regression test (apache#69) Co-authored-by: yixiutt <yixiu@selectdb.com> [schema change] be ssupport that delete use newest schema add delete regression test fix regression case (apache#107) tmp [feature](schema change) light schema change exclude rollup and agg/uniq/dup key type. [feature](schema change) fe olapTable maxUniqueId write in disk. [feature](schema change) add rpc iface for sc add column. [feature](schema change) add columnsDesc to TPushReq for ligtht sc. resolve the deadlock when schema change (apache#124) fix columns from fe don't has bitmap_index flag (apache#134) add update/delete case construct MATERIALIZED schema from origin schema when insert fix not vectorized compaction coredump use segment cache choose newest schema by schema version when compaction (apache#182) [bugfix](schema change) fix ligth schema change problem. [feature](schema change) light schema change add alter job. (#1) fix be ut [bug] (schema change) unique drop key column should not light schema change [feature](schema change) add schema change regression-test. fix regression test [bugfix](schema change) fix multi alter clauses for light schema change. (#2) [bugfix](schema change) fix multi clauses calculate column unique id (#3) modify PushTask process (apache#217) [Bugfix](schema change) fix jobId replay cause bdbje exception. [bug](schema change) fix max col unique id repeatitive. (apache#232) [optimize](schema change) modify pendingMaxColUniqueId generate rule. fix compaction error * fix be ut * fix snapshot load core fix unique_id error (apache#278) [refact](fe) remove redundant code for light schema change. (#4) [refact](fe) remove redundant code for light schema change. (#4) format fe core format be core fix be ut modify fe meta version fix rebase error flush schema into rowset_meta in old table [refactor](schema change) refact fe light schema change. (#5) delete the change of schemahash and support get max version schema * modify for review * fix be ut * fix schema change test

github-actionsBot added area/load Issues or PRs related to all kinds of load area/planner Issues or PRs related to the query planner area/vectorization kind/test labels Jun 14, 2022

Lchangliang force-pushed the feature_sc_optimize branch 9 times, most recently from 6b0ee82 to 0e62965Compare June 22, 2022 10:32

github-actionsBot added the area/spark-load Issues or PRs related to the spark load label Jun 27, 2022

Lchangliang force-pushed the feature_sc_optimize branch 2 times, most recently from 923cd3b to ea7d504Compare June 30, 2022 15:09

yiguolei reviewed Jul 1, 2022
View reviewed changes

Comment threadgensrc/proto/olap_file.proto Outdated

yiguolei reviewed Jul 1, 2022
View reviewed changes

Comment threadbe/src/exec/olap_scanner.cpp

Lchangliang force-pushed the feature_sc_optimize branch 2 times, most recently from 5c1a22b to ccbf8f9Compare July 2, 2022 06:32

yiguolei reviewed Jul 2, 2022
View reviewed changes

Comment threadbe/src/olap/tablet.cpp Outdated

morningman added kind/feature Categorizes issue or PR as related to a new feature. kind/meta-version-change Categorizes issue or PR as related to changing meta version labels Jul 2, 2022

morningman reviewed Jul 2, 2022
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java Outdated
Comment threadfe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java Outdated

Lchangliang force-pushed the feature_sc_optimize branch 3 times, most recently from f1c42c3 to 3055438Compare July 3, 2022 07:06

Lchangliang force-pushed the feature_sc_optimize branch 9 times, most recently from ac6ac9b to 47c163eCompare July 10, 2022 14:33

Lchangliangand others added 4 commits July 12, 2022 10:01

modify for review
ffabbf0

fix be ut
b968442

fix schema change test
672f617

Lchangliang force-pushed the feature_sc_optimize branch from 47c163e to 672f617Compare July 12, 2022 02:02

dataroaring approved these changes Jul 12, 2022
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2022

github-actionsBot added the reviewed label Jul 12, 2022

dataroaring merged commit 486cf0e into apache:masterJul 12, 2022

BiteTheDDDDt mentioned this pull request Jul 14, 2022
[Bug][Storage] fix core dump when backend start #10818
Closed

Lchangliang mentioned this pull request Jul 22, 2022
[Improvement](light-schema-change) support tablet schema cache #11131
Merged

yiguolei mentioned this pull request Aug 11, 2022
[refactor](delete predicate) Unify delete to segmentiterator #11650
Merged
13 tasks

yiguolei mentioned this pull request Aug 19, 2022
[bugfix](light weight schema change) support delete condition in schema change #11869
Merged
13 tasks

cambyzju mentioned this pull request May 11, 2024
[fix](index) should not use light index change for bloom filter index #34732
Merged

Navigation Menu

Uh oh!

[Feature] Lightweight schema change of add/drop column - #10136

[Feature] Lightweight schema change of add/drop column#10136
dataroaring merged 4 commits into
apache:masterfrom
Lchangliang:feature_sc_optimize

Lchangliang commented Jun 14, 2022•
edited
Loading

Uh oh!

Gabriel39 commented Jun 14, 2022

Uh oh!

yixiutt commented Jun 15, 2022

Uh oh!

yiguolei commented Jul 1, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Jul 12, 2022

Uh oh!

github-actionsBot commented Jul 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

Lchangliang commented Jun 14, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem Summary:

Checklist(Required)

Further comments

Uh oh!

Gabriel39 commented Jun 14, 2022

Uh oh!

yixiutt commented Jun 15, 2022

Uh oh!

yiguolei commented Jul 1, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Jul 12, 2022

Uh oh!

github-actionsBot commented Jul 12, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Lchangliang commented Jun 14, 2022•
edited
Loading