[Feature](Dynamic schema table) step1 support schema change expression by eldenmoon · Pull Request #17494 · apache/doris

eldenmoon · 2023-03-07T05:01:06Z

introduce a new type VARIANT to encapsulate dynamic generated columns for hidding the detail of types and names of newly generated columns
introduce a new expression SchemaChangeExpr for doing schema change for extensibility

Proposed changes

Issue Number: close#17493

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

eldenmoon · 2023-03-07T06:34:57Z

run buildall

github-actions

clang-tidy made some suggestions

eldenmoon · 2023-03-09T10:46:21Z

run buildall

xiaokang · 2023-03-10T00:10:57Z

 29: optional bool is_nullable

 30: optional TJsonLiteral json_literal
+ 31: optional TSchemaChangeInfo schema_change_info


It't somewhat strange that Expr has schema_change_info.

renamed to TSchemaChangeExpr

xiaokang · 2023-03-10T00:17:44Z

+ msg.node_type = TExprNodeType.SCHEMA_CHANGE_EXPR;
+ // set type info
+ TTypeDesc desc = new TTypeDesc();
+ desc.setTypes(new ArrayList<TTypeNode>());


what's the function of setTypes to an emply list?

variant.toThrift(desc) will add TTypeNode to the empty list

xiaokang · 2023-03-10T00:50:39Z

 }
 size_t operator()(const Int64& x) {
- // Only support Int32 and Int64
+ // Only Int64 at present


why delete support for Int32

since int32 is not enough to hold values when user first load int value then load bigint value, the bigint values will be overflow

xiaokang · 2023-03-10T00:58:21Z

+ column_object.finalize();
+ // flatten object columns for the purpose of extracting static columns and
+ // fill default values missing in static columns
+ schema_util::unfold_object(columns.size() - 1, columns, _slot_desc_index, slot_descs,


there is no flatten_object before, why add unfold_object?

flatten renamed to unfold

1. introduce a new type `VARIANT` to encapsulate dynamic generated columns for hidding the detail of types and names of newly generated columns 2. introduce a new expression `SchemaChangeExpr` for doing schema change for extensibility

eldenmoon · 2023-03-10T03:45:52Z

run buildall

eldenmoon · 2023-03-10T05:50:33Z

run p0

xiaokang

LGTM

github-actions · 2023-03-13T07:03:44Z

PR approved by anyone and no changes requested.

qidaye

LGTM

* [WIP](dynamic-table) support dynamic schema table (apache#16335) Issue Number: closeapache#16351 Dynamic schema table is a special type of table, it's schema change with loading procedure.Now we implemented this feature mainly for semi-structure data such as JSON, since JSON is schema self-described we could extract schema info from the original documents and inference the final type infomation.This speical table could reduce manual schema change operation and easily import semi-structure data and extends it's schema automatically. * [improve](dynamic-table) change `addColumns` RPC interface fields from `required` to `optional` and and config doc (apache#16632) * [doc](dynamic-table) Add docs for dynamic-table (apache#16669) * [improve](dynamic table) refine SegmentWriter columns writer generate (apache#16816) * [improve](dynamic table) refine SegmentWriter columns writer generate ``` Dynamic Block consists of two parts, dynamic part of columns and static part of columns static dynamic | ----- | ------- | the static ones are original _tablet_schame columns the dynamic ones are auto generated and extended from file scan. ``` **We should only consisder to use Block info to generte columns when it's a dynamic table load procudure.** And seperate the static ones and dynamic ones * test * [typo](docs)fix dynamic Table version label (apache#16895) * [Feature](Dynamic schema table) step1 support schema change expression (apache#17494) 1. introduce a new type `VARIANT` to encapsulate dynamic generated columns for hidding the detail of types and names of newly generated columns 2. introduce a new expression `SchemaChangeExpr` for doing schema change for extensibility * Fix compile * [Bug](dynamic-table) Fix column alignment logic and support filtering null values when slot is not null Before this PR when encountering null values with some columns which is specified as `NOT NULL`, null values will not be filtered,thi behavior does not match with the original load behavior. Second column alignment logic has bug : ``` template <typename ColumnInserterFn> void align_variant_by_name_and_type(ColumnObject& dst, const ColumnObject& src, size_t row_cnt, ColumnInserterFn inserter) { CHECK(dst.is_finalized() && src.is_finalized()); // Use rows() here instead of size(), since size() will check_consistency // but we could not check_consistency since num_rows will be upgraded even // if src and dst is empty, we just increase the num_rows of dst and fill // num_rows of default values when meet new data size_t num_rows = dst.rows(); ``` --------- Co-authored-by: jiafeng.zhang <zhangjf1@gmail.com>

apache#17494) 1. introduce a new type `VARIANT` to encapsulate dynamic generated columns for hidding the detail of types and names of newly generated columns 2. introduce a new expression `SchemaChangeExpr` for doing schema change for extensibility

github-actionsBot added area/load Issues or PRs related to all kinds of load area/planner Issues or PRs related to the query planner area/vectorization kind/test labels Mar 7, 2023

eldenmoon force-pushed the dynamic-table-x branch from c62c45f to a955e85Compare March 7, 2023 05:02

github-actionsBot reviewed Mar 7, 2023
View reviewed changes

Comment threadbe/src/vec/exprs/vschema_change_expr.h

eldenmoon force-pushed the dynamic-table-x branch from a955e85 to 1dcc39bCompare March 9, 2023 10:30

github-actionsBot reviewed Mar 9, 2023
View reviewed changes

Comment threadbe/src/vec/exec/scan/vfile_scanner.cpp
Comment threadbe/src/vec/exec/scan/vfile_scanner.cpp

xiaokang reviewed Mar 10, 2023
View reviewed changes

eldenmoon added 3 commits March 10, 2023 11:02

rebase
d95ef3f

modify
fd66af8

eldenmoon force-pushed the dynamic-table-x branch from ac1fcb0 to fd66af8Compare March 10, 2023 03:45

xiaokang approved these changes Mar 13, 2023
View reviewed changes

github-actionsBot added the reviewed label Mar 13, 2023

qidaye approved these changes Mar 13, 2023
View reviewed changes

qidaye merged commit 9b7596f into apache:masterMar 13, 2023

Uh oh!

Conversation

eldenmoon commented Mar 7, 2023

Proposed changes

Problem summary

Checklist(Required)

Further comments

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eldenmoon commented Mar 7, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Mar 9, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

eldenmoon commented Mar 10, 2023

Uh oh!

eldenmoon commented Mar 10, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Mar 13, 2023

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants