[Feature](Complex-type) Add struct and map type to Doris by xy720 · Pull Request #16444 · apache/doris

xy720 · 2023-02-06T08:37:53Z

Proposed changes

Support complex type struct and map to Doris.

Problem summary

This commit support:

1、Insert + select for struct/map type
2、Json stream load for struct type
3、m[key] function for map type

How to use:

Set the fe config to create table for struct and map type
1、admin set frontend config("enable_struct_type" = "true");
2、admin set frontend config("enable_map_type" = "true");

#16547

Co-authored-by: xy720 <xuyang25@baidu.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
Co-authored-by: hucheng01 <hucheng01@baidu.com>

Checklist(Required)

Does it affect the original behavior:
- Yes
- No
- I don't know
Has unit tests been added:
- Yes
- No
- No Need
Has document been added or modified:
- Yes
- No
- No Need
Does it need to update dependencies:
- Yes
- No
Are there any changes that cannot be rolled back:
- Yes (If Yes, please explain WHY)
- No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

hello-stephen · 2023-02-06T09:51:42Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.54 seconds
load time: 471 seconds
storage size: 17171008864 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230206134540_clickbench_pr_90973.html

morningman · 2023-02-06T14:20:36Z

LGTM with thrift and proto.
cc @xiaokang , PTAL

github-actions

clang-tidy made some suggestions

cambyzju

LGTM

github-actions · 2023-02-08T02:27:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-02-08T02:27:59Z

PR approved by anyone and no changes requested.

xiaokang

LGTM.

add some comments that can be fixed by further PRs

xiaokang · 2023-02-07T06:25:54Z

+ ;
+
+struct_literal ::=
+ LBRACE expr_list:list RBRACE


User may get confused if map literal and struct literal both use brace{}.

For now, we keep struct literal using brace{}.
After we implement struct() function, we can support paren() for struct.

xiaokang · 2023-02-07T06:37:17Z

+ public String getStringValue() {
+ List<String> list = new ArrayList<>(children.size());
+ children.forEach(v -> list.add(v.getStringValue()));
+ return "{" + StringUtils.join(list, ", ") + "}";


suggest using '(' ')' to keep consistency.

May be we should support '(' ')' only in insert stmt in future.
For select stmt and other places, we keep using brace{}.

xiaokang · 2023-02-08T04:16:42Z

@@ -95,7 +97,10 @@ class DataTypeFactory {
 });


register Map to _invert_data_type_map like Array

May be we will add this part in next map pr.

…truct implement (#14545) Co-authored-by: hucheng01 <hucheng01@baidu.com>

Fix the dead loop when create column struct Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

1、support insert null for struct type; 2、fix explain const exprs for strut type got wrong result; Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

Add complex type map to doris on vectorized engine

…r type (#16400)

cambyzju

LGTM

github-actions · 2023-02-10T02:54:43Z

PR approved by at least one committer and no changes requested.

This commit support: 1、Insert + select for struct/map type 2、Json stream load for struct type 3、m[key] function for map type How to use: Set the fe config to create table for struct and map type 1、admin set frontend config("enable_struct_type" = "true"); 2、admin set frontend config("enable_map_type" = "true"); apache#16547 Co-authored-by: xy720 <xuyang25@baidu.com> Co-authored-by: amory <wangqiannan@selectdb.com> Co-authored-by: cambyzju <zhuxiaoli01@baidu.com> Co-authored-by: hucheng01 <hucheng01@baidu.com>

github-actionsBot added area/planner Issues or PRs related to the query planner area/sql/function Issues or PRs related to the SQL functions area/vectorization kind/docs Categorizes issue or PR as related to documentation. labels Feb 6, 2023

xy720 changed the title ~~[Feature](Struct/Map-type) Add struct and map type to Doris~~[Feature](Complex-type) Add struct and map type to DorisFeb 6, 2023

xy720 added struct-type Issues or PRs related to struct type map-type Issues or PRs related to map type labels Feb 6, 2023

github-actionsBot added the kind/test label Feb 6, 2023

github-actionsBot reviewed Feb 7, 2023
View reviewed changes

Comment threadbe/src/olap/types.h
Comment threadbe/src/olap/types.h
Comment threadbe/src/olap/types.h
Comment threadbe/src/olap/types.h
Comment threadbe/test/vec/jsonb/serialize_test.cpp
Comment threadbe/test/vec/jsonb/serialize_test.cpp

cambyzju previously approved these changes Feb 8, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 8, 2023

github-actionsBot added the reviewed label Feb 8, 2023

xiaokang reviewed Feb 8, 2023
View reviewed changes

xiaokang previously approved these changes Feb 8, 2023
View reviewed changes

xy720 dismissed stale reviews from xiaokang and cambyzju via 7049decFebruary 8, 2023 07:34

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Feb 8, 2023

xy720 force-pushed the struct-map-type branch from 83a8194 to 552c49eCompare February 9, 2023 02:17

xy720and others added 8 commits February 9, 2023 20:48

test
9464e96

save
d4b6386

[test] (test) test modify branch (#14375)
da52757

[feature](struct-type) add the class ColumnStruct and class DataTypeS…
198e0b6
…truct implement (#14545) Co-authored-by: hucheng01 <hucheng01@baidu.com>

[WIP](struct-type) support struct-type in vectorize engine (#15665)
c662151

[fix](struct-type) struct create fix and add more struct codes (#15879)
b307c4c
Fix the dead loop when create column struct Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[feature](struct-type) adjust the vexpr for struct type (#15669)
ca1bf0a

save (#15906)
fb718ce

xy720and others added 19 commits February 9, 2023 20:48

save (#15908)
c9cd464

[fix](struct-type) support to remove double or single quote (#15977)
0f9be0e

[fix](struct-type) support struct type insert (#15941)
381ccb1
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[hotfix](struct-type) Fix type descriptor make array load core (#16020)
aacbb79

[fix](struct-type) select of struct return wrong result (#16032)
c94ff76
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[fix](struct-type) fix insert null and explain for struct type (#16054)
fbda784
1、support insert null for struct type; 2、fix explain const exprs for strut type got wrong result; Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>

[Feature](map) add map type to doris (#15966)
3d4d1a1
Add complex type map to doris on vectorized engine

[fix](map-type) fix compile error (#16413)
d8c9521

[fix](complex_type) fix clone_type_info function return default scala…
75bd917
…r type (#16400)

rebase master and fix compile error
2d99c5d

[bug](struct-type) fix be ut and format
569894f

[bug](struct-type) fix be ut and format
3084f61

[bug](struct-type) fix be ut and format
cb81a84

[bug](struct-type) fix array type miss scale
8af00eb

[bug](regression-test) fix array regression test
e6a2be5

[fix](struct-type) remove-empty-line
731d4ef

[fix] fix code for comment
df72861

[fix] fix fe ut
555cea6

fix regression-test core
ba9088c

xy720 force-pushed the struct-map-type branch from 552c49e to ba9088cCompare February 9, 2023 12:49

cambyzju approved these changes Feb 10, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 10, 2023

xy720 merged commit 1b3902b into masterFeb 10, 2023

xy720 mentioned this pull request Feb 10, 2023
[Feature](struct-type) Add implicitly cast for struct-type #16613
Merged
13 tasks

xy720 mentioned this pull request Feb 24, 2023
[Roadmap] Support Struct+Map Type #16547
Closed
88 tasks

adonis0147 deleted the struct-map-type branch November 13, 2023 06:17

Uh oh!

Conversation

xy720 commented Feb 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Problem summary

This commit support:

How to use:

Checklist(Required)

Further comments

Uh oh!

hello-stephen commented Feb 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

morningman commented Feb 6, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cambyzju left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 8, 2023

Uh oh!

github-actionsBot commented Feb 8, 2023

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xiaokangFeb 7, 2023

Choose a reason for hiding this comment

Uh oh!

xy720Feb 8, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xiaokangFeb 7, 2023

Choose a reason for hiding this comment

Uh oh!

xy720Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

xiaokangFeb 8, 2023

Choose a reason for hiding this comment

Uh oh!

xy720Feb 8, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cambyzju left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 10, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

xy720 commented Feb 6, 2023•
edited
Loading

hello-stephen commented Feb 6, 2023•
edited
Loading

xy720Feb 8, 2023•
edited
Loading