[opt](parquet) Support hive struct schema change by suxiaogang223 · Pull Request #32438 · apache/doris

suxiaogang223 · 2024-03-19T02:56:28Z

Proposed changes

Followup: #31128
This optimization allows doris to correctly read struct type data after changing the schema from hive.

Changing struct schema in hive:

hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet;
hive>insert into struct_test values> (1, named_struct('f1', 1, 'f2', 's1')),
> (2, named_struct('f1', 2, 'f2', 's2')),
> (3, named_struct('f1', 3, 'f2', 's3'));
hive> alter table struct_test change sf sf struct<f1:int, f3:string>;
hive>select*from struct_test;
OK
1	{"f1":1,"f3":null}
2	{"f1":2,"f3":null}
3	{"f1":3,"f3":null}
Time taken: 5.298 seconds, Fetched: 3 row(s)

The previous result of doris was:

mysql>select*from struct_test;
+------+-----------------------+
| id | sf |
+------+-----------------------+
| 1 | {"f1": 1, "f3": "s1"} |
| 2 | {"f1": 2, "f3": "s2"} |
| 3 | {"f1": 3, "f3": "s3"} |
+------+-----------------------+

Now the result is same as hive:

mysql>select*from struct_test;
+------+-----------------------+
| id | sf |
+------+-----------------------+
| 1 | {"f1": 1, "f3": null} |
| 2 | {"f1": 2, "f3": null} |
| 3 | {"f1": 3, "f3": null} |
+------+-----------------------+

doris-robot · 2024-03-19T02:56:32Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-03-19T03:01:41Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-19T03:02:46Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-19T08:45:32Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-20T03:14:19Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-20T07:36:24Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-03-20T09:18:49Z

clang-tidy review says "All clean, LGTM! 👍"

AshinGau · 2024-03-21T02:00:50Z

LGTM

github-actions · 2024-03-21T02:01:31Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-21T02:01:33Z

PR approved by anyone and no changes requested.

zy-kkk · 2024-03-21T02:17:20Z

run buildall

github-actions · 2024-03-21T03:14:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-21T03:24:45Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8728/24751)
Line Coverage: 27.07% (71437/263874)
Region Coverage: 26.31% (37068/140880)
Branch Coverage: 23.23% (18958/81624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/239de186759e80a4fab6e31c1d5172dba18ad47a_239de186759e80a4fab6e31c1d5172dba18ad47a/report/index.html

zy-kkk · 2024-03-21T03:33:22Z

run buildall

doris-robot · 2024-03-21T03:49:55Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8728/24751)
Line Coverage: 27.07% (71425/263874)
Region Coverage: 26.31% (37068/140880)
Branch Coverage: 23.22% (18954/81624)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0f8ee1b761998602b2326ebab4a9ba47725f3cc1_0f8ee1b761998602b2326ebab4a9ba47725f3cc1/report/index.html

impl insert null when struct's field is missing check field of sturct must be nullable fix get_rep_level and get_rep_level of StructColumnReader

suxiaogang223 · 2024-03-21T06:08:10Z

run buildall

github-actions · 2024-03-21T06:13:56Z

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 · 2024-03-21T06:24:48Z

run buildall

doris-robot · 2024-03-21T08:17:40Z

TeamCity be ut coverage result:
Function Coverage: 35.27% (8730/24750)
Line Coverage: 27.09% (71493/263910)
Region Coverage: 26.32% (37079/140885)
Branch Coverage: 23.23% (18966/81630)
Coverage Report: http://coverage.selectdb-in.cc/coverage/04e3a3699c5468a665e1f3ead99f4659d94ab9c7_04e3a3699c5468a665e1f3ead99f4659d94ab9c7/report/index.html

morningman

LGTM

Followup: #31128 This optimization allows doris to correctly read struct type data after changing the schema from hive. ## Changing struct schema in hive: ```sql hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet; hive> insert into struct_test values > (1, named_struct('f1', 1, 'f2', 's1')), > (2, named_struct('f1', 2, 'f2', 's2')), > (3, named_struct('f1', 3, 'f2', 's3')); hive> alter table struct_test change sf sf struct<f1:int, f3:string>; hive> select * from struct_test; OK 1 {"f1":1,"f3":null} 2 {"f1":2,"f3":null} 3 {"f1":3,"f3":null} Time taken: 5.298 seconds, Fetched: 3 row(s) ``` The previous result of doris was: ```sql mysql> select * from struct_test; +------+-----------------------+ | id | sf | +------+-----------------------+ | 1 | {"f1": 1, "f3": "s1"} | | 2 | {"f1": 2, "f3": "s2"} | | 3 | {"f1": 3, "f3": "s3"} | +------+-----------------------+ ``` Now the result is same as hive: ```sql mysql> select * from struct_test; +------+-----------------------+ | id | sf | +------+-----------------------+ | 1 | {"f1": 1, "f3": null} | | 2 | {"f1": 2, "f3": null} | | 3 | {"f1": 3, "f3": null} | +------+-----------------------+ ```

Followup: apache#31128 This optimization allows doris to correctly read struct type data after changing the schema from hive. ## Changing struct schema in hive: ```sql hive> create table struct_test(id int,sf struct<f1: int, f2: string>) stored as parquet; hive> insert into struct_test values > (1, named_struct('f1', 1, 'f2', 's1')), > (2, named_struct('f1', 2, 'f2', 's2')), > (3, named_struct('f1', 3, 'f2', 's3')); hive> alter table struct_test change sf sf struct<f1:int, f3:string>; hive> select * from struct_test; OK 1 {"f1":1,"f3":null} 2 {"f1":2,"f3":null} 3 {"f1":3,"f3":null} Time taken: 5.298 seconds, Fetched: 3 row(s) ``` The previous result of doris was: ```sql mysql> select * from struct_test; +------+-----------------------+ | id | sf | +------+-----------------------+ | 1 | {"f1": 1, "f3": "s1"} | | 2 | {"f1": 2, "f3": "s2"} | | 3 | {"f1": 3, "f3": "s3"} | +------+-----------------------+ ``` Now the result is same as hive: ```sql mysql> select * from struct_test; +------+-----------------------+ | id | sf | +------+-----------------------+ | 1 | {"f1": 1, "f3": null} | | 2 | {"f1": 2, "f3": null} | | 3 | {"f1": 3, "f3": null} | +------+-----------------------+ ```

suxiaogang223 marked this pull request as draft March 19, 2024 02:58

AshinGau reviewed Mar 19, 2024
View reviewed changes

suxiaogang223 force-pushed the hive_complex_type_change branch from 1e51e30 to a5cdbc5Compare March 20, 2024 03:08

suxiaogang223 force-pushed the hive_complex_type_change branch from a5cdbc5 to ad3d61bCompare March 20, 2024 07:31

suxiaogang223 marked this pull request as ready for review March 20, 2024 09:14

AshinGau approved these changes Mar 21, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2024

github-actionsBot added the reviewed label Mar 21, 2024

zy-kkk approved these changes Mar 21, 2024
View reviewed changes

suxiaogang223 force-pushed the hive_complex_type_change branch from 239de18 to 0f8ee1bCompare March 21, 2024 03:08

suxiaogang223 added 2 commits March 21, 2024 14:07

support alter struct type
bfd2458
impl insert null when struct's field is missing check field of sturct must be nullable fix get_rep_level and get_rep_level of StructColumnReader

add _read_column_names to StructColumnReader
04e3a36

suxiaogang223 force-pushed the hive_complex_type_change branch from 0f8ee1b to 04e3a36Compare March 21, 2024 06:07

morningman approved these changes Mar 22, 2024
View reviewed changes

morningman merged commit d584532 into apache:masterMar 22, 2024

suxiaogang223 deleted the hive_complex_type_change branch March 22, 2024 03:24

Uh oh!

Conversation

suxiaogang223 commented Mar 19, 2024• edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Changing struct schema in hive:

Uh oh!

doris-robot commented Mar 19, 2024

Uh oh!

github-actionsBot commented Mar 19, 2024

Uh oh!

github-actionsBot commented Mar 19, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Mar 19, 2024

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

github-actionsBot commented Mar 20, 2024

Uh oh!

AshinGau commented Mar 21, 2024

Uh oh!

github-actionsBot commented Mar 21, 2024

Uh oh!

github-actionsBot commented Mar 21, 2024

Uh oh!

zy-kkk commented Mar 21, 2024

Uh oh!

github-actionsBot commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

zy-kkk commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

suxiaogang223 commented Mar 21, 2024

Uh oh!

github-actionsBot commented Mar 21, 2024

Uh oh!

suxiaogang223 commented Mar 21, 2024

Uh oh!

doris-robot commented Mar 21, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

suxiaogang223 commented Mar 19, 2024•
edited by morningman
Loading