[fix](multi-catalog)fix maxcompute partition filter and session creation by wsjz · Pull Request #24911 · apache/doris

wsjz · 2023-09-26T07:45:59Z

Proposed changes

add maxcompute partition support
fix maxcompute partition filter
modify maxcompute session create method

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-10-15T10:23:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-15T11:11:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-15T11:11:57Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-15T11:28:13Z

clang-tidy review says "All clean, LGTM! 👍"

morningman · 2023-10-15T15:09:46Z

 }
 // reorder columns
 List<Column> columnList = curTableScan.getSchema().getColumns();
+ columnList.addAll(curTableScan.getSchema().getPartitionColumns());


Does curTableScan.getSchema().getColumns() contains partition columns?

not contains

morningman · 2023-10-15T15:11:25Z

+ .map(Column::getName)
+ .collect(Collectors.toSet());
+ List<Column> maxComputeColumns = new ArrayList<>(readColumns);
+ maxComputeColumns.removeIf(e -> partitionColumns.contains(e.getName()));


Why remove partition columns here?

max compute depart the partition columns from common columns

github-actions · 2023-10-16T07:21:18Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-16T07:59:52Z

clang-tidy review says "All clean, LGTM! 👍"

wsjz · 2023-10-16T09:24:18Z

run buildall

github-actions · 2023-10-16T09:30:16Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-16T09:38:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-16T10:30:16Z

TeamCity be ut coverage result:
Function Coverage: 36.70% (8230/22424)
Line Coverage: 28.85% (65958/228624)
Region Coverage: 27.55% (34204/124161)
Branch Coverage: 24.26% (17413/71782)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7335b80d9e94da758383965b91e236ae498674e0_7335b80d9e94da758383965b91e236ae498674e0/report/index.html

wsjz · 2023-10-16T10:34:22Z

run buildall

doris-robot · 2023-10-16T10:54:08Z

TeamCity be ut coverage result:
Function Coverage: 36.70% (8230/22424)
Line Coverage: 28.85% (65965/228624)
Region Coverage: 27.56% (34217/124161)
Branch Coverage: 24.27% (17418/71782)
Coverage Report: http://coverage.selectdb-in.cc/coverage/294432fe5af2126cf782cadcbfeebeb3961ef605_294432fe5af2126cf782cadcbfeebeb3961ef605/report/index.html

doris-robot · 2023-10-16T11:30:12Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.72 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162245617 Bytes

morningman

LGTM

github-actions · 2023-10-16T15:09:28Z

clang-tidy review says "All clean, LGTM! 👍"

morningman · 2023-10-17T01:53:51Z

run buildall

doris-robot · 2023-10-17T03:04:26Z

TeamCity be ut coverage result:
Function Coverage: 37.02% (8285/22378)
Line Coverage: 29.06% (66445/228664)
Region Coverage: 27.74% (34477/124265)
Branch Coverage: 24.37% (17518/71886)
Coverage Report: http://coverage.selectdb-in.cc/coverage/166acc2d1fd5c29c151a7272d78b59851062aded_166acc2d1fd5c29c151a7272d78b59851062aded/report/index.html

doris-robot · 2023-10-17T03:09:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.48 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162037094 Bytes

morningman

LGTM

github-actions · 2023-10-17T14:11:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-17T14:11:22Z

PR approved by anyone and no changes requested.

zhangguoqiang666

LGTM

…ion (#24911) add maxcompute partition support fix maxcompute partition filter modify maxcompute session create method

…ion (apache#24911) add maxcompute partition support fix maxcompute partition filter modify maxcompute session create method

wsjz marked this pull request as ready for review October 15, 2023 11:03

morningman reviewed Oct 15, 2023
View reviewed changes

wm1581066 added the dev/2.0.3 label Oct 16, 2023

wm1581066 added the usercase Important user case type label label Oct 16, 2023

morningman approved these changes Oct 16, 2023
View reviewed changes

wsjz added 2 commits October 16, 2023 22:59

[fix](multi-catalog)fix maxcompute parttiion filter and session creation
42b624a

1
166acc2

morningman approved these changes Oct 17, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 17, 2023

github-actionsBot added the reviewed label Oct 17, 2023

zhangguoqiang666 approved these changes Oct 17, 2023
View reviewed changes

morningman merged commit 18c2a13 into apache:masterOct 17, 2023

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Oct 18, 2023

xiaokang pushed a commit that referenced this pull request Oct 18, 2023
[fix](multi-catalog)fix maxcompute partition filter and session creat…
6e5082e
…ion (#24911) add maxcompute partition support fix maxcompute partition filter modify maxcompute session create method

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
[fix](multi-catalog)fix maxcompute partition filter and session creat…
9bf687d
…ion (apache#24911) add maxcompute partition support fix maxcompute partition filter modify maxcompute session create method

xiaokang mentioned this pull request Dec 4, 2023
Release Note 2.0.3 #27909
Closed

Uh oh!

Conversation

wsjz commented Sep 26, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actionsBot commented Oct 15, 2023

Uh oh!

github-actionsBot commented Oct 15, 2023

Uh oh!

github-actionsBot commented Oct 15, 2023

Uh oh!

github-actionsBot commented Oct 15, 2023

Uh oh!

morningmanOct 15, 2023

Choose a reason for hiding this comment

Uh oh!

wsjzOct 16, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morningmanOct 15, 2023

Choose a reason for hiding this comment

Uh oh!

wsjzOct 16, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Oct 16, 2023

Uh oh!

github-actionsBot commented Oct 16, 2023

Uh oh!

wsjz commented Oct 16, 2023

Uh oh!

github-actionsBot commented Oct 16, 2023

Uh oh!

github-actionsBot commented Oct 16, 2023

Uh oh!

doris-robot commented Oct 16, 2023

Uh oh!

wsjz commented Oct 16, 2023

Uh oh!

doris-robot commented Oct 16, 2023

Uh oh!

doris-robot commented Oct 16, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 16, 2023

Uh oh!

morningman commented Oct 17, 2023

Uh oh!

doris-robot commented Oct 17, 2023

Uh oh!

doris-robot commented Oct 17, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 17, 2023

Uh oh!

github-actionsBot commented Oct 17, 2023

Uh oh!

zhangguoqiang666 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

wsjz commented Sep 26, 2023•
edited
Loading

wsjzOct 16, 2023•
edited
Loading