Skip to content

[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.(#40225 , #40888 ,#41386 ) - #41610

Merged
morningman merged 6 commits into
apache:branch-2.1from
hubgeter:pick_21_mc_storage_api
Oct 11, 2024
Merged

[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.(#40225 , #40888 ,#41386 )#41610
morningman merged 6 commits into
apache:branch-2.1from
hubgeter:pick_21_mc_storage_api

Conversation

@hubgeter

@hubgeterhubgeter commented Oct 9, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

bp #40225 , #40888 ,#41386

Proposed changes

Among them, #40225 is the new api of mc,
#40888 is used to fix the bug when reading null between the new and old apis,
#41386 is used for compatibility between the new and old versions

@hubgeter
[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage…
1c00e65
… API. (apache#40225)
Refactoring maxCompute catalog using Storage API.
Storage API :
https://help.aliyun.com/zh/maxcompute/user-guide/open-storage-sample-java-sdk?spm=a2c4g.11186623.0.i0
```
The following are required:
CREATE CATALOG mc PROPERTIES (
"type" = "max_compute",
"mc.default.project" = "xxx",
"mc.access_key" = "xxx",
"mc.secret_key" = "xxxx",
"mc.endpoint" = "xxxx"
);
Optional parameters:
Configuration Item Default Value
"mc.quota" = "pay-as-you-go"
"mc.split_strategy" = "byte_size" Split according to file size
"mc.split_byte_size" = "268435456" You can set the file size of each split
"mc.split_strategy" = "row_count" Split according to the number of rows of data
"mc.split_row_count" = "1048576" You can set how many lines to read for each split
```
@hubgeter
[fix](maxCompute)Fixed the bug that extra nulls are read when reading…
bff8d73
… maxcompute. (apache#40888)
before pr apache#40225 ## Proposed changes
Fixed a bug where when reading maxcompute, if there are null values ​​in
a batch, null values ​​will always be read out.
@hubgeter
[Enhancement](maxCompute)Implement compatibility with existing maxcom…
d212a98
…pute catalogs from previous versions. (apache#41386)
before pr apache#40225
@doris-robot

doris-robot commented Oct 9, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hubgeter

hubgeter commented Oct 9, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Oct 9, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Oct 9, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.09% (9340/25879)
Line Coverage: 27.66% (76725/277358)
Region Coverage: 26.44% (39376/148902)
Branch Coverage: 23.22% (20028/86240)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d212a980bf517b398c0ad5dbbf2e7d3aa0250ae0_d212a980bf517b398c0ad5dbbf2e7d3aa0250ae0/report/index.html

@hubgeter
hubgeterforce-pushed the pick_21_mc_storage_api branch from 68a2fee to d1961ffCompareOctober 10, 2024 06:28
@hubgeter

hubgeter commented Oct 10, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Oct 10, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Oct 10, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Oct 10, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.08% (9338/25880)
Line Coverage: 27.66% (76710/277367)
Region Coverage: 26.44% (39371/148905)
Branch Coverage: 23.23% (20030/86242)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d1961ff1b8f71a1956a893230583df6bff8df741_d1961ff1b8f71a1956a893230583df6bff8df741/report/index.html

@hubgeter
hubgeterforce-pushed the pick_21_mc_storage_api branch from d1961ff to a5dcbfdCompareOctober 10, 2024 09:09
@hubgeter

hubgeter commented Oct 10, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Oct 10, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Oct 10, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.09% (9340/25880)
Line Coverage: 27.65% (76690/277367)
Region Coverage: 26.44% (39377/148905)
Branch Coverage: 23.23% (20031/86242)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a5dcbfd692ed333f5a87901f4f64159abfe66baa_a5dcbfd692ed333f5a87901f4f64159abfe66baa/report/index.html

@morningman
morningman merged commit 8c0f73c into apache:branch-2.1Oct 11, 2024
@yiguoleiyiguolei mentioned this pull request Nov 6, 2024
Closed
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 20, 2024
@hubgeter
Revert "[Enhancement](MaxCompute)Refactoring maxCompute catalog using…
edab727
… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)"
This reverts commit 8c0f73c.
hubgeter added a commit to hubgeter/doris that referenced this pull request Nov 20, 2024
@hubgeter
Revert "[Enhancement](MaxCompute)Refactoring maxCompute catalog using…
b551c95
… Storage API.(apache#40225 , apache#40888 ,apache#41386 ) (apache#41610)"
This reverts commit 8c0f73c.
@yiguoleiyiguolei mentioned this pull request Jan 19, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

meta-change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hubgeter@doris-robot@morningman