Skip to content

[Feature][Stream Load] Add "show stream load" to show stream load record - #5488

Merged
yangzhg merged 1 commit into
apache:masterfrom
weizuo93:show-stream-load
Apr 29, 2021
Merged

[Feature][Stream Load] Add "show stream load" to show stream load record#5488
yangzhg merged 1 commit into
apache:masterfrom
weizuo93:show-stream-load

Conversation

@weizuo93

@weizuo93weizuo93 commented Mar 9, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Add the sql of show stream load to show the result of recent stream load jobs.

img_58a449aa-9d4b-4bbc-85a8-cbd64fc7659l

Types of changes

What types of changes does your code introduce to Doris?
Put an x in the boxes that apply

  • [] Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • [] Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [] Documentation Update (if none of the other choices apply)
  • Code refactor (Modify the code structure, format the code, etc...)

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

  • [] I have created an issue on (Fix #ISSUE) and described the bug/feature there in detail
  • Compiling and unit tests pass locally with my changes
  • [] I have added tests that prove my fix is effective or that my feature works
  • [] If these changes need document changes, I have updated the document
  • Any dependent changes have been merged

@weizuo93weizuo93 changed the title [Sql][Stream Load] Add show stream load function for [Sql][Stream Load] Add show stream load functionMar 9, 2021
@weizuo93weizuo93 changed the title [Sql][Stream Load] Add show stream load function[Sql][Stream Load] Add "show stream_load" function to show the result of stream load.Mar 9, 2021
@weizuo93weizuo93 changed the title [Sql][Stream Load] Add "show stream_load" function to show the result of stream load.[Sql][Stream Load] Add "show stream_load" function to show the result of stream loadMar 9, 2021
@weizuo93weizuo93 changed the title [Sql][Stream Load] Add "show stream_load" function to show the result of stream load[Feature][Stream Load] Add "show stream_load" function to show the result of stream loadMar 9, 2021
@morningmanmorningman self-assigned this Mar 21, 2021
@morningmanmorningman added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 21, 2021
acelyc111
acelyc111 reviewed Mar 24, 2021
View reviewed changes
Comment threadbe/src/http/action/stream_load.cpp Outdated
Comment threadbe/src/http/action/stream_load.cpp Outdated
Comment threadbe/src/http/action/stream_load.cpp Outdated
Comment threadbe/src/olap/olap_server.cpp Outdated
Comment threadbe/src/runtime/stream_load/stream_load_record.cpp Outdated
Comment threadbe/src/runtime/stream_load/stream_load_record.cpp
Comment threadbe/src/runtime/stream_load/stream_load_record.cpp Outdated
Comment threadbe/src/runtime/stream_load/stream_load_record.h
Comment threadbe/src/olap/storage_engine.cpp Outdated
Comment threadbe/src/runtime/stream_load/stream_load_context.cpp
@kangkaisen

kangkaisen commented Mar 28, 2021

Copy link
Copy Markdown
Contributor

@weizuo93 Hi, Thanks for your contribution.
I don't understand why we need to store the stream load meta both in FE and BE. why we couldn't get stream load info from FE meta?

@weizuo93

weizuo93 commented Mar 30, 2021•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@weizuo93 Hi, Thanks for your contribution.
I don't understand why we need to store the stream load meta both in FE and BE. why we couldn't get stream load info from FE meta?

@kangkaisen Our goal is to store the stream load meta in FE. The reason why the stream load meta is stored on BE first is to send the meta to FE in bulk to avoid calling RPC every time the stream load finished. After a certain period of time, stream load meta will be removed from BE.

@kangkaisen

kangkaisen commented Mar 30, 2021

Copy link
Copy Markdown
Contributor

@weizuo93
Hi, I mean In LoadManager, dbIdToLabelToLoadJobs already has the stream load join info, why we still need to store stream load job meta in BE? in other words, I think we don't need to store stream load job meta in BE.

@weizuo93

weizuo93 commented Mar 30, 2021•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@weizuo93
Hi, I mean In LoadManager, dbIdToLabelToLoadJobs already has the stream load join info, why we still need to store stream load job meta in BE? in other words, I think we don't need to store stream load job meta in BE.

@kangkaisen As far as I know, there is no info about stream load in LoadManager and dbIdToLabelToLoadJobs. LoadManager only manages bulk load.

@kangkaisen

kangkaisen commented Mar 30, 2021

Copy link
Copy Markdown
Contributor

@weizuo93 You are right. I am wrong. This PR could go ahead. (But store some load job meta in FE and store some load job meta in BE, which seems not very good)

@weizuo93

weizuo93 commented Mar 30, 2021•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@weizuo93 You are right. I am wrong. This PR could go ahead. (But store some load job meta in FE and store some load job meta in BE, which seems not very good)

@kangkaisen
Thanks for your comment. There is code overlap between this PR and PR #5452. After discussion with Mingyu Chen, we first push forward PR #5452. About storing stream load meta in BE, it's a tradeoff. If BE reports the stream load result to FE when each stream load job finished, it may result in too much stress for FE RPC. So the stream load results are temporarily stored on BE and then sent to FE by batch. After finished data sending, stream load meta will be removed from BE soon. Do you have any better idea?

@kangkaisen

kangkaisen commented Mar 31, 2021

Copy link
Copy Markdown
Contributor

@weizuo93 Ok, Thanks for your explain. which is OK for me

@weizuo93
weizuo93force-pushed the show-stream-load branch 2 times, most recently from 32232d9 to d039ef2CompareApril 25, 2021 12:27
@yangzhg

yangzhg commented Apr 26, 2021

Copy link
Copy Markdown
Member

show stream load is better, like show routine load

@weizuo93

weizuo93 commented Apr 26, 2021

Copy link
Copy Markdown
ContributorAuthor

show stream load is better, like show routine load

OK, thank you.

@weizuo93weizuo93 changed the title [Feature][Stream Load] Add "show stream_load" function to show the result of stream load[Feature][Stream Load] Add "show stream load" to show stream load recordApr 26, 2021
yangzhg
yangzhg previously approved these changes Apr 27, 2021
View reviewed changes

@yangzhgyangzhg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yangzhgyangzhg added the approved Indicates a PR has been approved by one committer. label Apr 27, 2021
yangzhg
yangzhg reviewed Apr 28, 2021
View reviewed changes
Comment threadfe/fe-core/src/main/java/org/apache/doris/common/Config.java Outdated
yangzhg
yangzhg approved these changes Apr 28, 2021
View reviewed changes

@yangzhgyangzhg left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yangzhg
yangzhg merged commit 9001fd2 into apache:masterApr 29, 2021
EmmyMiao87 pushed a commit to EmmyMiao87/incubator-doris that referenced this pull request May 14, 2021
@weizuo93weizuo@morningman-cmy
support show stream load sql (apache#5488)
6d5a538
Co-authored-by: weizuo <weizuo@xiaomi.com>
@morningmanmorningman mentioned this pull request Oct 10, 2021
Closed
@weizuo93
weizuo93 deleted the show-stream-load branch March 9, 2022 07:04
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@acelyc111acelyc111acelyc111 left review comments
@yangzhgyangzhgyangzhg approved these changes

Assignees

@morningmanmorningman

Labels

approvedIndicates a PR has been approved by one committer.kind/featureCategorizes issue or PR as related to a new feature.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@weizuo93@kangkaisen@yangzhg@acelyc111@morningman