Skip to content

[fix](remote scan) Fix remote scan paralism - #42633

Merged
yiguolei merged 4 commits into
apache:branch-2.1from
RoanHeNaN:fix-remote-scan
Oct 31, 2024
Merged

[fix](remote scan) Fix remote scan paralism#42633
yiguolei merged 4 commits into
apache:branch-2.1from
RoanHeNaN:fix-remote-scan

Conversation

@RoanHeNaN

@RoanHeNaNRoanHeNaN commented Oct 28, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Introduced by #41273.

The modification of _max_scan_thread_num is a long story. So, the modification i will describe only contains three timestamp. 1. Before #41273; 2. After #41273; 3. After this pr.

We have two dimension, so i will use a table. The value of the table is the _max_scan_thread_num when queries running serially.

C refers to num of cpu cores.

  1. Before 41273
EngineOLAPScanOperatorFileScanOperator
PipelineXmin(2C, num_of_scanner)4
Old-Pipeline44
Non-Pipeline44
  1. After 41273
EngineOLAPScanOperatorFileScanOperator
PipelineXmin(2C, num_of_scanner)min(2C, num_of_scanner)
Old-Pipeline44
Non-Pipeline44
  1. After this pr
EngineOLAPScanOperatorFileScanOperator
PipelineXmin(2C, num_of_scanner)4
Old-Pipeline44
Non-Pipeline44

@doris-robot

doris-robot commented Oct 28, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@RoanHeNaN

RoanHeNaN commented Oct 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

1 similar comment
@RoanHeNaN

RoanHeNaN commented Oct 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Oct 28, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Oct 28, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@RoanHeNaN

RoanHeNaN commented Oct 28, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Oct 28, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@RoanHeNaN

RoanHeNaN commented Oct 29, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Oct 29, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Oct 29, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.02% (9368/26005)
Line Coverage: 27.58% (76953/278969)
Region Coverage: 26.37% (39531/149907)
Branch Coverage: 23.15% (20108/86844)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f4d462968f2375c3b54d83e0a72c9065d9ed4f55_f4d462968f2375c3b54d83e0a72c9065d9ed4f55/report/index.html

yiguolei
yiguolei reviewed Oct 31, 2024
View reviewed changes
Comment threadbe/src/vec/exec/scan/scanner_context.h
RuntimeProfile::Counter* _scale_up_scanners_counter = nullptr;
QueryThreadContext _query_thread_context;
bool _ignore_data_distribution = false;
bool _is_file_scan_operator;

@yiguoleiyiguoleiOct 31, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should set default value

@yiguolei
yiguolei merged commit 012c2d7 into apache:branch-2.1Oct 31, 2024
@RoanHeNaN
RoanHeNaN deleted the fix-remote-scan branch October 31, 2024 06:49
@RoanHeNaNRoanHeNaN mentioned this pull request Nov 8, 2024
Merged
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguoleiyiguoleiyiguolei left review comments
@Gabriel39Gabriel39Awaiting requested review from Gabriel39

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@RoanHeNaN@doris-robot@yiguolei