[feature-wip] (arrow-flight) (step1) BE support Arrow Flight server, read data only by xinyiZzz · Pull Request #23765 · apache/doris

xinyiZzz · 2023-09-01T10:08:18Z

Proposed changes

Design Documentation Linked to #25514

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-09-01T10:15:50Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-01T11:21:46Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-09-01T13:20:44Z

run buildall

xinyiZzz · 2023-09-01T13:22:26Z

run buildall

github-actions · 2023-09-01T13:28:23Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-01T13:29:28Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2023-09-01T13:38:03Z

+
+ static arrow::Result<std::shared_ptr<FlightSqlServer>> create();
+
+ arrow::Result<std::unique_ptr<arrow::flight::FlightDataStream>> DoGetStatement(


do_get_statement

DoGetStatement override arrow::flight::sql::FlightSqlServerBase:: DoGetStatement cannot be renamed = =

yiguolei · 2023-09-01T13:41:06Z

 int num_rows = result->result_batch.rows.size();

- while ((!_batch_queue.empty() && _buffer_rows > _buffer_limit) && !_is_cancelled) {
+ while ((!_mysql_batch_queue.empty() && _buffer_rows > _buffer_limit) && !_is_cancelled) {


maybe we need another pr to set buffer limit from session variable. Then for arrow flight we may need change the session variable to get better performance.

yiguolei · 2023-09-01T13:42:11Z

+_mysql_batch_queue.push_back(std::move(result));
 }
 _buffer_rows += num_rows;
- _data_arrival.notify_one();


better not delete this line.
it seems useless but it is very dangerous to delete it.

If add batch the same BufferControlBlock to FE and Arrow Flight at the same time, such as BE informing FE that data sending is completed, this may be a problem.
I restore it first and think about it after debugging with FE.

github-actions · 2023-09-01T14:03:56Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-09-01T14:22:39Z

run buildall

xinyiZzz · 2023-09-01T14:24:18Z

run buildall

github-actions · 2023-09-01T14:32:37Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei

LGTM

github-actions · 2023-09-01T16:58:09Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-01T16:58:12Z

PR approved by anyone and no changes requested.

xinyiZzz · 2023-09-03T17:26:58Z

run buildall

github-actions · 2023-09-03T17:33:50Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-09-04T03:36:28Z

run buildall

github-actions · 2023-09-04T03:42:00Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-04T04:47:07Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.07 seconds
stream load tsv: 531 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162028603 Bytes

Gabriel39

LGTM

github-actions · 2023-09-04T10:01:44Z

PR approved by at least one committer and no changes requested.

yiguolei

LGTM

…server, read data only (apache#23765)" This reverts commit 039c76c.

xinyiZzz marked this pull request as draft September 1, 2023 10:08

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from 7ec077f to 70fcba0Compare September 1, 2023 11:13

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from 70fcba0 to 3f1e6ceCompare September 1, 2023 13:19

xinyiZzz marked this pull request as ready for review September 1, 2023 13:20

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from 3f1e6ce to 0a442b0Compare September 1, 2023 13:22

yiguolei reviewed Sep 1, 2023
View reviewed changes

Comment threadbe/src/runtime/result_buffer_mgr.h Outdated

yiguolei reviewed Sep 1, 2023
View reviewed changes

Comment threadbe/src/service/arrow_flight/flight_sql_statement_batch_reader.cpp Outdated

yiguolei reviewed Sep 1, 2023
View reviewed changes

Comment threadbe/src/common/config.cpp Outdated

yiguolei reviewed Sep 1, 2023
View reviewed changes

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from 0a442b0 to bb0974eCompare September 1, 2023 13:57

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from 968d54d to a620028Compare September 1, 2023 14:24

yiguolei previously approved these changes Sep 1, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2023

github-actionsBot added the reviewed label Sep 1, 2023

xinyiZzz dismissed yiguolei’s stale review via 27f1aa1September 3, 2023 17:26

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from a620028 to 27f1aa1Compare September 3, 2023 17:26

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 3, 2023

xinyiZzz added 4 commits September 4, 2023 11:34

1
51c793a

2
57fc356

3
da258c3

4
e123377

xinyiZzz force-pushed the 20230823_arrow_flight_be3 branch from 27f1aa1 to e123377Compare September 4, 2023 03:34

Gabriel39 approved these changes Sep 4, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2023

yiguolei approved these changes Sep 4, 2023
View reviewed changes

yiguolei merged commit 039c76c into apache:masterSep 4, 2023

Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Sep 4, 2023
Revert "[feature-wip] (arrow-flight) (step1) BE support Arrow Flight …
b02fe8b
…server, read data only (apache#23765)" This reverts commit 039c76c.

Gabriel39 mentioned this pull request Sep 4, 2023
Revert "[feature-wip] (arrow-flight) (step1) BE support Arrow Flight … #23886
Closed

xinyiZzz mentioned this pull request Sep 14, 2023
[Flight-SQL] use Flight_SQL_JDBC to connect server failed apache/arrow#35777
Closed

xinyiZzz mentioned this pull request Oct 17, 2023
[Feature] Doris support Arrow Flight SQL protocol #25514
Closed
1 task


		static arrow::Result<std::shared_ptr<FlightSqlServer>> create();

		arrow::Result<std::unique_ptr<arrow::flight::FlightDataStream>> DoGetStatement(

Uh oh!

Conversation

xinyiZzz commented Sep 1, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

xinyiZzz commented Sep 1, 2023

Uh oh!

xinyiZzz commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

Uh oh!

Uh oh!

yiguoleiSep 1, 2023

Choose a reason for hiding this comment

Uh oh!

xinyiZzzSep 1, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yiguoleiSep 1, 2023

Choose a reason for hiding this comment

Uh oh!

xinyiZzzSep 1, 2023

Choose a reason for hiding this comment

Uh oh!

yiguoleiSep 1, 2023

Choose a reason for hiding this comment

Uh oh!

xinyiZzzSep 1, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

xinyiZzz commented Sep 1, 2023

Uh oh!

xinyiZzz commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

github-actionsBot commented Sep 1, 2023

Uh oh!

xinyiZzz commented Sep 3, 2023

Uh oh!

github-actionsBot commented Sep 3, 2023

Uh oh!

xinyiZzz commented Sep 4, 2023

Uh oh!

github-actionsBot commented Sep 4, 2023

Uh oh!

hello-stephen commented Sep 4, 2023

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 4, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

xinyiZzz commented Sep 1, 2023•
edited
Loading