[feature-wip](arrow-flight)(step2) FE support Arrow Flight server by xinyiZzz · Pull Request #24314 · apache/doris

xinyiZzz · 2023-09-13T10:29:52Z

Proposed changes

Design Documentation Linked to #25514

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

xinyiZzz · 2023-09-13T10:30:12Z

run buildall

github-actions · 2023-09-13T10:36:03Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-13T11:30:41Z

TeamCity be ut coverage result:
Function Coverage: 37.01% (7968/21530)
Line Coverage: 28.96% (63852/220515)
Region Coverage: 27.91% (33158/118821)
Branch Coverage: 24.45% (17005/69552)
Coverage Report: http://coverage.selectdb-in.cc/coverage/809bf307f36428f4c22eb9d4c0edf2a1d39f614a_809bf307f36428f4c22eb9d4c0edf2a1d39f614a/report/index.html

xinyiZzz · 2023-09-13T12:35:09Z

run buildall

github-actions · 2023-09-13T12:37:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-13T12:52:08Z

TeamCity be ut coverage result:
Function Coverage: 37.00% (7966/21530)
Line Coverage: 28.94% (63824/220515)
Region Coverage: 27.89% (33137/118821)
Branch Coverage: 24.44% (16996/69552)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2b5f04db10f781718b866c7ea71e4dbf31df1c32_2b5f04db10f781718b866c7ea71e4dbf31df1c32/report/index.html

doris-robot · 2023-09-13T13:17:04Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.9 seconds
stream load tsv: 608 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 63 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162461852 Bytes

xinyiZzz · 2023-09-13T15:13:23Z

run buildall

github-actions · 2023-09-13T15:19:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-13T15:44:11Z

TeamCity be ut coverage result:
Function Coverage: 36.95% (7966/21560)
Line Coverage: 28.92% (63821/220712)
Region Coverage: 27.86% (33136/118929)
Branch Coverage: 24.42% (16998/69600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/101e138a9e8eee49684676380674cf8373444b9e_101e138a9e8eee49684676380674cf8373444b9e/report/index.html

doris-robot · 2023-09-13T15:48:30Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.33 seconds
stream load tsv: 615 seconds loaded 74807831229 Bytes, about 116 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162190748 Bytes

xinyiZzz · 2023-09-13T16:33:03Z

run buildall

github-actions · 2023-09-13T16:40:29Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-13T16:59:23Z

TeamCity be ut coverage result:
Function Coverage: 36.95% (7966/21560)
Line Coverage: 28.92% (63824/220712)
Region Coverage: 27.87% (33147/118929)
Branch Coverage: 24.43% (17000/69600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f8ef6df883c7aa0093f083c0ec0c4a406ee5a5c7_f8ef6df883c7aa0093f083c0ec0c4a406ee5a5c7/report/index.html

doris-robot · 2023-09-13T17:15:18Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.86 seconds
stream load tsv: 609 seconds loaded 74807831229 Bytes, about 117 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162362325 Bytes

github-actions · 2023-09-15T10:58:20Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-18T02:50:41Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2023-09-19T16:33:14Z

run buildall

github-actions · 2023-09-19T16:39:23Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-19T17:06:48Z

TeamCity be ut coverage result:
Function Coverage: 36.74% (7979/21716)
Line Coverage: 28.81% (64035/222259)
Region Coverage: 27.71% (33234/119945)
Branch Coverage: 24.34% (17034/69994)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cebb67f983c2ba48bfd1be5414d96e53b6eb4c0a_cebb67f983c2ba48bfd1be5414d96e53b6eb4c0a/report/index.html

doris-robot · 2023-09-19T17:27:31Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.61 seconds
stream load tsv: 595 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17162404293 Bytes

yiguolei

LGTM

Gabriel39

LGTM

github-actions · 2023-09-20T06:40:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-20T06:40:49Z

PR approved by anyone and no changes requested.

…ache#24314) This is a POC, the design documentation will be updated soon

* Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField` * Introduced by apache#24314

#30337) * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField` * Introduced by #24314

* Similar to apache#30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by apache#24314

…#30441) * Similar to #30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by #24314

…#30433) * Similar to #30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by #24314

apache#30337) * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `FrontendHbResponse.readField` * Introduced by apache#24314

…apache#30433) * Similar to apache#30337 * Between branch-1.2-lts and branch-2.1, `FrontendHbResponse` has upgrade compatiblity problem, because `arrowFlightSqlPort` field, only metaVerserion less than FeMetaVersion.VERSION_121 will call `BackendHbResponse.readField` * Introduced by apache#24314

xinyiZzz changed the title ~~[feature-wip] (arrow-flight) (step2) FE support Arrow Flight server~~[feature-wip](arrow-flight)(step2) FE support Arrow Flight serverSep 13, 2023

xinyiZzz force-pushed the 20230911_support_arrow_flight_sql5 branch from 2b5f04d to 101e138Compare September 13, 2023 15:13

yiguolei reviewed Sep 14, 2023
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/common/proc/BackendsProcDir.java Outdated

xinyiZzz marked this pull request as draft September 15, 2023 07:18

yiguolei reviewed Sep 16, 2023
View reviewed changes

Comment threadbe/src/runtime/buffer_control_block.cpp

yiguolei reviewed Sep 16, 2023
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/qe/QeService.java

yiguolei reviewed Sep 16, 2023
View reviewed changes

Comment threadgensrc/proto/internal_service.proto Outdated

yiguolei reviewed Sep 16, 2023
View reviewed changes

Comment threadgensrc/proto/internal_service.proto Outdated

yiguolei reviewed Sep 16, 2023
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightStatementContext.java Outdated

xinyiZzz force-pushed the 20230911_support_arrow_flight_sql5 branch from 3524285 to 14098ceCompare September 18, 2023 02:42

xinyiZzz added 3 commits September 20, 2023 00:32

5
c9e70cc

6
141f403

8
cebb67f

xinyiZzz force-pushed the 20230911_support_arrow_flight_sql5 branch from c8e01a6 to cebb67fCompare September 19, 2023 16:32

yiguolei approved these changes Sep 20, 2023
View reviewed changes

Gabriel39 approved these changes Sep 20, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2023

github-actionsBot added the reviewed label Sep 20, 2023

yiguolei merged commit fc12362 into apache:masterSep 20, 2023

JingDas pushed a commit to JingDas/doris that referenced this pull request Sep 20, 2023
[feature-wip](arrow-flight)(step2) FE support Arrow Flight server (ap…
146a8b3
…ache#24314) This is a POC, the design documentation will be updated soon

xinyiZzz mentioned this pull request Oct 17, 2023
[Feature] Doris support Arrow Flight SQL protocol #25514
Closed
1 task

swjtu-zhanglei mentioned this pull request Jan 24, 2024
[fix](fe) Fix FrontendHbResponse serialization compatibility problem #30337
Merged

swjtu-zhanglei mentioned this pull request Jan 26, 2024
[fix](fe) Fix BackendHbResponse serialization compatibility problem #30433
Merged

swjtu-zhanglei mentioned this pull request Jan 26, 2024
[fix](fe) Fix BackendHbResponse serialization compatibility problem #30441
Merged

Uh oh!

Conversation

xinyiZzz commented Sep 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

xinyiZzz commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

xinyiZzz commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

xinyiZzz commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

xinyiZzz commented Sep 13, 2023

Uh oh!

github-actionsBot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

doris-robot commented Sep 13, 2023

Uh oh!

Uh oh!

github-actionsBot commented Sep 15, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

xinyiZzz commented Sep 19, 2023

Uh oh!

github-actionsBot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 20, 2023

Uh oh!

github-actionsBot commented Sep 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xinyiZzz commented Sep 13, 2023•
edited
Loading