Skip to content

[optimize] Optimize spark load/broker load reading parquet format file - #3878

Merged
morningman merged 8 commits into
apache:masterfrom
xy720:spark_load_parquet_buffer_reader
Jun 23, 2020
Merged

[optimize] Optimize spark load/broker load reading parquet format file#3878
morningman merged 8 commits into
apache:masterfrom
xy720:spark_load_parquet_buffer_reader

Conversation

@xy720

@xy720xy720 commented Jun 16, 2020•
edited
Loading

Copy link
Copy Markdown
Member

Please see the description in #3877 .
spark load #3433

@yiguolei

yiguolei commented Jun 16, 2020

Copy link
Copy Markdown
Contributor

Great job

kangkaisen
kangkaisen reviewed Jun 16, 2020
View reviewed changes
Comment threadbe/src/exec/buffered_reader.h Outdated
@kangkaisen

kangkaisen commented Jun 16, 2020

Copy link
Copy Markdown
Contributor

@xy720 Hi, Please add the UT for new BufferedReader class, Thanks.

morningman
morningman reviewed Jun 16, 2020
View reviewed changes
Comment threadbe/src/exec/buffered_reader.cpp Outdated
Comment threadbe/src/exec/buffered_reader.h Outdated
Comment threadbe/src/exec/buffered_reader.h Outdated
Comment threadbe/src/exec/buffered_reader.h Outdated
Comment threadbe/src/exec/buffered_reader.cpp Outdated
Comment threadbe/src/exec/buffered_reader.cpp
Comment threadbe/src/exec/buffered_reader.cpp
@morningmanmorningman added area/load Issues or PRs related to all kinds of load kind/improvement area/broker Issues or PRs related to broker labels Jun 16, 2020
@xy720
xy720force-pushed the spark_load_parquet_buffer_reader branch from 35f1391 to 9165165CompareJune 19, 2020 02:44
morningman
morningman reviewed Jun 20, 2020
View reviewed changes
Comment threadbe/test/exec/buffered_reader_test.cpp Outdated
Comment threadbe/test/exec/buffered_reader_test.cpp
Comment threadbe/src/exec/buffered_reader.cpp Outdated
morningman
morningman previously approved these changes Jun 22, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningmanmorningman added the approved Indicates a PR has been approved by one committer. label Jun 22, 2020
wutiangan
wutiangan reviewed Jun 22, 2020
View reviewed changes
Comment threadbe/src/exec/broker_reader.cpp Outdated
wutiangan
wutiangan reviewed Jun 22, 2020
View reviewed changes
Comment threadbe/src/exec/buffered_reader.cpp Outdated
morningman
morningman approved these changes Jun 22, 2020
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
morningman merged commit c50a310 into apache:masterJun 23, 2020
@xy720xy720 mentioned this pull request Jul 1, 2020
Closed
@EmmyMiao87EmmyMiao87 mentioned this pull request Aug 17, 2020
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kangkaisenkangkaisenkangkaisen left review comments
@morningmanmorningmanmorningman approved these changes
+1 more reviewer
@wutianganwutianganwutiangan left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.area/brokerIssues or PRs related to brokerarea/loadIssues or PRs related to all kinds of loadkind/improvement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xy720@yiguolei@kangkaisen@morningman@wutiangan