Skip to content

ARROW-12597: [C++] Enable per-row-group parallelism in async Parquet reader - #10482

Closed
lidavidm wants to merge 6 commits into
apache:masterfrom
lidavidm:arrow-12597
Closed

ARROW-12597: [C++] Enable per-row-group parallelism in async Parquet reader#10482
lidavidm wants to merge 6 commits into
apache:masterfrom
lidavidm:arrow-12597

Conversation

@lidavidm

@lidavidmlidavidm commented Jun 8, 2021

Copy link
Copy Markdown
Member

This adds an OptionalParallelForAsync which lets us have per-row-group parallelism without nested parallelism in the async Parquet reader. This also uses TransferAlways, taking care of ARROW-12916. enable_parallel_column_conversion is kept as it still affects the threaded scanner.

@github-actions

github-actionsBot commented Jun 8, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12597

@lidavidm

lidavidm commented Jun 8, 2021

Copy link
Copy Markdown
MemberAuthor

S3 Median Scan Time (s)(2)

Not much difference in a benchmark; the most pronounced change is when files << cores (this was a 4 vcpu machine), which I think makes sense since with many files, file-level parallelism takes hold.

pitrou
pitrou reviewed Jun 10, 2021
View reviewed changes
Comment threadcpp/src/arrow/util/parallel.h Outdated
Comment threadcpp/src/arrow/util/parallel.h Outdated
Comment threadcpp/src/parquet/arrow/reader.cc Outdated
Comment threadcpp/src/parquet/arrow/reader.cc Outdated
Comment threadcpp/src/parquet/arrow/reader.cc Outdated
Comment threadcpp/src/parquet/arrow/reader.cc Outdated
Comment threadcpp/src/parquet/arrow/reader.cc Outdated
@pitrou
pitrou self-requested a review June 15, 2021 13:53
pitrou
pitrou approved these changes Jun 15, 2021
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thank you very much!

@pitrou

pitrou commented Jun 15, 2021

Copy link
Copy Markdown
Member

Rebased, can merge if green.

@lidavidm
lidavidm deleted the arrow-12597 branch June 15, 2021 15:22
sjperkins pushed a commit to sjperkins/arrow that referenced this pull request Jun 23, 2021
@lidavidm@sjperkins
ARROW-12597: [C++] Enable per-row-group parallelism in async Parquet …
522696b
…reader
This adds an OptionalParallelForAsync which lets us have per-row-group parallelism without nested parallelism in the async Parquet reader. This also uses TransferAlways, taking care of ARROW-12916. `enable_parallel_column_conversion` is kept as it still affects the threaded scanner.
Closesapache#10482 from lidavidm/arrow-12597
Authored-by: David Li <li.davidm96@gmail.com>
Signed-off-by: David Li <li.davidm96@gmail.com>
This was referenced Jun 15, 2021
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Parquet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lidavidm@pitrou