Skip to content

ARROW-6964: [C++][Dataset] Add multithread support to Scanner::ToTable - #5721

Closed
fsaintjacques wants to merge 1 commit into
apache:masterfrom
fsaintjacques:ARROW-6964-parallel-scanner-to-table
Closed

ARROW-6964: [C++][Dataset] Add multithread support to Scanner::ToTable#5721
fsaintjacques wants to merge 1 commit into
apache:masterfrom
fsaintjacques:ARROW-6964-parallel-scanner-to-table

Conversation

@fsaintjacques

@fsaintjacquesfsaintjacques commented Oct 23, 2019•
edited
Loading

Copy link
Copy Markdown
Contributor

The caller may request a parallel construction of the table. Scanner was refactored to own the ScanOptions and ScanContext members.

fsaintjacques
fsaintjacques commented Oct 23, 2019
View reviewed changes
Comment threadcpp/src/arrow/dataset/scanner.cc Outdated

@fsaintjacquesfsaintjacquesOct 23, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer the explicit loop than nested Visitor.

fsaintjacques
fsaintjacques commented Oct 23, 2019
View reviewed changes
Comment threadcpp/src/arrow/dataset/scanner.h Outdated

@fsaintjacquesfsaintjacquesOct 23, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it's time to expose a common ResourceContext class that has a MemoryPool and a ThreadPool?

@github-actions

github-actionsBot commented Oct 23, 2019

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-6964

pitrou
pitrou reviewed Oct 24, 2019
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some comments.

Comment threadcpp/src/arrow/dataset/scanner.cc Outdated

@pitroupitrouOct 24, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's useful to make this inline.

Comment threadcpp/src/arrow/dataset/scanner.cc Outdated
Comment threadcpp/src/arrow/dataset/scanner.h Outdated
Comment threadcpp/src/arrow/dataset/scanner.cc Outdated
bkietz
bkietz requested changes Oct 24, 2019
View reviewed changes

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few small comments

Comment threadcpp/src/arrow/dataset/scanner_test.cc Outdated
Comment threadcpp/src/arrow/dataset/scanner_test.cc Outdated
Comment threadcpp/src/arrow/dataset/scanner.h Outdated

@bkietzbkietzOct 24, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this should be a task group instead of a thread pool. Then users can pass a serial task group to signal single threaded operation

Comment threadcpp/src/arrow/dataset/scanner.cc Outdated

@bkietzbkietzOct 24, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
batches.emplace_back(batch);
batches.emplace_back(std::move(batch));

Comment threadcpp/src/arrow/dataset/scanner.cc Outdated
Comment threadcpp/src/arrow/dataset/scanner.cc Outdated
Comment threadcpp/src/arrow/dataset/scanner.cc Outdated
Comment threadcpp/src/arrow/dataset/scanner.cc Outdated
@fsaintjacques
fsaintjacquesforce-pushed the ARROW-6964-parallel-scanner-to-table branch from 8d6e72e to 10ed013CompareOctober 24, 2019 16:02
@fsaintjacques
ARROW-6964: [C++][Dataset] Add multithread support to Scanner::ToTable
210f190
The caller may request a parallel construction of the table. Scanner was
refactor to own the ScanOptions and ScanContext members.
The `use_threads` options was added to ScanOptions so the caller can
indicate if Scanner is allowed to use parallelism.
@fsaintjacques
fsaintjacquesforce-pushed the ARROW-6964-parallel-scanner-to-table branch from 10ed013 to 210f190CompareOctober 24, 2019 17:27
@asfimportasfimport mentioned this pull request Apr 10, 2020
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitroupitroupitrou left review comments
@bkietzbkietzbkietz requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fsaintjacques@bkietz@pitrou