Uh oh!
There was an error while loading. Please reload this page.
ARROW-11561: [Rust][DataFusion] Add Send + Sync to MemTable::load - #9448
Closed
seddonm1 wants to merge 1 commit into
Closed
ARROW-11561: [Rust][DataFusion] Add Send + Sync to MemTable::load#9448seddonm1 wants to merge 1 commit into
seddonm1 wants to merge 1 commit into
Conversation
codecov-io
commented
Feb 8, 2021
Codecov Report
@@ Coverage Diff @@## master #9448 +/- ##
==========================================
- Coverage 82.14% 82.09% -0.06%
==========================================
Files 232 233 +1 Lines 54150 54371 +221 ==========================================
+ Hits 44484 44637 +153 - Misses 9666 9734 +68
Continue to review full report at Codecov.
|
alamb
reviewed
Feb 9, 2021
alamb
left a comment
Contributor
There was a problem hiding this comment.
I think this is a good idea, though it might cause a small breaking API change.
| /// Create a mem table by reading from another data source | ||
| pub async fn load( | ||
| t: &dyn TableProvider, | ||
| t: Box<dyn TableProvider + Send + Sync>, |
Contributor
There was a problem hiding this comment.
Suggested change
| t:Box<dyn TableProvider + Send + Sync>, | |
| t:Arc<dyn TableProvider + Send + Sync>, |
I suggest using Arc to follow the convention in ExecutionContext:
https://github.com/apache/arrow/blob/master/rust/datafusion/src/execution/context.rs#L565
I don't think there is any reason load needs exclusive ownership over the TableProvider
ContributorAuthor
There was a problem hiding this comment.
Thanks @alamb .
You make a good point and it is confusing to me why Box is used in some places and Arc in others (given the I thought ideal characteristics of Arc): e.g.
/// Execution context for registering data sources and executing queries#[derive(Clone)]pubstructExecutionContextState{/// Data sources that are registered with the contextpubdatasources:HashMap<String,Arc<dynTableProvider + Send + Sync>>,vs
pubfn register_table(&mutself,name:&str,provider:Box<dynTableProvider + Send + Sync>,Contributor
There was a problem hiding this comment.
I can take a shot at cleaning up all the APIs to take an Arc over the next few days. I think that would be the ideal outcome.
ContributorAuthor
There was a problem hiding this comment.
Great. I did replace everything with Arc locally for testing and tests pass at least.
alamb
approved these changes
Feb 10, 2021
sgnkc pushed a commit
to sgnkc/arrow
that referenced
this pull request
Feb 17, 2021
This PR adds Send + Sync to the MemTable::load method to allow implementation of a `persist` method like Spark's Dataframe in an async function. Closesapache#9448 from seddonm1/send-sync Authored-by: Mike Seddon <seddonm1@gmail.com> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
jorgecarleitao pushed a commit
that referenced
this pull request
Feb 17, 2021
…r> rather than Box and Arc NOTE: This is a backwards incompatible change in DataFusion Inspired by a conversation with @seddonm1#9448 (comment) and #9445 as well as some upcoming needs in IOx (a consumer of DataFusion) # Rationale: * No `TableProvider` APIs actually require ownership of the `TableProvider` (they all take `&self`) * Internally DataFusion was storing the TableProvider as an Arc already and inconsistently uses `Box`d and `Arc`d table providers (e.g. in [`LogicalPlan::TableScan`](https://github.com/apache/arrow/blob/437c9173c3e067712eb714c643ca839acc7ed7f6/rust/datafusion/src/logical_plan/plan.rs#L125)) * This change allows the same `TableProvider` instance to be reused easily for different `ExecutionContext`s # Changes * Change all uses of `TableProvider` to be wrapped in `Arc` rather than `Box` Closes#9487 from alamb/alamb/arcd_table_provider Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
alamb added a commit
to apache/arrow-rs
that referenced
this pull request
Apr 20, 2021
…r> rather than Box and Arc NOTE: This is a backwards incompatible change in DataFusion Inspired by a conversation with @seddonm1apache/arrow#9448 (comment) and apache/arrow#9445 as well as some upcoming needs in IOx (a consumer of DataFusion) # Rationale: * No `TableProvider` APIs actually require ownership of the `TableProvider` (they all take `&self`) * Internally DataFusion was storing the TableProvider as an Arc already and inconsistently uses `Box`d and `Arc`d table providers (e.g. in [`LogicalPlan::TableScan`](https://github.com/apache/arrow/blob/f055d5e8ee8c6065f38d8351e3f668a43358cd98/rust/datafusion/src/logical_plan/plan.rs#L125)) * This change allows the same `TableProvider` instance to be reused easily for different `ExecutionContext`s # Changes * Change all uses of `TableProvider` to be wrapped in `Arc` rather than `Box` Closes#9487 from alamb/alamb/arcd_table_provider Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Jorge C. Leitao <jorgecarleitao@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds Send + Sync to the MemTable::load method to allow implementation of a
persistmethod like Spark's Dataframe in an async function.