Uh oh!
There was an error while loading. Please reload this page.
ARROW-16085: [C++][R] InMemoryDataset::ReplaceSchema does not alter scan output - #13088
ARROW-16085: [C++][R] InMemoryDataset::ReplaceSchema does not alter scan output#13088wjones127 wants to merge 5 commits into
Conversation
lidavidm
commented
May 9, 2022
The other way (arguably) would be to have |
| ASSERT_OK_AND_ASSIGN(scanner_builder, dataset->NewScan()); | ||
| ASSERT_OK_AND_ASSIGN(scanner, scanner_builder->Finish()); | ||
| ASSERT_NOT_OK(scanner->ToTable()); |
There was a problem hiding this comment.
Can we have a more explicit check, e.g. with EXPECT_RAISES_WITH_MESSAGE_THAT?
There was a problem hiding this comment.
Yeah it was raising NotImplemented, but I realized we probably would rather raise TypeError, so I added a check for project-ability where the schema consistency check was before.
| auto batch2 = ConstantArrayGenerator::Zeroes(kBatchSize, schema_); | ||
| RecordBatchVector batches{batch1, batch2}; | ||
| auto dataset = std::make_shared<InMemoryDataset>(schema_, batches); |
There was a problem hiding this comment.
Do we actually want this to be valid though? I would expect the batches of a dataset to have a consistent schema
There was a problem hiding this comment.
In file fragments, it's totally normal to have a physical schema that is different from the dataset schema.
This came up when I realized we could create a union dataset out of filesystem ones but not in-memory ones if the schemas differed.
The other way (arguably) would be to have ReplaceSchema project the batches (though that is a lot more work).
I thought about that, but then are we materializing the projected batches before any scan is started? It seems more efficient for the projection to happen as part of the scan.
There was a problem hiding this comment.
Hmm, good point about the fragments.
I was thinking InMemoryDataset already has all the data in memory, so it's not a big deal anyways. But yes, that's unnecessary work compared to this.
| " which did not match InMemorySource's: ", *schema); | ||
| } | ||
| RETURN_NOT_OK(CheckProjectable(*schema, *batch->schema())); |
There was a problem hiding this comment.
It feels like this could be a construction-time check to avoid repeated checking except there is no way to return a Status there, unfortunately. (Not a big deal, though.)
There was a problem hiding this comment.
I thought about that, but would have to change this to a ::Make() method and didn't want to go that far here.
ursabot
commented
May 11, 2022
Benchmark runs are scheduled for baseline = 35119f2 and contender = 5b653ee. 5b653ee is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
…Hub issue numbers (#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338#5198ARROW-6364#5201ARROW-6323#5169ARROW-6278#5141ARROW-6360#5329ARROW-6533#5450ARROW-6348#5223ARROW-6337#5399ARROW-10850#9128ARROW-10624#9092ARROW-10386#8549ARROW-6994#23308ARROW-12774#10320ARROW-12670#10287ARROW-16828#13484ARROW-14989#13482ARROW-16977#13514ARROW-13404#10999ARROW-16887#13601ARROW-15906#13206ARROW-15280#13171ARROW-16144#13183ARROW-16511#13105ARROW-16085#13088ARROW-16715#13555ARROW-16268#13550ARROW-16700#13518ARROW-16807#13583ARROW-16871#13517ARROW-16415#13190ARROW-14821#12154ARROW-16439#13174ARROW-16394#13118ARROW-16516#13163ARROW-16395#13627ARROW-14848#12589ARROW-16407#13196ARROW-16653#13506ARROW-14575#13160ARROW-15271#13170ARROW-16703#13650ARROW-16444#13397ARROW-15016#13541ARROW-16776#13563ARROW-15622#13090ARROW-18131#14484ARROW-18305#14581ARROW-18285#14615 * Closes: #33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Feels a little funny that deleting this code just makes it work, so I added a decent number of tests to make sure differing schemas are handled. LMK if you think I missed something.