Uh oh!
There was an error while loading. Please reload this page.
GH-39565: [C++] Do not concatenate ChunkedArray when running take function - #39566
GH-39565: [C++] Do not concatenate ChunkedArray when running take function#39566amol- wants to merge 28 commits into
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
pitrou
commented
Feb 28, 2024
@felipecrv@amol- Should this PR be kept open now that #40206 was merged? |
amol-
commented
Feb 28, 2024
I think so, this PR is focused on optimizing TakeCA, while the one that was merged was focused on TakeCC |
felipecrv
commented
Feb 28, 2024
Before my PR: Next step (and goal of amol's PR/issue pair): 0 concatenations. |
felipecrv
commented
May 17, 2024
I opened #41700 which can handle all the fixed-width types without concatenation. |
Rationale for this change
We can avoid extra unecessary work and memory consumption of concatenating chunks when running take, we can directly run the take on the chunks at the only cost of remapping the indices which are usually much fewer than the size of the array we are applying take on.
Are these changes tested?
Two tests already existed that verify take on ChunkedArray and they covered the corner cases well, the only tweak necessary to those tests was that now take returns a chunkedarray made of multiple chunks instead of a single one.