Uh oh!
There was an error while loading. Please reload this page.
Draft PyArrow Dataset reader impl - #21
Conversation
wjones127
commented
Feb 20, 2022
FYI I am going to set this aside for now, since I think this really needs the Arrow C Stream Interface to be reliable. Right now it just wraps a Python iterator and holds onto the GIL while waiting for the PyArrow scanner to generate each batch. I'm running into some GIL deadlocks, so it would be nice to eliminate the GIL stuff from record batch streaming. |
kdbrooks
commented
Jul 18, 2022
I believe I have a working solution for this that I developed for the company I work for. I will get a PR out there soon. Is there still a need for this? |
wjones127
commented
Jul 18, 2022
This was mostly an experimental curiosity, but a PR would be cool if you are willing :) I probably won't get around to finishing this for a while. |
kdbrooks
commented
Jul 19, 2022
@wjones127 I've created a PR, #59 I couldn't add you as a reviewer after I made the PR but I'd love to have your feedback on it. |
Work in progress. Working toward being able to stream record batches from a PyArrow dataset.
Fixes#10.