chore(python/sedonadb): Upgrade PyO3 and enable free-threaded Python support - #1012
Conversation
paleolimbot
left a comment
There was a problem hiding this comment.
I will take a closer look Monday, but I do think it's worth some effort to avoid with_context() (in particular, waiting for a future just to kick off an operation!)
| pub struct InternalContext { | ||
| pub inner: SedonaContext, | ||
| pub inner: Arc<Mutex<SedonaContext>>, | ||
| pub runtime: Arc<Runtime>, | ||
| } |
There was a problem hiding this comment.
I think it may be possible to avoid the Mutex here, which is causing some complexity.
The DataFusion SessionContext, on which the SedonaContext is based, is thread safe (I think) because its state() already uses a RwLock. We may have to put one or more members of the SedonaContext behind a RwLock as well, which would only require operations that update the context to lock it (e.g., registering things).
There was a problem hiding this comment.
@paleolimbot Thanks! I agreed that the Python-side Arc<Mutex<SedonaContext>> was too broad.
I reworked this to keep normal SedonaContext / SessionContext operations lock-free from the Python wrapper, and moved synchronization to the mutable function registry instead. SedonaContext.functions is now a private RwLock<FunctionSet> with narrow helper methods for lookup and registration. InternalContext owns SedonaContext directly again, and register_component() uses those helpers.
I also removed the unused Python-token parameters after the refactor.
We can troubleshoot this if it causes issues...I think it is something to do with the |
Summary
Closes #236
Testing
cargo test for sedonadb and sedonadb-zarr reached linking locally but failed because libpython3.9 was not available in this local macOS Python environment.