Skip to content

chore(python/sedonadb): Upgrade PyO3 and enable free-threaded Python support - #1012

Merged
paleolimbot merged 4 commits into
apache:mainfrom
ajaypadwal73:perf/pyo3-free-threaded
Jun 29, 2026
Merged

chore(python/sedonadb): Upgrade PyO3 and enable free-threaded Python support#1012
paleolimbot merged 4 commits into
apache:mainfrom
ajaypadwal73:perf/pyo3-free-threaded

Conversation

@ajaypadwal73

@ajaypadwal73 ajaypadwal73 commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • upgrade PyO3 to 0.26 for the Python extension crates
  • mark the Python modules as free-threading compatible with gil_used = false
  • protect the Python InternalContext state with explicit locking and update callback paths to use attach/detach
  • add a threaded context/dataframe regression test

Closes #236

Testing

  • cargo check -p sedonadb
  • cargo check -p sedonadb-zarr
  • .venv/bin/python -m pytest python/sedonadb/tests/test_context.py::test_shared_context_and_dataframe_from_threads -vv
  • .venv/bin/python -m pytest python/sedonadb/tests/test_dataframe.py::test_to_view python/sedonadb/tests/test_dataframe.py::test_count -vv
  • .venv/bin/python -m pytest python/sedonadb/tests/test_context.py -vv (10 passed; 2 network-backed tests failed locally while reading GitHub/S3 URLs)

cargo test for sedonadb and sedonadb-zarr reached linking locally but failed because libpython3.9 was not available in this local macOS Python environment.

@github-actions
github-actions Bot requested a review from paleolimbot June 27, 2026 12:11

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will take a closer look Monday, but I do think it's worth some effort to avoid with_context() (in particular, waiting for a future just to kick off an operation!)

Comment on lines 39 to 42
pub struct InternalContext {
pub inner: SedonaContext,
pub inner: Arc<Mutex<SedonaContext>>,
pub runtime: Arc<Runtime>,
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it may be possible to avoid the Mutex here, which is causing some complexity.

The DataFusion SessionContext, on which the SedonaContext is based, is thread safe (I think) because its state() already uses a RwLock. We may have to put one or more members of the SedonaContext behind a RwLock as well, which would only require operations that update the context to lock it (e.g., registering things).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paleolimbot Thanks! I agreed that the Python-side Arc<Mutex<SedonaContext>> was too broad.

I reworked this to keep normal SedonaContext / SessionContext operations lock-free from the Python wrapper, and moved synchronization to the mutable function registry instead. SedonaContext.functions is now a private RwLock<FunctionSet> with narrow helper methods for lookup and registration. InternalContext owns SedonaContext directly again, and register_component() uses those helpers.

I also removed the unused Python-token parameters after the refactor.

@paleolimbot paleolimbot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@paleolimbot

Copy link
Copy Markdown
Member

cargo test for sedonadb and sedonadb-zarr reached linking locally but failed because libpython3.9 was not available in this local macOS Python environment.

We can troubleshoot this if it causes issues...I think it is something to do with the pyo3/extension-module feature but may clear up with a cargo clean on your end.

@paleolimbot paleolimbot changed the title Upgrade PyO3 and enable free-threaded Python support chore(python/sedonadb): Upgrade PyO3 and enable free-threaded Python support Jun 29, 2026
@paleolimbot
paleolimbot merged commit dfad028 into apache:main Jun 29, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade pyo3 and enable free-threaded Python support

2 participants