[SPARK-56642][SQL] Add pipelined JVM-Python UDF data transfer - #55552

Closed

viirya wants to merge 39 commits into

apache:masterfrom

viirya:pipelined-python-udf

Commits on Jun 10, 2026

[PYTHON] Add pipelined (async) JVM-Python UDF data transfer
viirya
committed
cf72fb1
[PYTHON] Redesign pipelined UDF to use full-duplex blocking I/O
viirya
committed
8e25c6a
[PYTHON] Use direct channel read to avoid potential blockingLock contention
viirya
committed
5469dd2
[SQL][PYTHON] Replace synchronized with volatile in InMemoryRowQueue for SPSC
viirya
committed
235adca
[PYTHON] Use thread pool for pipelined writer and propagate TaskContext
viirya
committed
674466a
[SQL][PYTHON] Remove @volatile from RowQueue counters and writeOffset
viirya
committed
529e1de
[PYTHON] Restrict pipelined mode to scalar UDF eval types
viirya
committed
fa6e4ee
[PYTHON] Materialize lazy iterators in pipelined reader thread
viirya
committed
f7c09a9
[PYTHON] Fix InputFileBlockHolder and idle timeout in pipelined mode
viirya
committed
5da1c45
[PYTHON] Fix config version from 4.1.0 to 4.2.0
viirya
committed
1593511
[PYTHON] Remove ad-hoc verify script
viirya
committed
df1229e
[SQL][PYTHON] Use lock-free InMemoryRowQueue only in pipelined mode
viirya
committed
a88642c
[PYTHON] Add dedicated tests for pipelined Python UDF execution
viirya
committed
b5aa78c
[PYTHON] Register test_pipelined_udf in test modules
viirya
committed
b7e36fd
[PYTHON] Add bindingPolicy to pipelined UDF configs
viirya
committed
9ab0eed
[SQL][PYTHON] Use volatile writeOffset for proper SPSC memory visibility
viirya
committed
793ca78
[PYTHON] Shut down socket on writer thread failure to prevent reader hang
viirya
committed
dec1b50
[PYTHON] Match sync mode idle timeout behavior in pipelined mode
viirya
committed
071ca3f
[PYTHON] Fix reader thread hang on main thread failure
viirya
committed
a28740e
[PYTHON] Restrict pipelined mode to UDF eval types only
viirya
committed
db0091d
[PYTHON] Always reset SO_TIMEOUT for reused pipelined workers
viirya
committed
5fc5be9
[SQL][PYTHON] Read volatile writeOffset unconditionally in doRemove
viirya
committed
23dcd3f
[PYTHON] Fix queueDepth config doc to match actual behavior
viirya
committed
429c89f
[SQL][PYTHON] Publish end-of-page marker with volatile store in lock-free mode
viirya
committed
b555952
[PYTHON] Remove unused time import in bench_pipelined_udf.py
viirya
committed
76b9b68
[PYTHON] Fix ruff format issues in Python files
viirya
committed
6e501b6
[PYTHON] Fix remaining ruff format issues
viirya
committed
a979c57
[PYTHON] Address PR review feedback
viirya
committed
4cb5e64
[PYTHON] Add ASV benchmark for pipelined UDF execution
viirya
committed
d4a3fca
[PYTHON] Fix ruff format issues
viirya
committed
4fd028b
[PYTHON] Address review feedback from Yicong-Huang
viirya
committed
3108e3f
[PYTHON] Fix ruff format in serializers.py
viirya
committed
07a3415
[PYTHON] Fix SPARK-33277-class use-after-free in pipelined mode
viirya
committed
83a5ce2
[PYTHON] Bump pipelined config version to 4.3.0
viirya
committed
67ba7aa
[PYTHON] Address Yicong-Huang's second-pass review on pipelined UDF
viirya
committed
15648ca
[PYTHON] Add wide-row benchmark for pipelined UDF
viirya
committed
5bbeaa4
[PYTHON] Move warnings import to module top to satisfy ruff
viirya
committed
47ee8b4
[PYTHON] Adapt pipelined_process to SPARK-56519 worker.py refactor
viirya
committed
df1b6d0
[PYTHON] Fix ruff format in wide-row benchmark
viirya
committed
2d8e034