feat: introduce pluggable SpillFile trait and TempFileFactory for custom spill backends

[pantShrey]

Which issue does this PR close?

• ClosesAllow pluggable file backends in DiskManager and IPCStreamWriter to support non-OS file systems #21215, depends on refactor: Update SortMergeJoin to use async spill abstractions #22230

Rationale for this change

DataFusion’s spill infrastructure is tightly coupled to OS-level files, with no extension points for alternative storage backends. DiskManager cannot be customized for file creation, and IPCStreamWriter depends on OS file paths.
This prevents integration in environments where temporary storage must be managed by the host system. For example, Postgres extensions (e.g., ParadeDB) require spill files to go through BufFile APIs to respect temp_tablespaces, enforce temp_file_limit, and integrate with transaction-scoped cleanup. Since BufFile has no OS-visible path, it cannot work with the current design.
A secondary motivation raised by @alamb is supporting object storage backends (S3, GCS) for spilling, which require async IO and cannot use std::io::Write or std::io::Read.

What changes are included in this PR?

• Introduced SpillFile, SpillWriter, and TempFileFactory traits to abstract spill file handling
• Added DiskManagerMode::Custom to allow pluggable backends
• Updated DiskManager to return Arc<dyn SpillFile> instead of OS-bound types
• Refactored write path using SpillWriteAdapter to bridge sync Arrow writers with backend-agnostic writers
• Refactored read path to use async streaming (Stream<Item = Result<Bytes>>) instead of blocking state machines
• Updated spill-related components to operate on Arc<dyn SpillFile>
• Migrated the Sort-Merge Join (SMJ) operator to use the async spill abstraction

Are these changes tested?

Yes. Existing spill tests cover the full read/write flow.

• Fixed test_disk_usage_decreases_as_files_consumed by correcting a pre-existing off-by-one assumption in file rotation
• Fixed test_preserve_order_with_spilling by just asserting spilling occurs (spill_count>0) and output batches are sorted

Are there any user-facing changes?

Yes this introduces API changes:

• Spill-related APIs now use Arc<dyn SpillFile> instead of RefCountedTempFile
• New public traits: SpillFile, SpillWriter, TempFileFactory
• Added DiskManagerMode::Custom for custom backends

Custom spill backends can now be implemented and plugged in via DiskManager.

[pantShrey]

@alamb I opened this draft PR to get early feedback on the architecture.

1. The first point is around the sync read path. I introduced open_sync_reader because SortMergeJoin currently has synchronous, blocking code paths that directly open files using paths and BufReader, instead of going through the spill abstractions. Converting this to fully async would significantly increase the scope of this PR.

   • Does it make sense to keep this escape hatch for now and handle making these operators async in a follow-up PR?

2. The second point is regarding test failures. I have not modified the original 64B limit in the tests because I wanted guidance here. Currently, the repartition test in mod.rs is failing, and it seems related to spilling not being triggered correctly, the new SpillWriteAdapter adds slight allocation overhead which makes the original 64-byte memory limit too tight for the merge heap to initialize (~296 bytes needed), bumping up the memory limit causes the test to not spill anymore, I believe increasing the test data size might solve the issue, but am not sure.

I might be missing something here, so would really appreciate your guidance.

[alamb]

Thanks -- will try and look at this shortly

[alamb]

@alamb I opened this draft PR to get early feedback on the architecture.

1. The first point is around the sync read path. I introduced open_sync_reader because SortMergeJoin currently has synchronous, blocking code paths that directly open files using paths and BufReader, instead of going through the spill abstractions. Converting this to fully async would significantly increase the scope of this PR.

   • Does it make sense to keep this escape hatch for now and handle making these operators async in a follow-up PR?

Kind of, though it seems like accumulating technical debt as we'll have APIs that will not be needed once we complete the work for SortMergeJoin

What do you think about making a first PR to migrate SortMergeJoin to use the spill abstraction?

2. The second point is regarding test failures. I have not modified the original 64B limit in the tests because I wanted guidance here. Currently, the repartition test in mod.rs is failing, and it seems related to spilling not being triggered correctly, the new SpillWriteAdapter adds slight allocation overhead which makes the original 64-byte memory limit too tight for the merge heap to initialize (~296 bytes needed), bumping up the memory limit causes the test to not spill anymore, I believe increasing the test data size might solve the issue, but am not sure.

Makes sense to me

[alamb]

Thanks @pantShrey - I reviewed this and the basic idea looks good to me. I do think it would be nice to have a unified (async) IO abstraction rather than leaving some hook around for sync IO and making this API more complicated

[alamb]

A small nit: I think "custom" is a somewhat unecessary term here . Perhaps this

 factory:Option<Arc<dyn TempFileFactory>>,

or

 temp_file_factory:Option<Arc<dyn TempFileFactory>>,

would be more consistent with the rest of the codebase

[alamb]

maybe "file spill writer"?

[alamb]

This is pretty similar to https://doc.rust-lang.org/std/io/trait.Write.html 🤔

[pantShrey]

Yes, you are right. The reason I didn't use Write trait which uses &[u8] was for ownership reasons. Some backends might queue chunks to a background task (e.g., S3 multipart via a channel) and need to hold the data past the write() call's return. &[u8] can't express that, and it would force a second copy between the SpillWriteAdapter and the SpillWriter.
Also, the custom SpillWriter trait contains finish(), which maps perfectly to complete_multipart_upload for S3 and resource owner cleanup for Postgres.

[alamb]

This is all true -- however, I think that since the underlying IPC writer takes a std::io::Write, forcing all backends to use Bytes will likely require an extra unecessary copy (see comments below on SpillWriterAdapter) anways.

If you use a std::io::write like interface here, backends that want to queue chunks can do so (by copying into Bytes buffers themselves)

Thus what i suggest is:

1. Change this to look more like std::io::wrote:

fnwrite(&mutself,data:&[u8]) -> Result<()>;

Which will allow you to get rid of the write adapter

[pantShrey]

@alamb Thank you so much for the review! I scoped out the SortMergeJoin migration today, specifically looking at bitwise_stream.rs and process_key_match_with_filter, to see what it would take.

Because SortMergeJoin currently reads from the spill file via a synchronous for loop inside a hand-rolled poll state machine, making the read path truly async requires a major rewrite. We can't just .await the stream, so we may need to store the SendableRecordBatchStream in the execution state and manually persist variables like matched_count across Poll::Pending yields.

Because ParadeDB is hoping to unblock their Postgres integration next week, I'm worried a state machine rewrite of this scale will stall them.

Would you be open to merging this core abstraction first (with open_sync_reader marked as #[deprecated])? I can open a dedicated tracking issue for the SortMergeJoin async migration and tackle it as a fast follow-up PR.

I am happy to defer to your judgment if you feel the tech debt must be addressed first!

[alamb]

I am happy to defer to your judgment if you feel the tech debt must be addressed first!

How about we try it in parallel?

[pantShrey]

I am happy to defer to your judgment if you feel the tech debt must be addressed first!

How about we try it in parallel?

@alamb sure, i have already started to work on that locally while waiting for the response

also i am actually still stuck on the test repartition::test::test_preserve_order_with_spilling

The issue stems from the fact that RepartitionMerge now requires more memory than a RepartitionExec node, this greedily allocates memory to RepartitionExec which could have spilled instead of RepartitionMerge which cannot spill.

I would really appreciate any guidance on this, am I missing something obvious here?

[alamb]

test_preserve_order_with_spilling

Sadly I am not familar with this test so I don't have a lot to offer you

Maybe you can look at git history and see who introduced the test and maybe they might have some ideas

[pantShrey]

Hey @adriangb, Andrew suggested I reach out to you since you originally authored repartition::test::test_preserve_order_with_spilling. I'm currently hitting a wall with it while migrating the spilling architecture to async streams.

The test is currently stuck in a memory-accounting deadlock. Here’s what is happening:

• If I set the memory pool limit tight enough to force a spill, RepartitionMerge panics during initialization. It needs to reserve some memory to set up its streams, but exhausts the pool before completing its unspillable setup.

• However, if I increase the pool limit to give Merge enough headroom to initialize safely and then scale up the data volume to force overflow, the RepartitionExec producers greedily consume the additional memory first. This either ends up starving Merge again or allows the query to complete entirely in memory without triggering a spill.

I was able to trigger a spill once by setting the test memory limit to 608 B, but even that was not sufficient for the test to pass reliably.

Is there a correct or idiomatic way to configure this test (batch sizes, data volume, memory pool limits, etc.) to reliably force a RepartitionExec spill without violating the Merge operator’s baseline initialization overhead? Or am I approaching this incorrectly and missing something obvious?

I would really appreciate any guidance you could provide.

[adriangb]

IIRC that test was added when we added spilling to RepartitionExec. Conceptually the test is simple: if RepartitionExec is configured to preserve order and it spills we need to make sure that spilling did not shuffle the data. The orchestration however is difficult: forcing a RepartitionExec to spill usually requires skewed upstream partition consumption rates. You could try to change the test to eg use a GroupBy or maybe we can use a RepartitionExec in isolation if we pull from the streams in the right way. I think the structure can be changed quite a bit as long as we preserve the semantic meaning of the test, I am not surprised that it is pretty fragile to changes.

[pantShrey]

@alamb I’ve addressed the nits and force-pushed the updates. Could you please trigger the CI and take another look when you have a moment? In the meantime, I am working on migrating SortMergeJoin to the new spill abstractions in parallel so that both can be reviewed quickly. Thank you again for your time!

[pantShrey]

@adriangb Thank you so much for the guidance! I updated the test to simply assert that a spill does occur
(spill_count > 0) and that the batch output order remains perfectly sorted, rather than trying to force every single batch to spill. I hope this aligns with the semantic purpose you originally envisioned for the test. I really appreciate your help getting me unstuck here!

[adriangb]

@adriangb Thank you so much for the guidance! I updated the test to simply assert that a spill does occur (spill_count > 0) and that the batch output order remains perfectly sorted, rather than trying to force every single batch to spill. I hope this aligns with the semantic purpose you originally envisioned for the test. I really appreciate your help getting me unstuck here!

That makes sense to me.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

 Cloning apache/main
Building datafusion-execution v54.0.0 (current)
Built [ 29.037s] (current)
Parsing datafusion-execution v54.0.0 (current)
Parsed [ 0.025s] (current)
Building datafusion-execution v54.0.0 (baseline)
Built [ 29.097s] (baseline)
Parsing datafusion-execution v54.0.0 (baseline)
Parsed [ 0.025s] (baseline)
Checking datafusion-execution v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.328s] 223 checks: 220 pass, 3 fail, 0 warn, 30 skip
--- failure auto_trait_impl_removed: auto trait no longer implemented ---
Description:
A public type has stopped implementing one or more auto traits. This can break downstream code that depends on the traits being implemented.
ref: https://doc.rust-lang.org/reference/special-types-and-traits.html#auto-traits
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/auto_trait_impl_removed.ron
Failed in:
type DiskManagerBuilder is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:38
type DiskManagerBuilder is no longer RefUnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:38
type DiskManager is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:154
type DiskManager is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:154
type DiskManagerMode is no longer UnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:123
type DiskManagerMode is no longer RefUnwindSafe, in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:123
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron
Failed in:
variant DiskManagerMode:Custom in /home/runner/work/datafusion/datafusion/datafusion/execution/src/disk_manager.rs:135
--- failure inherent_method_missing: pub method removed or renamed ---
Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/inherent_method_missing.ron
Failed in:
RefCountedTempFile::update_disk_usage, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/7f766cb34f54fa9bd7bceab2096115af131e1f2c/datafusion/execution/src/disk_manager.rs:334
RefCountedTempFile::current_disk_usage, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/7f766cb34f54fa9bd7bceab2096115af131e1f2c/datafusion/execution/src/disk_manager.rs:382
Summary semver requires new major version: 3 major and 0 minor checks failed
Finished [ 59.781s] datafusion-execution
Building datafusion-physical-plan v54.0.0 (current)
Built [ 35.687s] (current)
Parsing datafusion-physical-plan v54.0.0 (current)
Parsed [ 0.151s] (current)
Building datafusion-physical-plan v54.0.0 (baseline)
Built [ 35.360s] (baseline)
Parsing datafusion-physical-plan v54.0.0 (baseline)
Parsed [ 0.142s] (baseline)
Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.919s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 73.637s] datafusion-physical-plan

[pantShrey]

cargo-semver-checks flagged DiskManagerMode::Custom as a breaking change since the enum isn't
#[non_exhaustive]. Happy to add it if preferred, but wanted to check first since it would affect downstream users matching on this enum.

[pantShrey]

Hey @alamb, quick update! While working on the SortMergeJoin async migration in parallel, I realised the changes were actually quite contained (~260 insertions, ~170 deletions). Rather than opening a second stacked PR and temporarily introducing the open_sync tech debt to main, I went ahead and rolled the refactor directly into this PR to keep things clean. I hope this approach is okay with you!

I believe the PR is now ready for review, so I've marked it as such. I'd appreciate another look whenever you have the time. Thank you!

[alamb]

Rather than opening a second stacked PR and temporarily introducing the open_sync tech debt to main, I went ahead and rolled the refactor directly into this PR to keep things clean. I hope this approach is okay with you!

Grerat-- can you please make a PR for just the SMJ refactor and then stack this PR on it?

[alamb]

That will make it easier / faster to review (I am not a SMJ expert so I can't really review that part effiicently)

[pantShrey]

Hey @alamb, quick update!

I've reworked both PRs to make them easier to review independently:

• refactor: Update SortMergeJoin to use async spill abstractions #22230 (SMJ refactor) no longer depends on this PR, it now migrates SortMergeJoin to async spill abstractions while keeping the concrete RefCountedTempFile type, so it can be reviewed and merged standalone.
• This PR now only does one focused thing: introduces the SpillFile trait + TempFileFactory and swaps the internal type from RefCountedTempFile → Arc<dyn SpillFile> and internal changes in the spill module. I've also removed the open_sync tech debt that was here before, and have added the skip validation in streamdecoder

The plan is for #22230 to merge first, then I'll rebase this on top of it. Would be grateful if you could take a look whenever you get the chance!

[pantShrey]

@alamb I spent some time looking into the spill_io regression.

From what I can tell, increasing the ReaderStream capacity somewhat restores the benchmark performance locally. I initially tried going up to 1 MB, but capacities around 256 KB started failing the SQL logic tests, so for now I've pushed 128 KB.

My current understanding is that the previous StreamReader + BufReader implementation, despite having an 8 KB buffer, would typically read an entire IPC frame without yielding. With the current Tokio async stream, once the initial
8 KB is consumed, the task yields repeatedly while reading the remainder of the frame. Combined with the "copy" into the decoder's scratch buffer, this seems to add noticeable overhead for multi-MB frames/batches.

tokio_util::io::ReaderStream::with_capacity(file,128*1024)

I've pushed the change mainly so the CI benchmarks can run and to get your thoughts. If this direction makes sense, or should I make it configurable instead?

[alamb]

run benchmarks spill_io

[alamb]

I took a look at the new code and it looks good to me. I'll plan to merge this PR subject to the benchmarks looking good

Thanks @pantShrey

[alamb]

👍

[alamb]

run benchmark external_aggr sort_tpch spill_io

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4834930095-736-2qkg5 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing abstract-spill-file (4b93040) to 367f08e (merge-base) diff using: spill_io
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4834955270-739-6fvlr 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing abstract-spill-file (4b93040) to 367f08e (merge-base) diff using: spill_io
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4834955270-737-5gqsv 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing abstract-spill-file (4b93040) to 367f08e (merge-base) diff using: external_aggr
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance:c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4834955270-738-55th8 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Comparing abstract-spill-file (4b93040) to 367f08e (merge-base) diff using: sort_tpch
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD abstract-spill-file
----- ---- -------------------
spill_compression/q16/lz4_frame 1.12 39.4±2.14ms ? ?/sec 1.00 35.3±0.39ms ? ?/sec
spill_compression/q16/uncompressed 1.28 28.8±4.65ms ? ?/sec 1.00 22.5±1.21ms ? ?/sec
spill_compression/q16/zstd 1.00 64.4±2.44ms ? ?/sec 1.02 65.5±0.44ms ? ?/sec
spill_compression/q2/lz4_frame 1.08 18.5±2.08ms ? ?/sec 1.00 17.2±0.14ms ? ?/sec
spill_compression/q2/uncompressed 1.77 14.0±1.21ms ? ?/sec 1.00 7.9±0.42ms ? ?/sec
spill_compression/q2/zstd 1.02 29.7±0.29ms ? ?/sec 1.00 29.2±0.38ms ? ?/sec
spill_compression/q20/lz4_frame 1.05 26.0±3.18ms ? ?/sec 1.00 24.8±0.70ms ? ?/sec
spill_compression/q20/uncompressed 1.65 19.0±4.11ms ? ?/sec 1.00 11.5±0.62ms ? ?/sec
spill_compression/q20/zstd 1.01 46.2±1.00ms ? ?/sec 1.00 45.8±0.59ms ? ?/sec
spill_compression/wide/lz4_frame 1.00 85.0±4.16ms ? ?/sec 1.05 89.4±2.21ms ? ?/sec
spill_compression/wide/uncompressed 1.42 83.9±41.36ms ? ?/sec 1.00 59.2±2.86ms ? ?/sec
spill_compression/wide/zstd 1.00 162.9±6.30ms ? ?/sec 1.02 165.9±0.72ms ? ?/sec
spill_io/StreamReader/read_100/ 1.50 36.6±4.89ms ? ?/sec 1.00 24.4±0.77ms ? ?/sec

Resource Usage

spill_io — base (merge-base)

Metric	Value
Wall time	325.1s
Peak memory	263.4 MiB
Avg memory	56.3 MiB
CPU user	123.1s
CPU sys	47.1s
Peak spill	0 B

spill_io — branch

Metric	Value
Wall time	325.1s
Peak memory	304.1 MiB
Avg memory	80.4 MiB
CPU user	136.2s
CPU sys	55.3s
Peak spill	0 B

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

group HEAD abstract-spill-file
----- ---- -------------------
spill_compression/q16/lz4_frame 1.17 40.3±2.82ms ? ?/sec 1.00 34.4±0.58ms ? ?/sec
spill_compression/q16/uncompressed 1.45 28.8±4.72ms ? ?/sec 1.00 19.8±1.24ms ? ?/sec
spill_compression/q16/zstd 1.00 64.7±2.92ms ? ?/sec 1.00 65.0±0.71ms ? ?/sec
spill_compression/q2/lz4_frame 1.09 18.4±2.12ms ? ?/sec 1.00 16.9±0.21ms ? ?/sec
spill_compression/q2/uncompressed 1.98 14.8±4.53ms ? ?/sec 1.00 7.5±0.37ms ? ?/sec
spill_compression/q2/zstd 1.05 30.5±3.25ms ? ?/sec 1.00 29.2±0.38ms ? ?/sec
spill_compression/q20/lz4_frame 1.05 25.6±2.96ms ? ?/sec 1.00 24.3±0.72ms ? ?/sec
spill_compression/q20/uncompressed 1.97 19.5±5.44ms ? ?/sec 1.00 9.9±0.35ms ? ?/sec
spill_compression/q20/zstd 1.02 46.3±0.90ms ? ?/sec 1.00 45.3±0.41ms ? ?/sec
spill_compression/wide/lz4_frame 1.05 92.4±16.09ms ? ?/sec 1.00 87.6±1.33ms ? ?/sec
spill_compression/wide/uncompressed 1.39 80.4±10.30ms ? ?/sec 1.00 57.9±3.06ms ? ?/sec
spill_compression/wide/zstd 1.00 162.8±7.24ms ? ?/sec 1.01 165.1±0.56ms ? ?/sec
spill_io/StreamReader/read_100/ 1.62 37.6±6.18ms ? ?/sec 1.00 23.2±0.80ms ? ?/sec

Resource Usage

spill_io — base (merge-base)

Metric	Value
Wall time	235.1s
Peak memory	243.2 MiB
Avg memory	75.1 MiB
CPU user	119.7s
CPU sys	47.0s
Peak spill	0 B

spill_io — branch

Metric	Value
Wall time	280.1s
Peak memory	361.5 MiB
Avg memory	118.3 MiB
CPU user	137.6s
CPU sys	50.6s
Peak spill	0 B

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and abstract-spill-file
--------------------
Benchmark sort_tpch1.json
--------------------
┏━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ abstract-spill-file ┃ Change ┃
┡━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Q1 │ 110.91 / 111.86 ±0.85 / 113.26 ms │ 111.92 / 112.72 ±0.83 / 113.80 ms │ no change │
│ Q2 │ 100.79 / 101.50 ±0.84 / 103.14 ms │ 99.56 / 100.30 ±0.65 / 101.49 ms │ no change │
│ Q3 │ 644.30 / 650.02 ±4.19 / 656.33 ms │ 642.78 / 646.66 ±3.44 / 652.21 ms │ no change │
│ Q4 │ 178.26 / 183.66 ±6.86 / 196.88 ms │ 179.31 / 180.62 ±1.19 / 182.80 ms │ no change │
│ Q5 │ 260.34 / 261.52 ±1.53 / 264.46 ms │ 259.96 / 260.76 ±0.76 / 261.84 ms │ no change │
│ Q6 │ 274.19 / 274.50 ±0.16 / 274.65 ms │ 273.69 / 274.75 ±1.01 / 276.56 ms │ no change │
│ Q7 │ 446.89 / 449.24 ±2.71 / 454.46 ms │ 445.51 / 446.94 ±1.69 / 450.23 ms │ no change │
│ Q8 │ 310.90 / 321.85 ±11.45 / 342.19 ms │ 310.48 / 319.68 ±10.01 / 337.86 ms │ no change │
│ Q9 │ 332.15 / 349.14 ±14.24 / 366.84 ms │ 330.84 / 340.22 ±5.68 / 346.53 ms │ no change │
│ Q10 │ 466.39 / 473.45 ±4.67 / 479.97 ms │ 470.20 / 474.78 ±2.83 / 478.38 ms │ no change │
│ Q11 │ 229.22 / 247.24 ±17.39 / 271.41 ms │ 228.15 / 239.37 ±13.22 / 262.59 ms │ no change │
└───────┴────────────────────────────────────┴────────────────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 3423.97ms │
│ Total Time (abstract-spill-file) │ 3396.80ms │
│ Average Time (HEAD) │ 311.27ms │
│ Average Time (abstract-spill-file) │ 308.80ms │
│ Queries Faster │ 0 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 11 │
│ Queries with Failure │ 0 │
└────────────────────────────────────┴───────────┘

Resource Usage

sort_tpch — base (merge-base)

Metric	Value
Wall time	20.0s
Peak memory	2.5 GiB
Avg memory	1.2 GiB
CPU user	64.1s
CPU sys	2.8s
Peak spill	0 B

sort_tpch — branch

Metric	Value
Wall time	20.0s
Peak memory	2.5 GiB
Avg memory	1.2 GiB
CPU user	64.4s
CPU sys	2.5s
Peak spill	0 B

---

File an issue against this benchmark runner

[adriangbot]

🤖 Benchmark completed (GKE) | trigger

Instance:c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected

Details

Comparing HEAD and abstract-spill-file
--------------------
Benchmark external_aggr.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ abstract-spill-file ┃ Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ Q1(64.0 MB) │ 56.78 / 59.80 ±3.68 / 67.02 ms │ 55.29 / 57.89 ±2.47 / 61.73 ms │ no change │
│ Q1(32.0 MB) │ 53.78 / 56.67 ±1.67 / 58.29 ms │ 50.23 / 52.54 ±1.74 / 54.98 ms │ +1.08x faster │
│ Q1(16.0 MB) │ 52.48 / 55.14 ±2.37 / 58.50 ms │ 53.83 / 55.24 ±1.57 / 58.10 ms │ no change │
│ Q2(512.0 MB) │ 287.06 / 301.86 ±19.14 / 335.82 ms │ 267.01 / 282.20 ±9.85 / 295.77 ms │ +1.07x faster │
│ Q2(256.0 MB) │ 247.80 / 259.58 ±18.58 / 296.58 ms │ 244.64 / 247.82 ±2.49 / 251.51 ms │ no change │
│ Q2(128.0 MB) │ 249.87 / 277.59 ±42.37 / 359.92 ms │ 243.18 / 248.16 ±3.21 / 252.38 ms │ +1.12x faster │
│ Q2(64.0 MB) │ 250.91 / 264.10 ±20.55 / 305.06 ms │ 247.45 / 250.98 ±3.38 / 257.01 ms │ no change │
│ Q2(32.0 MB) │ 320.96 / 324.18 ±1.97 / 326.12 ms │ 316.89 / 320.98 ±3.33 / 326.99 ms │ no change │
└──────────────┴────────────────────────────────────┴───────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 1598.91ms │
│ Total Time (abstract-spill-file) │ 1515.80ms │
│ Average Time (HEAD) │ 199.86ms │
│ Average Time (abstract-spill-file) │ 189.47ms │
│ Queries Faster │ 3 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 5 │
│ Queries with Failure │ 0 │
└────────────────────────────────────┴───────────┘

Resource Usage

external_aggr — base (merge-base)

Metric	Value
Wall time	540.1s
Peak memory	589.5 MiB
Avg memory	10.1 MiB
CPU user	25.4s
CPU sys	4.2s
Peak spill	0 B

external_aggr — branch

Metric	Value
Wall time	545.1s
Peak memory	484.1 MiB
Avg memory	9.2 MiB
CPU user	27.2s
CPU sys	3.9s
Peak spill	0 B

---

File an issue against this benchmark runner

[alamb]

Al right -- I think this is looking good and I put it in th emerge queue

[pantShrey]

@alamb Thank you so much for all your time and guidance on this one! I learned a ton working through the architecture with you. I really appreciate your help throughout the whole process and for getting this merged!

[philippemnoel]

Thank you to the both of you!! 🙏

[alamb]

@alamb Thank you so much for all your time and guidance on this one! I learned a ton working through the architecture with you. I really appreciate your help throughout the whole process and for getting this merged!

Thank you for bearing with my @pantShrey -- this is a tricky one as it involved API design and making sure we didn't mess up existing performance. I think it is a nice addition to the DataFusion APUs

@philippemnoel 🥂

Any chance that one of you can help review an example I made using these new APIs:

• Add ObjectStore-backed TempFileFactor / spill example #23170