Skip to content

TPC-DS parallel generation + io_uring streaming + tpch cleanup - #13

Merged
tsafin merged 12 commits into
masterfrom
tsafin/parallel_tpcds
Mar 18, 2026
Merged

TPC-DS parallel generation + io_uring streaming + tpch cleanup#13
tsafin merged 12 commits into
masterfrom
tsafin/parallel_tpcds

Conversation

@tsafin

@tsafintsafin commented Mar 17, 2026•
edited
Loading

Copy link
Copy Markdown
Owner
  • DS-10.1: Fork-after-init parallel generation for TPC-DS (--parallel, --parallel-tables N) with rolling N-slot window — all 24 tables generated concurrently with a single dsdgen init
  • DS-10.2/10.3: IoUringPool (anchor ring + ATTACH_WQ) and IoUringOutputStream (Arrow OutputStream backed by io_uring worker thread) injected into Parquet streaming path in parallel children
  • TPC-H port: ported the generic parallel + io_uring approach from tpcds_main.cpp to src/main.cpp — removes ad-hoc --async-io / --lance-io-uring flags, replaces flat fork loop with rolling N-slot window, adds --parallel-tables N and explicit --io-uring flag, wires single-table path too
  • CLI cleanup: removed 7 low-level --lance-* tuning options that don't belong in a benchmark CLI
  • README rewrite: accurate CLI reference for both binaries, Docker quick-start, removed stale optimization histories and project structure tree

Key flags added

BinaryFlagDescription
both--parallelFork-after-init parallel generation
both--parallel-tables NMax concurrent children (default: all)
both--io-uringExplicit opt-in for io_uring writes
tpch(already had) --zero-copyNow also required reminder: use at SF≥5 with --parallel

Test plan

  • tpch_benchmark --parallel --zero-copy --scale-factor 1 --max-rows 0 — all 8 tables, no crash
  • tpch_benchmark --parallel --zero-copy --io-uring --scale-factor 1 --max-rows 0 — io_uring=yes in banner
  • tpcds_benchmark --parallel --zero-copy --scale-factor 1 --max-rows 0 — all 24 tables
  • tpch_benchmark --help — no stale flags (--async-io, --lance-* gone)
  • Build with TPCH_ENABLE_LANCE=OFF and TPCH_ENABLE_LANCE=ON both succeed

tsafinand others added 10 commits March 16, 2026 23:34
@tsafin@claude
docs: add parallel TPC-DS generation + general io_uring architecture …
64372ee
…plan
Describes DS-10 design:
- Fork-after-init for all 24 TPC-DS tables (Phase DS-10.1)
- io_uring anchor ring as both ATTACH_WQ source and process scheduler
- pidfd + IORING_OP_POLL_ADD as the N-slot semaphore (rolling fork, no spin)
- Format-agnostic IoUringOutputStream injected at create_writer() factory
- Lessons generalised from Lance io_uring (persistent ring, no SQPOLL/O_DIRECT,
sysfs QD calibration, ATTACH_WQ shared worker pool)
- Rationale for rejecting pre-forked job-server pattern (static work set)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
tpcds: DS-10.1 — fork-after-init parallel generation (--parallel)
81c7539
Add --parallel flag to generate all 24 TPC-DS tables simultaneously using
fork-after-init: distributions loaded once in parent, children inherit via
COW with no re-initialisation. Rolling N-slot window via waitpid(-1) keeps
at most --parallel-tables N children active (default: all tables).
Key changes:
- DSDGenWrapper: add prepare_for_fork() and clear_tmp_path() for safe fork
(parent owns tmp dist file; children clear the path before exiting)
- tpcds_main: extract dispatch_generation() — eliminates 24-entry if-else
duplication between single-table and parallel paths
- ALL_TPCDS_TABLES ordered: small dims first → slots open quickly for facts
- Each child prints its own timing line; parent prints wall-time summary
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
docs: update parallel_uring_arch.md with DS-10.1 completion status
bb356e0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
tpcds: DS-10.2 IoUringPool + IoUringOutputStream; DS-10.3 Parquet str…
141624a
…eam injection
DS-10.2 — IoUringPool + IoUringOutputStream (format-agnostic async I/O):
include/tpch/io_uring_pool.hpp
src/async/io_uring_pool.cpp
• IoUringPool singleton: init() creates anchor io_uring ring with sysfs
queue-depth calibration (nr_requests/2, clamped [8,128]).
• watch_child(pidfd, user_data): submits POLL_ADD on the anchor ring —
in-flight count IS the N-slot semaphore (replaces waitpid/-1 in DS-10.2+).
• wait_any(): blocks until ≥1 child exits; returns batch of user_data values.
• create_child_ring_struct(): allocates a new io_uring ring with
IORING_SETUP_ATTACH_WQ → all child rings share one kernel worker pool.
• free_ring(): releases child ring (called by IoUringOutputStream dtor).
• Stub (TPCH_ENABLE_ASYNC_IO=OFF): available() returns false; all calls no-op.
include/tpch/io_uring_output_stream.hpp
src/async/io_uring_output_stream.cpp
• arrow::io::OutputStream backed by io_uring.
• Worker thread owns the ring for the file lifetime (persistent ring).
• Write(): pre-claims offset atomically (fetch_add), copies to WriteJob,
enqueues to MPSC queue, blocks on future until worker drains CQEs.
• Worker loop: 512 KB SQE chunks; submit when SQ full; drain all in-flight.
• No SQPOLL, no O_DIRECT (both hurt on WSL2/VirtIO).
• Sync fallback (ring==nullptr): pwrite(2) directly in Write(), no thread.
DS-10.3 — inject IoUringOutputStream into Parquet streaming path:
include/tpch/parquet_writer.hpp, src/writers/parquet_writer.cpp
• ParquetWriter::set_output_stream(shared_ptr<OutputStream>): inject external
stream; init_file_writer() uses it instead of FileOutputStream::Open().
src/tpcds_main.cpp
• In generate_all_tables_parallel(): call IoUringPool::init(output_dir) before
fork so children inherit the anchor ring fd for ATTACH_WQ.
• In run_table_child(): when IoUringPool::available() && parquet+zero-copy,
create child ring, wrap in IoUringOutputStream, inject into ParquetWriter.
CMakeLists.txt
• Add io_uring_pool.cpp, io_uring_output_stream.cpp unconditionally (both
handle #ifdef TPCH_ENABLE_ASYNC_IO internally; stubs when flag is OFF).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
docs: update parallel_uring_arch.md with DS-10.2 and DS-10.3 completion
962e7a3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
tpch: port generic parallel+io_uring approach from tpcds
ff3f4c6
Replace ad-hoc --async-io and --lance-io-uring flags with the generic
IoUringPool+IoUringOutputStream infrastructure already used by tpcds.
Changes:
- Remove Options::async_io, Options::lance_io_uring
- Remove AsyncIOContext single-table path (dead code since DS-10.3)
- Remove #ifdef TPCH_ENABLE_ASYNC_IO guards in parse_args
- Add Options::parallel_tables + --parallel-tables <N> flag
- Replace generate_all_tables_parallel_v2 flat-fork loop with rolling
N-slot window matching tpcds_main.cpp: run_table_child + generate_all_tables_parallel
- Inject IoUringOutputStream into Parquet streaming path in children
(same as tpcds DS-10.3)
- Print per-table completion lines to stdout (matches tpcds style)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
tpch: add explicit --io-uring flag, wire single-table path
e23b0ea
Replace implicit io_uring auto-detection (format==parquet && zero_copy)
with an explicit --io-uring CLI flag, matching the user-controlled
approach in tpcds.
- Add Options::io_uring + OPT_IO_URING + --io-uring to parse_args/usage
- Extract wire_io_uring() helper: Lance → enable_io_uring(true);
Parquet → inject IoUringOutputStream; other formats → no-op
- Gate IoUringPool::init() in generate_all_tables_parallel on opts.io_uring
- Wire single-table main() path: IoUringPool::init() + wire_io_uring()
(was completely missing before)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
tpch: remove low-level lance tuning options from CLI
26e4a79
Drop seven research knobs that don't belong in a benchmark CLI:
--lance-rows-per-file, --lance-rows-per-group, --lance-max-bytes-per-file,
--lance-skip-auto-cleanup, --lance-stream-queue, --lance-stats-level,
--lance-cardinality-sample-rate (and corresponding Options fields,
OPT_LANCE_* constants, parse_args cases, and the writer-setup block).
The single-table Lance path now mirrors run_table_child: only wires
enable_streaming_write() based on --zero-copy / --zero-copy-mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
docs: rewrite README — accurate CLI docs, Docker, remove stale content
3687eb8
- Remove: project structure tree, Phase 12/14 optimization histories,
future enhancements, architecture notes, outdated benchmark table,
references to removed flags (--use-dbgen, --async-io, --lance-*)
- Add: Docker quick-start with ghcr.io/tsafin/tpch-cpp-all image
- Add: complete, accurate tpch_benchmark and tpcds_benchmark CLI reference
reflecting current flags (--parallel-tables, --io-uring, etc.)
- Add: TPC-DS tables reference table
- Update: build options table (add TPCDS_ENABLE, remove stale entries)
- Update: dependencies table (add liburing, Rust entries)
- Add: performance notes (zero-copy requirement, io_uring trade-offs, ASAN warning)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin@claude
tpch: fix wire_io_uring Lance branch — guard with TPCH_LANCE_IO_URING
8b8dd38
enable_io_uring() is only declared when TPCH_LANCE_IO_URING is defined.
The CI lance build (TPCH_ENABLE_LANCE=ON, TPCH_LANCE_IO_URING not set)
hit a compile error. Gate the call on both defines.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin

tsafin commented Mar 17, 2026

Copy link
Copy Markdown
OwnerAuthor

@codex review

@tsafin
tsafin requested a review from CopilotMarch 17, 2026 22:07
CopilotAI reviewed Mar 17, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a generalized io_uring write backend and extends the benchmark drivers to support fork-after-init parallel generation (TPC-DS: all 24 tables; TPC-H: improved parallel controls), including Parquet streaming integration via injected arrow::io::OutputStream.

Changes:

  • Add IoUringPool + IoUringOutputStream and compile them into the core library (with runtime fallback when TPCH_ENABLE_ASYNC_IO is off/unavailable).
  • Enable Parquet streaming writer to accept an injected arrow::io::OutputStream (used for io_uring).
  • Implement TPC-DS parallel generation (--parallel, --parallel-tables) and update TPC-H CLI to support --parallel-tables and a unified --io-uring flag.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Show a summary per file
FileDescription
src/writers/parquet_writer.cppUses an injected Arrow OutputStream when provided for streaming Parquet writes.
include/tpch/parquet_writer.hppDeclares set_output_stream() and stores injected_stream_.
src/tpcds_main.cppAdds parallel all-tables generation + io_uring stream injection for Parquet zero-copy children.
src/main.cppReworks CLI flags and wiring for io_uring + parallel generation (TPC-H).
src/dsdgen/dsdgen_wrapper.cppAdds fork-safety helpers (prepare_for_fork, clear_tmp_path).
include/tpch/dsdgen_wrapper.hppDeclares fork-safety helpers for TPC-DS generator.
include/tpch/io_uring_pool.hppNew API for initializing/creating rings (anchor + child rings).
src/async/io_uring_pool.cppImplements io_uring ring init + child ring creation (stubbed when disabled).
include/tpch/io_uring_output_stream.hppNew Arrow OutputStream abstraction backed by io_uring (or sync fallback).
src/async/io_uring_output_stream.cppImplements io_uring-backed writes with a worker thread and chunked SQEs.
CMakeLists.txtAdds new io_uring sources to core build.
README.mdUpdates project scope (TPC-H + TPC-DS), flags, and usage examples.
docs/parallel_uring_arch.mdNew design/architecture writeup for parallel generation + io_uring integration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadsrc/async/io_uring_output_stream.cpp
Comment threadsrc/async/io_uring_output_stream.cpp
Comment threadsrc/async/io_uring_output_stream.cpp
Comment threadinclude/tpch/io_uring_output_stream.hpp
Comment threadsrc/writers/parquet_writer.cpp
Comment threadsrc/tpcds_main.cpp
tsafinand others added 2 commits March 19, 2026 00:34
@tsafin@claude
fix: address Copilot review comments on io_uring + parquet writer
4228ff2
io_uring_output_stream.cpp:
- Add missing <cerrno> include (comment #2949813816)
- Store chunk size in sqe->user_data; detect partial writes in CQE
handler (comment #2949813835)
- Move --inflight inside if(r==0) branch so it isn't decremented when
io_uring_wait_cqe() returns an error (comment #2949813848)
- Fix misleading comment: Write() waits for CQEs (page cache), not
fdatasync — no durability claim for benchmark use (comment #2949813868)
parquet_writer.cpp:
- set_output_stream(): validate streaming_mode_==true and
parquet_file_writer_==null before accepting the injected stream
(comment #2949813889)
tpcds_main.cpp:
- Move writer->close() inside the try/catch so close failures are
caught and reported as a non-zero exit code (comment #2949813909)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@tsafin
docs: fix stale Flush/Close comments in IoUringOutputStream header
73900f8
@tsafin
tsafin merged commit 4b897c7 into masterMar 18, 2026
74 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tsafin