docs: expand tuning guide with performance and memory configs

[andygrove]

Summary

Audited all CometConf entries in preparation for the 1.0 release and expanded the Tuning Guide with the user-facing performance and memory knobs that were previously undocumented.

New sections in tuning.md:

• Batch Size — spark.comet.batchSize (memory vs. throughput tradeoff) and its shuffle-writer cousin spark.comet.shuffle.jvm.batchSize
• Limiting Spill Disk Usage — spark.comet.maxTempDirectorySize (100 GB default)
• Parquet Reader Tuning — parallel I/O thread pool, range coalescing, filter pushdown / late materialization
• Iceberg Scan Tuning — spark.comet.scan.icebergNative.dataFileConcurrencyLimit
• Reducing Row/Columnar Conversion Overhead — spark.comet.exec.transitionRevert.{enabled,maxTransitions}
• Metrics Overhead — why spark.comet.metrics.enabled is off by default
• Expanded Shuffle Compression with codec / zstd level knobs

Also:

• Fixed an incorrect claim that Comet overrides Spark LZ4 shuffle compression with ZSTD (the default codec is lz4).
• Moved spark.comet.maxTempDirectorySize from the exec category to tuning so it renders in the Memory & Tuning table in configs.md, and expanded its doc string.

Test plan

• Docs build renders new sections correctly
• configs.md still generates cleanly (moved config appears in Memory & Tuning table)

[mbutrovich]

First pass, thanks @andygrove!

[mbutrovich]

This subsection documents four configs that no longer exist on main. spark.comet.parquet.read.parallel.io.enabled, spark.comet.parquet.read.parallel.io.thread-pool.size, spark.comet.parquet.read.io.mergeRanges, and spark.comet.parquet.read.io.mergeRanges.delta were removed in b016555 (#4981), which merged on 2026-07-20, a week after this branch was cut. Grepping spark/, common/, and native/ on main finds no references to any of them outside generated target/ output. #4981 removed them specifically because they misled operators tuning benchmarks, so documenting them again would reverse that. Could you drop ### Parallel I/O entirely?

[mbutrovich]

Is "per executor" the right scope here? The DiskManagerBuilder carrying with_max_temp_directory_size is constructed in prepare_datafusion_session_context (native/core/src/execution/jni_api.rs:555), which createPlan calls once per native plan per task, and the comment just below notes the context is scoped to an executing Spark task. If that reading is right, each task gets its own 100 GB allowance and an executor running N concurrent tasks can reach N times that. The aggregate is what someone protecting a shared disk needs to reason about, so stating the per-task scope would make the last sentence of this paragraph actionable.

[mbutrovich]

Same scope question as tuning.md:120. The expanded doc string is clearer than what it replaces, and adding that the limit applies per task rather than per executor would make it complete, since this string is what shows up in the generated configs.md table.

[mbutrovich]

Once this is rebased, main already has a ### Parquet Native Scans section (the row group split mismatch and issue #3817) currently nested under ## Shuffle at tuning.md:189. That leaves Parquet guidance in two places with one of them under the wrong parent. Would you move that section under this new ## Parquet Reader Tuning heading as part of the rebase?

[mbutrovich]

This is an h3 under ## Shuffle, but spark.comet.exec.transitionRevert.* counts columnar-to-row transitions across a whole stage and is not shuffle specific. The maxTransitions doc string mentions columnar shuffle only as a proxy for why each C2R implies a matching R2C. Promoting this to its own h2 would keep the ## Shuffle section scoped to shuffle.

[mbutrovich]

Both paragraphs scope compression to native shuffle, which leaves the columnar (JVM) shuffle path unaddressed even though the guide documents it above at line 199. spark.comet.exec.shuffle.compression.codec is read by CometShuffleExternalSorter.java:137 and CometDiskBlockWriter.java:145, so it governs columnar shuffle too. And spark.shuffle.compress is consulted only in CometNativeShuffleWriter.scala:223, so setting it to false does not disable compression for columnar shuffle. Could you say which knob applies to which writer? Replacing the old ZSTD claim was the right call and this would finish the correction.

[mbutrovich]

COMET_ICEBERG_NATIVE_ENABLED is createWithDefault(true), so spark.comet.scan.icebergNative.enabled=true reads here as an opt-in when it is already the default. The rest of this section matches the source: dataFileConcurrencyLimit defaults to 1 and its doc string suggests 2 to 8.

[andygrove]

Thanks for the thorough review, @mbutrovich! I merged main and pushed fixes for all seven comments in 266f251:

• Parallel I/O section — dropped entirely; the four configs were removed in chore: remove dead Parquet parallel-IO configs #4981.
• maxTempDirectorySize scope — reworded in both tuning.md and the CometConf doc string to make clear the limit is per Spark task, so an executor running N concurrent tasks may use up to N× this value on shared disks.
• ### Parquet Native Scans — moved from under ## Shuffle to the new ## Parquet Reader Tuning section so all Parquet guidance is co-located.
• Reducing Row/Columnar Conversion Overhead — promoted from h3-under-Shuffle to its own h2, since transitionRevert.* counts C2R transitions across the whole stage and isn't shuffle-specific.
• Shuffle compression — rewrote the section to distinguish scopes: spark.comet.exec.shuffle.compression.codec governs both native and JVM columnar shuffle writers; spark.shuffle.compress=false only disables compression on the native path (JVM columnar always compresses).
• Iceberg wording — fixed to say spark.comet.scan.icebergNative.enabled is enabled by default, and noted dataFileConcurrencyLimit defaults to 1.

PTAL when you get a chance.

[mbutrovich]

LGTM pending #4986, which I'll go review now. Thanks @andygrove!