refactor(physical-plan): externalize statistics traversal into StatisticsContext

[asolimando]

Which issue does this PR close?

• ClosesDecouple operator statistics propagation from traversal/caching via statistics_from_inputs #22958.

Rationale for this change

Follow-up to #21815. Per @2010YOUY01's suggestion there, this decouples statistics
traversal/caching from each operator's computation: operators now express only local
propagation, while the walk and cache live in an external StatisticsContext.

What changes are included in this PR?

• StatisticsContext owns the bottom-up walk + per-walk cache; compute(plan, args)
  resolves children, then calls the operator.
• ExecutionPlan::statistics_from_inputs(input_stats, args) — stateless local
  propagation from pre-resolved child stats; default delegates to deprecated
  partition_statistics.
• ExecutionPlan::child_stats_requests(partition) -> Vec<ChildStats> — per-child
  directive (At(Option<usize>) / Skip) for which partition each child is computed
  at (e.g. broadcast joins request the build side overall; UnionExec skips
  non-owning inputs).
• Removes the unreleased statistics_with_args; partition_statistics stays
  deprecated.
• Migrates all operators/callers; updates the 55.0.0 upgrade guide.

Are these changes tested?

Yes — existing statistics tests now run through StatisticsContext::compute, plus new
unit tests for the cache and the compute_statistics benchmark.

Are there any user-facing changes?

Yes (public ExecutionPlan API): partition_statistics deprecated in favor of
statistics_from_inputs; new StatisticsContext / ChildStats; unreleased
statistics_with_args removed. Upgrade guide updated.

---

Reviewer note: statistics_from_inputs takes args: &StatisticsArgs (currently just
partition) as a non-breaking extension seam rather than a bare partition — happy
to change if preferred (see #22958).

---

Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

 Cloning apache/main
Building datafusion v54.0.0 (current)
Built [ 106.387s] (current)
Parsing datafusion v54.0.0 (current)
Parsed [ 0.034s] (current)
Building datafusion v54.0.0 (baseline)
Built [ 104.278s] (baseline)
Parsing datafusion v54.0.0 (baseline)
Parsed [ 0.035s] (baseline)
Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.633s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 212.976s] datafusion
Building datafusion-datasource v54.0.0 (current)
Built [ 38.423s] (current)
Parsing datafusion-datasource v54.0.0 (current)
Parsed [ 0.032s] (current)
Building datafusion-datasource v54.0.0 (baseline)
Built [ 38.231s] (baseline)
Parsing datafusion-datasource v54.0.0 (baseline)
Parsed [ 0.033s] (baseline)
Checking datafusion-datasource v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.247s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 77.995s] datafusion-datasource
Building datafusion-ffi v54.0.0 (current)
Built [ 60.040s] (current)
Parsing datafusion-ffi v54.0.0 (current)
Parsed [ 0.061s] (current)
Building datafusion-ffi v54.0.0 (baseline)
Built [ 61.060s] (baseline)
Parsing datafusion-ffi v54.0.0 (baseline)
Parsed [ 0.064s] (baseline)
Checking datafusion-ffi v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.241s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 122.674s] datafusion-ffi
Building datafusion-physical-optimizer v54.0.0 (current)
Built [ 39.439s] (current)
Parsing datafusion-physical-optimizer v54.0.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-physical-optimizer v54.0.0 (baseline)
Built [ 39.314s] (baseline)
Parsing datafusion-physical-optimizer v54.0.0 (baseline)
Parsed [ 0.022s] (baseline)
Checking datafusion-physical-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.129s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 80.119s] datafusion-physical-optimizer
Building datafusion-physical-plan v54.0.0 (current)
Built [ 37.361s] (current)
Parsing datafusion-physical-plan v54.0.0 (current)
Parsed [ 0.140s] (current)
Building datafusion-physical-plan v54.0.0 (baseline)
Built [ 37.926s] (baseline)
Parsing datafusion-physical-plan v54.0.0 (baseline)
Parsed [ 0.139s] (baseline)
Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.640s] 223 checks: 221 pass, 2 fail, 0 warn, 30 skip
--- failure inherent_method_missing: pub method removed or renamed ---
Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/inherent_method_missing.ron
Failed in:
StatisticsArgs::compute_child_statistics, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/3f697da4e542e016b2a8f02a2a7f5d525c982695/datafusion/physical-plan/src/statistics.rs:119
StatisticsArgs::compute_child_statistics, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/3f697da4e542e016b2a8f02a2a7f5d525c982695/datafusion/physical-plan/src/statistics.rs:119
--- failure trait_method_missing: pub trait method removed or renamed ---
Description:
A trait method is no longer callable, and may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#trait-item-signature
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/trait_method_missing.ron
Failed in:
method statistics_with_args of trait ExecutionPlan, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/3f697da4e542e016b2a8f02a2a7f5d525c982695/datafusion/physical-plan/src/execution_plan.rs:569
method statistics_with_args of trait ExecutionPlan, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/3f697da4e542e016b2a8f02a2a7f5d525c982695/datafusion/physical-plan/src/execution_plan.rs:569
Summary semver requires new major version: 2 major and 0 minor checks failed
Finished [ 77.301s] datafusion-physical-plan

[asolimando]

@2010YOUY01, @alamb: I have prepared this follow-up PR after #21815, it would be great if you could take a look at your convenience, I am in particular looking for guidance on the point raised in #22958 (comment).

cc: @xudong963 in case you are interested in taking a look at this PR as well.

Thank you folks for all the design and code feedback so far!

[xudong963]

The changes look good to me.

There are two issues are about the API migration boundary:

1. Old callers calling migrated nodes can get unknown stats. Many built-in operators now implement only statistics_from_inputs. But the deprecated partition_statistics default does not delegate to statistics_from_inputs; it just returns Statistics::new_unknown. So any remaining direct caller of partition_statistics can silently lose useful statistics.

2. New callers calling old custom nodes can do wasted child traversal.
   StatisticsContext::compute always computes child stats before calling statistics_from_inputs. For old third-party plans that only override partition_statistics, the default statistics_from_inputs
   ignores those child stats and delegates to partition_statistics. So the new API may perform extra work, or even surface child-stat errors, that the old implementation never required.

[asolimando]

Thanks @xudong963 for your feedback.

1. Old callers calling migrated nodes can get unknown stats. Many built-in operators now implement only statistics_from_inputs. But the deprecated partition_statistics default does not delegate to statistics_from_inputs; it just returns Statistics::new_unknown. So any remaining direct caller of partition_statistics can silently lose useful statistics.

This is correct but it's the same one-way-delegation tradeoff we accepted in #21815, it's already documented in the upgrade guide.

2. New callers calling old custom nodes can do wasted child traversal.
   StatisticsContext::compute always computes child stats before calling statistics_from_inputs. For old third-party plans that only override partition_statistics, the default statistics_from_inputs
   ignores those child stats and delegates to partition_statistics. So the new API may perform extra work, or even surface child-stat errors, that the old implementation never required.

This is indeed a regression w.r.t. #21815, whose default never walked children. I considered several options but I finally settled on flipping the default child_stats_requests to Skip to restore the behavior of #21815.

This way, a node that only overrides partition_statistics now declares no children, so there's no walk, and no risk of surfaced child errors. Nodes that read input_stats declare the children they use.

Side note: the semver check is technically correct but this is a change just between #21815 and this PR, so we don't need to go through a deprecation process as nothing was released in the meantime, correct?

[xudong963]

Side note: the semver check is technically correct but this is a change just between #21815 and this PR, so we don't need to go through a deprecation process as nothing was released in the meantime, correct?

yeah, makes sense

[asolimando]

@alamb@2010YOUY01, the PR is in good state after addressing the comments from @xudong963, the core changes are small and limited (number of affected files seems high but changes are mostly mechanical), would you be able to take a look and share your thoughts? Thanks!

[xudong963]

FYI, I plan to merge the PR, then @asolimando can iterate on the area. If you have other suggestions later, welcome to comment!

[alamb]

🎉