Add logical range partitioning representation

[gene-bordegaray]

Which issue does this PR close?

• ClosesAdd logical range partitioning representation #22778.
• Related: [DISCUSSION] Extending Partitioning to Support More Variants #21992, [EPIC] Implement Range Partitioning #22395.
• Needed by Add ListingOptions::output_partitioning and FileScanConfig::output_partitioning for pre-defined file partitioning #22657.

Rationale for this change

Declared scan output partitioning should use logical partitioning metadata, not physical partitioning types. This adds logical range partitioning so range-partitioned sources can declare their layout at the logical layer.

What changes are included in this PR?

• Add logical Partitioning::Range and RangePartitioning.
• Move SplitPoint and shared split-point validation to datafusion-common.
• Wire logical range partitioning through expression traversal, rewrites, and display.
• Keep planning, logical proto, and Substrait support explicitly unsupported for now.

Are these changes tested?

Yes. Unit tests added

Are there any user-facing changes?

Yes. This adds public logical range partitioning API. No breaking API changes.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

 Cloning apache/main
Building datafusion v54.0.0 (current)
Built [ 91.233s] (current)
Parsing datafusion v54.0.0 (current)
Parsed [ 0.031s] (current)
Building datafusion v54.0.0 (baseline)
Built [ 90.354s] (baseline)
Parsing datafusion v54.0.0 (baseline)
Parsed [ 0.032s] (baseline)
Checking datafusion v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.696s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 184.393s] datafusion
Building datafusion-common v54.0.0 (current)
Built [ 30.327s] (current)
Parsing datafusion-common v54.0.0 (current)
Parsed [ 0.055s] (current)
Building datafusion-common v54.0.0 (baseline)
Built [ 30.417s] (baseline)
Parsing datafusion-common v54.0.0 (baseline)
Parsed [ 0.055s] (baseline)
Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.769s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 62.597s] datafusion-common
Building datafusion-expr v54.0.0 (current)
Built [ 24.126s] (current)
Parsing datafusion-expr v54.0.0 (current)
Parsed [ 0.069s] (current)
Building datafusion-expr v54.0.0 (baseline)
Built [ 23.997s] (baseline)
Parsing datafusion-expr v54.0.0 (baseline)
Parsed [ 0.069s] (baseline)
Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 1.438s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron
Failed in:
variant Partitioning:Range in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4451
variant Partitioning:Range in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4451
--- warning partial_ord_enum_variants_reordered: enum variants reordered in #[derive(PartialOrd)] enum ---
Description:
A public enum that derives PartialOrd had its variants reordered. #[derive(PartialOrd)] uses the enum variant order to set the enum's ordering behavior, so this change may break downstream code that relies on the previous order.
ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/partial_ord_enum_variants_reordered.ron
Failed in:
Partitioning::DistributeBy moved from position 3 to 4, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4453
Partitioning::DistributeBy moved from position 3 to 4, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:4453
Summary semver requires new major version: 1 major and 0 minor checks failed
Warning produced 1 major and 0 minor level warnings
Finished [ 50.576s] datafusion-expr
Building datafusion-physical-expr v54.0.0 (current)
Built [ 26.646s] (current)
Parsing datafusion-physical-expr v54.0.0 (current)
Parsed [ 0.048s] (current)
Building datafusion-physical-expr v54.0.0 (baseline)
Built [ 26.491s] (baseline)
Parsing datafusion-physical-expr v54.0.0 (baseline)
Parsed [ 0.046s] (baseline)
Checking datafusion-physical-expr v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.405s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure struct_missing: pub struct removed or renamed ---
Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/struct_missing.ron
Failed in:
struct datafusion_physical_expr::SplitPoint, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/2231dfcfec6ed67c8b624f6f394b6d09d3843f69/datafusion/physical-expr/src/partitioning.rs:225
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 54.445s] datafusion-physical-expr
Building datafusion-proto v54.0.0 (current)
Built [ 53.620s] (current)
Parsing datafusion-proto v54.0.0 (current)
Parsed [ 0.017s] (current)
Building datafusion-proto v54.0.0 (baseline)
Built [ 53.196s] (baseline)
Parsing datafusion-proto v54.0.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.292s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 108.517s] datafusion-proto
Building datafusion-substrait v54.0.0 (current)
Built [ 313.977s] (current)
Parsing datafusion-substrait v54.0.0 (current)
Parsed [ 0.016s] (current)
Building datafusion-substrait v54.0.0 (baseline)
Built [ 315.378s] (baseline)
Parsing datafusion-substrait v54.0.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-substrait v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.239s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 631.359s] datafusion-substrait

[gene-bordegaray]

cc: @gabotechs@stuhood

[stuhood]

Thanks!

[stuhood]

This is a TODO, right? Should it point at a ticket?

[gene-bordegaray]

yes good catch I can add the epic: #22395

[gene-bordegaray]

I created small issues for each not impl. They are not very great descriptions but for tracking for now

• Support planning logical range repartitioning #22786
• Support logical protobuf serialization for range repartitioning #22787
• Support Substrait exchange output for range repartitioning #22788

[gabotechs]

Just left some non blocking comments. Pretty straight forward PR! thanks @gene-bordegaray and @stuhood.

[gene-bordegaray]

Sorry for not responding when travelling but addressed comments, thanks @gabotechs 😄

[gabotechs]

I know 👍 it's completely fine, you responded with code so that's good enough!