Skip to content

feat: add ASOF joins to DataFrame API - #23831

Draft
Xuanwo wants to merge 5 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-dataframe
Draft

feat: add ASOF joins to DataFrame API#23831
Xuanwo wants to merge 5 commits into
apache:mainfrom
Xuanwo:xuanwo/asof-dataframe

Conversation

@Xuanwo

@XuanwoXuanwo commented Jul 23, 2026

Copy link
Copy Markdown
Member

Which issue does this PR close?

Rationale for this change

This is the DataFrame API layer of the ASOF JOIN stack. It exposes the logical
contract through the fluent API without coupling that API review to SQL parsing
or protobuf changes.

GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
Xuanwo/datafusion@xuanwo/asof-logical...xuanwo/asof-dataframe

What changes are included in this PR?

  • Add DataFrame::join_asof for expression-based equality keys.
  • Add DataFrame::join_asof_using for same-named equality keys.
  • Export AsOfMatch and Operator from the DataFusion prelude.
  • Add a DataFrame-only execution test that exercises the broadcast physical
    operator without depending on SQL syntax.

Are these changes tested?

Yes:

  • cargo fmt --all
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test -p datafusion --test core_integration dataframe::join_asof --all-features
  • The extended workspace test command from the contributor guide

Are there any user-facing changes?

Users can build ASOF joins through DataFrame::join_asof and
DataFrame::join_asof_using. The USING form follows DataFusion's existing
name-resolution contract: wildcard output exposes one unqualified key while
both qualified input keys remain addressable.

This PR should be reviewed and merged after #23829. It does not depend on the
optional floating-point follow-up #24375.

@github-actionsgithub-actionsBot added sql SQL Planner logical-expr Logical plan and expressions optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate proto Related to proto crate physical-plan Changes to the physical-plan crate labels Jul 23, 2026
@github-actions

github-actionsBot commented Jul 23, 2026

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
 Cloning apache/main
Building datafusion v55.0.0 (current)
Built [ 46.627s] (current)
Parsing datafusion v55.0.0 (current)
Parsed [ 0.030s] (current)
Building datafusion v55.0.0 (baseline)
Built [ 46.570s] (baseline)
Parsing datafusion v55.0.0 (baseline)
Parsed [ 0.030s] (baseline)
Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.737s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 95.911s] datafusion
Building datafusion-expr v55.0.0 (current)
Built [ 23.187s] (current)
Parsing datafusion-expr v55.0.0 (current)
Parsed [ 0.065s] (current)
Building datafusion-expr v55.0.0 (baseline)
Built [ 23.524s] (baseline)
Parsing datafusion-expr v55.0.0 (baseline)
Parsed [ 0.066s] (baseline)
Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.487s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 49.365s] datafusion-expr
Building datafusion-optimizer v55.0.0 (current)
Built [ 21.842s] (current)
Parsing datafusion-optimizer v55.0.0 (current)
Parsed [ 0.026s] (current)
Building datafusion-optimizer v55.0.0 (baseline)
Built [ 21.620s] (baseline)
Parsing datafusion-optimizer v55.0.0 (baseline)
Parsed [ 0.027s] (baseline)
Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.182s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 44.571s] datafusion-optimizer
Building datafusion-physical-plan v55.0.0 (current)
Built [ 30.434s] (current)
Parsing datafusion-physical-plan v55.0.0 (current)
Parsed [ 0.125s] (current)
Building datafusion-physical-plan v55.0.0 (baseline)
Built [ 30.538s] (baseline)
Parsing datafusion-physical-plan v55.0.0 (baseline)
Parsed [ 0.126s] (baseline)
Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.825s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 63.151s] datafusion-physical-plan
Building datafusion-proto v55.0.0 (current)
Built [ 42.981s] (current)
Parsing datafusion-proto v55.0.0 (current)
Parsed [ 0.015s] (current)
Building datafusion-proto v55.0.0 (baseline)
Built [ 42.957s] (baseline)
Parsing datafusion-proto v55.0.0 (baseline)
Parsed [ 0.016s] (baseline)
Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.131s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 87.384s] datafusion-proto
Building datafusion-sql v55.0.0 (current)
Built [ 33.587s] (current)
Parsing datafusion-sql v55.0.0 (current)
Parsed [ 0.029s] (current)
Building datafusion-sql v55.0.0 (baseline)
Built [ 33.679s] (baseline)
Parsing datafusion-sql v55.0.0 (baseline)
Parsed [ 0.029s] (baseline)
Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.261s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 68.537s] datafusion-sql
Building datafusion-substrait v55.0.0 (current)
Built [ 246.523s] (current)
Parsing datafusion-substrait v55.0.0 (current)
Parsed [ 0.016s] (current)
Building datafusion-substrait v55.0.0 (baseline)
Built [ 247.288s] (baseline)
Parsing datafusion-substrait v55.0.0 (baseline)
Parsed [ 0.015s] (baseline)
Checking datafusion-substrait v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.249s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 495.807s] datafusion-substrait

@github-actionsgithub-actionsBot added the auto detected api change Auto detected API change label Jul 23, 2026
@codecov-commenter

codecov-commenter commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 61.12469% with 159 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.52%. Comparing base (c4910e0) to head (f2bcdf6).
⚠️ Report is 20 commits behind head on main.

Files with missing linesPatch %Lines
datafusion/expr/src/logical_plan/plan.rs50.76%81 Missing and 15 partials ⚠️
datafusion/expr/src/logical_plan/builder.rs60.71%17 Missing and 5 partials ⚠️
datafusion/core/src/dataframe/mod.rs43.33%15 Missing and 2 partials ⚠️
datafusion/expr/src/logical_plan/display.rs0.00%10 Missing ⚠️
datafusion/core/src/physical_planner.rs82.92%5 Missing and 2 partials ⚠️
datafusion/expr/src/logical_plan/tree_node.rs93.02%0 Missing and 3 partials ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs92.85%1 Missing and 1 partial ⚠️
datafusion/optimizer/src/optimizer.rs50.00%0 Missing and 2 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #23831 +/- ##
==========================================
+ Coverage 81.47% 81.52% +0.04% 
==========================================
Files 1122 1123 +1 Lines 403629 406508 +2879 Branches 403629 406508 +2879 ==========================================
+ Hits 328866 331387 +2521 - Misses 55510 55725 +215 - Partials 19253 19396 +143 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@XuanwoXuanwo mentioned this pull request Jul 23, 2026
@github-actionsgithub-actionsBot added the physical-expr Changes to the physical-expr crates label Aug 14, 2026
@Xuanwo
Xuanwoforce-pushed the xuanwo/asof-dataframe branch from 90448bd to 7713821CompareAugust 20, 2026 17:56
@Xuanwo
Xuanwoforce-pushed the xuanwo/asof-dataframe branch from 7713821 to 7ef360bCompareAugust 28, 2026 05:40
@github-actionsgithub-actionsBot removed physical-expr Changes to the physical-expr crates physical-plan Changes to the physical-plan crate labels Aug 28, 2026
@github-actionsgithub-actionsBot added the physical-plan Changes to the physical-plan crate label Aug 30, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api changeAuto detected API changecoreCore DataFusion cratelogical-exprLogical plan and expressionsoptimizerOptimizer rulesphysical-planChanges to the physical-plan crateprotoRelated to proto cratesqlSQL PlannersubstraitChanges to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Xuanwo@codecov-commenter