Skip to content

feat: allow pushdown of dynamic filters having partition cols - #18172

Merged
adriangb merged 10 commits into
apache:mainfrom
feniljain:feat-dynamic-filter-join-partition-col
Oct 29, 2025
Merged

feat: allow pushdown of dynamic filters having partition cols#18172
adriangb merged 10 commits into
apache:mainfrom
feniljain:feat-dynamic-filter-join-partition-col

Conversation

@feniljain

@feniljainfeniljain commented Oct 20, 2025

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Included in the issue

Are these changes tested?

While I have tested this on local with a local TPCDS-like dataset, I would appreciate if someone provides me a good way to add tests for the same 😅

@github-actionsgithub-actionsBot added the datasource Changes to the datasource crate label Oct 20, 2025
@feniljain
feniljainforce-pushed the feat-dynamic-filter-join-partition-col branch from 2e64a1e to 1a02eaeCompareOctober 20, 2025 09:38
@feniljainfeniljain changed the title feat: allow pushdown of dynamic filters with expr having partition colsfeat: allow pushdown of dynamic filters having partition colsOct 20, 2025
@feniljain
feniljainforce-pushed the feat-dynamic-filter-join-partition-col branch 4 times, most recently from 115d3c1 to 2d8fb9cCompareOctober 20, 2025 10:08
adriangb
adriangb reviewed Oct 20, 2025
View reviewed changes
Comment threaddatafusion/datasource-parquet/src/source.rs Outdated
@adriangbadriangb mentioned this pull request Oct 20, 2025
Closed
adriangb
adriangb reviewed Oct 21, 2025
View reviewed changes
Comment threaddatafusion/datasource-parquet/src/source.rs Outdated
@github-actionsgithub-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Oct 22, 2025
@feniljain
feniljainforce-pushed the feat-dynamic-filter-join-partition-col branch 2 times, most recently from e58d9ec to 2c1ee6eCompareOctober 22, 2025 06:20
@adriangb

adriangb commented Oct 22, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

@feniljain mind if I push 7818a42c7 to your branch / this PR?

@feniljain

feniljain commented Oct 22, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Sure thing! No worries, thanks for the patch :)

@github-actionsgithub-actionsBot added the core Core DataFusion crate label Oct 23, 2025
@adriangb
adriangbforce-pushed the feat-dynamic-filter-join-partition-col branch from 8290691 to 05284cfCompareOctober 24, 2025 14:12
@github-actionsgithub-actionsBot added the documentation Improvements or additions to documentation label Oct 24, 2025
@adriangb
adriangbforce-pushed the feat-dynamic-filter-join-partition-col branch from 22079c6 to 503a5d8CompareOctober 27, 2025 01:52
@adriangb

adriangb commented Oct 27, 2025

Copy link
Copy Markdown
Contributor

@feniljain thanks for your patience. I had to resolve some conflicts but I think this is looking good now. I'll give it one last review tomorrow and then we can probably merge it.

adriangb
adriangb reviewed Oct 27, 2025
View reviewed changes
Comment threaddocs/source/library-user-guide/upgrading.md
_execution plan_ of the query. With this release, `DESCRIBE query` now outputs
the computed _schema_ of the query, consistent with the behavior of `DESCRIBE table_name`.

### Introduction of `TableSchema` and changes to `FileSource::with_schema()` method

@adriangbadriangbOct 27, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that TableSchema was not added in this PR but this PR does use it in a way that introduces a notable breaking change, so I think it's an appropriate time to add it to the upgrade guide.

friendlymatthew
friendlymatthew approved these changes Oct 27, 2025
View reviewed changes

@friendlymatthewfriendlymatthew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me. Thanks @feniljain

@adriangb
adriangb enabled auto-merge October 27, 2025 15:49
@adriangbadriangb mentioned this pull request Oct 27, 2025
Closed
@adriangb
adriangbforce-pushed the feat-dynamic-filter-join-partition-col branch from 503a5d8 to b6375bbCompareOctober 28, 2025 16:03
@adriangb

adriangb commented Oct 28, 2025

Copy link
Copy Markdown
Contributor

@alamb, @xudong963 or @Jefffrey could one of you approve or review this PR if you have time? It's good to me but since I pushed some commits I cannot approve it. I'd like to merge it before it gets more conflicts (I've had to resolve already).

feniljainand others added 8 commits October 28, 2025 18:39
@feniljain@adriangb
test: update sqllogictests
c7bd8d8
@adriangb
use TableSchema
b5daaa3
@adriangb
add helper methods
2a54d3e
@adriangb
lint
da2d358
@adriangb@claude
docs: Add upgrade guide entry for TableSchema and FileSource changes
9fcefcb
Added documentation explaining the introduction of the TableSchema
struct and the breaking change to FileSource::with_schema() method
signature (changed from SchemaRef to TableSchema).
Includes:
- Overview of TableSchema purpose and structure
- Who is affected by the change
- Migration guide for custom FileSource implementations
- Code examples showing how to update implementations
- Examples of using TableSchema directly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
@adriangb
fix merge
991110d
@adriangb
fmt docs
8d6597d
@adriangb
some touches
220060d
@adriangb
adriangbforce-pushed the feat-dynamic-filter-join-partition-col branch from b6375bb to 220060dCompareOctober 28, 2025 23:39
@Jefffrey

Jefffrey commented Oct 29, 2025

Copy link
Copy Markdown
Contributor

@alamb, @xudong963 or @Jefffrey could one of you approve or review this PR if you have time? It's good to me but since I pushed some commits I cannot approve it. I'd like to merge it before it gets more conflicts (I've had to resolve already).

I can try take a look at this if I'm able, though I am not too familiar with dynamic filters in DataFusion

@alamb

alamb commented Oct 29, 2025

Copy link
Copy Markdown
Contributor

@alamb, @xudong963 or @Jefffrey could one of you approve or review this PR if you have time? It's good to me but since I pushed some commits I cannot approve it. I'd like to merge it before it gets more conflicts (I've had to resolve already).

FWIW I don't think there is any reason you can't approve it after pushing commits

@adriangb

adriangb commented Oct 29, 2025

Copy link
Copy Markdown
Contributor

@alamb, @xudong963 or @Jefffrey could one of you approve or review this PR if you have time? It's good to me but since I pushed some commits I cannot approve it. I'd like to merge it before it gets more conflicts (I've had to resolve already).

FWIW I don't think there is any reason you can't approve it after pushing commits

Hmm it didn't seem to let me hit the queue button but I guess as a matter of procedure probably good to get another approval anyway :)

@adriangb
adriangb added this pull request to the merge queueOct 29, 2025
Merged via the queue into apache:main with commit dfba228Oct 29, 2025
29 checks passed
@adriangb

adriangb commented Oct 29, 2025

Copy link
Copy Markdown
Contributor

Huh got it to go now!

tschwarzinger pushed a commit to tschwarzinger/datafusion that referenced this pull request Nov 2, 2025
@feniljain@adriangb
@claude@tschwarzinger
feat: allow pushdown of dynamic filters having partition cols (apache…
fe3426c
…#18172)
## Which issue does this PR close?
- Closesapache#18171
## Rationale for this change
Included in the issue
## Are these changes tested?
While I have tested this on local with a local TPCDS-like dataset, I
would appreciate if someone provides me a good way to add tests for the
same 😅
---------
Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
codetyri0n pushed a commit to codetyri0n/datafusion that referenced this pull request Nov 11, 2025
@feniljain@adriangb
@claude@codetyri0n
feat: allow pushdown of dynamic filters having partition cols (apache…
9b33c92
…#18172)
## Which issue does this PR close?
- Closesapache#18171
## Rationale for this change
Included in the issue
## Are these changes tested?
While I have tested this on local with a local TPCDS-like dataset, I
would appreciate if someone provides me a good way to add tests for the
same 😅
---------
Co-authored-by: Adrian Garcia Badaracco <1755071+adriangb@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
@feniljain
feniljain deleted the feat-dynamic-filter-join-partition-col branch November 19, 2025 05:23
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adriangbadriangbadriangb approved these changes
+1 more reviewer
@friendlymatthewfriendlymatthewfriendlymatthew approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

coreCore DataFusion cratedatasourceChanges to the datasource cratedocumentationImprovements or additions to documentationsqllogictestSQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Dynamic filters with partition columns do not get pushed down through DataSourceExec

5 participants

@feniljain@adriangb@Jefffrey@alamb@friendlymatthew