Skip to content

Minor: comments that explain the schema used in simply_expressions - #7747

Merged
alamb merged 3 commits into
apache:mainfrom
alamb:alamb/simplify_expr_docs
Oct 10, 2023
Merged

Minor: comments that explain the schema used in simply_expressions#7747
alamb merged 3 commits into
apache:mainfrom
alamb:alamb/simplify_expr_docs

Conversation

@alamb

@alambalamb commented Oct 5, 2023

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #7699

Rationale for this change

I am somewhat embarrassed that I did not fully understand @Dandandan 's fix in https://github.com/apache/arrow-datafusion/pull/7699/files and then spent some non trivial time tracking the same issue down in IOx and filed a ticket #7746 only to find it had already been fixed

What changes are included in this PR?

Add some comments to try and explain the change more generally

BTW I made a diagram explaining what is going on, that I wanted to share even though I am not sure it belongs in the comments directly:

 Query: SELECT t.a FROM t WHERE t.b > 5 ┌────────────────────┐ │ Projection(t.a) │ └────────────────────┘ ▲ │ │ ┌────────────────────┐ Filter ┌────────────────────┐ Projection ┌────────────────────┐ │ Filter(t.b > 5) │ ────Pushdown──▶ │ Projection(t.a) │ ────Pushdown──▶ │ Projection(t.a) │ └────────────────────┘ └────────────────────┘ └────────────────────┘ ▲ ▲ ▲ │ │ │ │ │ │ ┌────────────────────┐ ┌────────────────────┐ ┌────────────────────┐ │ Scan │ │ Scan │ │ Scan │ └────────────────────┘ │ filter=(t.b > 5) │ │ filter=(t.b > 5) │ └────────────────────┘ │ projection=(t.a) │ └────────────────────┘ Initial Plan Projection pushdown notes that If `TableProviderFilterPushDown` the scan only needs t.a returns true, filter pushdown pushes the filter into the scan BUT internally evaluating the predicate still requires t.b 

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added the optimizer Optimizer rules label Oct 5, 2023
Comment threaddatafusion/optimizer/src/simplify_expressions/simplify_exprs.rs Outdated
@comphead

Copy link
Copy Markdown
Contributor

The diagram is awesome, surely we can find a place for it in docs

@alamb

alamb commented Oct 6, 2023

Copy link
Copy Markdown
ContributorAuthor

The diagram is awesome, surely we can find a place for it in docs

I have included it in #7759

@alamb

Copy link
Copy Markdown
ContributorAuthor

@Dandandan or @comphead could I ask one of you to approve this PR (it needs a review prior to being able to merge):

Screenshot 2023-10-10 at 12 18 44 PM

@compheadcomphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@alamb
alamb merged commit 94153b4 into apache:mainOct 10, 2023
@alamb

Copy link
Copy Markdown
ContributorAuthor

Thank you @comphead

@alamb
alamb deleted the alamb/simplify_expr_docs branch October 10, 2023 17:40
devinjdangelo pushed a commit to devinjdangelo/arrow-datafusion that referenced this pull request Oct 11, 2023
…pache#7747)
* Minor: comments that explain the schema used in simply_expressions
* Apply suggestions from code review
Co-authored-by: comphead <comphead@users.noreply.github.com>
---------
Co-authored-by: comphead <comphead@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizerOptimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@alamb@comphead