Skip to content

Fix Filter / where clause without column names is removed in optimization pass - #225

Merged
jorgecarleitao merged 4 commits into
apache:masterfrom
Dandandan:where
Apr 30, 2021
Merged

Fix Filter / where clause without column names is removed in optimization pass#225
jorgecarleitao merged 4 commits into
apache:masterfrom
Dandandan:where

Conversation

@Dandandan

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#220

Rationale for this change

DataFusion gives wrong results for filters that don't reference any column

What changes are included in this PR?

Workarounds the issue by not pushing down predicates that do not reference any column (1=2, FALSE, TRUE).
As we should be able to remove these anyway with constant folding and removing/replacing WHERE FALSE/ WHERE TRUE, I think this is a good option.

Are there any user-facing changes?

No, results should be correct now.

@DandandanDandandan changed the title Fix Filter /where clause without column names is removed in optimization pass #220Fix Filter /where clause without column names is removed in optimization passApr 29, 2021
@DandandanDandandan changed the title Fix Filter /where clause without column names is removed in optimization passFix Filter / where clause without column names is removed in optimization passApr 29, 2021
@codecov-commenter

codecov-commenter commented Apr 29, 2021

Copy link
Copy Markdown

Codecov Report

Merging #225 (94c5b3a) into master (e86ad26) will increase coverage by 0.12%.
The diff coverage is 100.00%.

Impacted file tree graph

@@ Coverage Diff @@## master #225 +/- ##
==========================================
+ Coverage 76.30% 76.43% +0.12% 
==========================================
Files 134 135 +1 Lines 23170 23264 +94 ==========================================
+ Hits 17681 17782 +101 + Misses 5489 5482 -7 
Impacted FilesCoverage Δ
datafusion/src/optimizer/filter_push_down.rs97.73% <100.00%> (+0.07%)⬆️
datafusion/tests/sql.rs99.88% <100.00%> (+1.42%)⬆️
datafusion/src/catalog/information_schema.rs86.72% <0.00%> (-2.06%)⬇️
datafusion/src/physical_plan/hash_join.rs85.94% <0.00%> (-0.49%)⬇️
datafusion/src/execution/context.rs92.63% <0.00%> (-0.31%)⬇️
ballista/rust/executor/src/main.rs0.00% <0.00%> (ø)
datafusion/src/datasource/datasource.rs100.00% <0.00%> (ø)
datafusion/src/optimizer/eliminate_limit.rs88.88% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e86ad26...94c5b3a. Read the comment docs.

Comment threaddatafusion/src/optimizer/filter_push_down.rs Outdated
@jorgecarleitao
jorgecarleitao merged commit 2423ff0 into apache:masterApr 30, 2021

@jorgecarleitaojorgecarleitao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, ready to sail! :)

@alamb

alamb commented May 3, 2021

Copy link
Copy Markdown
Contributor

Thanks @Dandandan

@houqphouqp added the bug Something isn't working label Jul 31, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bugSomething isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filter / where clause without column names is removed in optimization pass

5 participants

@Dandandan@codecov-commenter@alamb@houqp@jorgecarleitao