Uh oh!
There was an error while loading. Please reload this page.
[SPARK-12231][SQL]create a combineFilters' projection when we call buildPartitionedTableScan - #10388
[SPARK-12231][SQL]create a combineFilters' projection when we call buildPartitionedTableScan#10388kevinyu98 wants to merge 6 commits into
Conversation
kevinyu98
commented
Dec 21, 2015
@marmbrus : Can you help take a look at this PR? Thanks for your review. |
marmbrus
commented
Dec 21, 2015
ok to test |
SparkQA
commented
Dec 21, 2015
Test build #48127 has finished for PR 10388 at commit
|
There was a problem hiding this comment.
Don't call toSet. Anything involving attributes set logic should be done with an AttributeSet (which ignores cosmetic differences like capitalization).
There was a problem hiding this comment.
Hi Michael: Sure, will make the changes.
SparkQA
commented
Dec 24, 2015
Test build #48315 has finished for PR 10388 at commit
|
kevinyu98
commented
Dec 25, 2015
I delete the test cases from DataFrameNaFunctionsSuite.scala. I checked the previous failure, not sure why it is failed. I worked when I run the local test on my laptop. then I re-run the sql test buckets, seems fine. $ build/sbt sql/test-only [info] Passed: Total 1522, Failed 0, Errors 0, Passed 1522, Ignored 10 |
SparkQA
commented
Dec 25, 2015
Test build #48319 has finished for PR 10388 at commit
|
marmbrus
commented
Dec 28, 2015
Thanks, merging to master. |
…uildPartitionedTableScan Hello Michael & All: We have some issues to submit the new codes in the other PR(#10299), so we closed that PR and open this one with the fix. The reason for the previous failure is that the projection for the scan when there is a filter that is not pushed down (the "left-over" filter) could be different, in elements or ordering, from the original projection. With this new codes, the approach to solve this problem is: Insert a new Project if the "left-over" filter is nonempty and (the original projection is not empty and the projection for the scan has more than one elements which could otherwise cause different ordering in projection). We create 3 test cases to cover the otherwise failure cases. Author: Kevin Yu <qyu@us.ibm.com> Closes#10388 from kevinyu98/spark-12231. (cherry picked from commit fd50df4) Signed-off-by: Cheng Lian <lian@databricks.com>
liancheng
commented
Feb 1, 2016
Cherry-picked to branch-1.6. |
Hello Michael & All:
We have some issues to submit the new codes in the other PR(#10299), so we closed that PR and open this one with the fix.
The reason for the previous failure is that the projection for the scan when there is a filter that is not pushed down (the "left-over" filter) could be different, in elements or ordering, from the original projection.
With this new codes, the approach to solve this problem is:
Insert a new Project if the "left-over" filter is nonempty and (the original projection is not empty and the projection for the scan has more than one elements which could otherwise cause different ordering in projection).
We create 3 test cases to cover the otherwise failure cases.