SPARK-1487 [SQL] Support record filtering via predicate pushdown in Parquet

[AndreSchumacher]

Simple filter predicates such as LessThan, GreaterThan, etc., where one side is a literal and the other one a NamedExpression are now pushed down to the underlying ParquetTableScan. Here are some results for a microbenchmark with a simple schema of six fields of different types where most records failed the test:

	Uncompressed	Compressed
File size	10 GB	2 GB
Speedup	2	1.8

Since mileage may vary I added a new option to SparkConf:

org.apache.spark.sql.parquet.filter.pushdown

Default value would be true and setting it to false disables the pushdown. When most rows are expected to pass the filter or when there are few fields performance can be better when pushdown is disabled. The default should fit situations with a reasonable number of (possibly nested) fields where not too many records on average pass the filter.

Because of an issue with Parquet (see here) currently only predicates on non-nullable attributes are pushed down. If one would know that for a given table no optional fields have missing values one could also allow overriding this.

[AndreSchumacher]

@marmbrus would be great if you could have a look when you have some time. Thanks!

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AndreSchumacher]

This PR will need to be revised depending on the outcome of #482

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14381/

[marmbrus]

How hard would it be to move the logic that determines what filters can be pushed down here, into the planner, so that we can avoid the double evaluation?

[marmbrus]

Very cool. This should be a pretty big performance win. Only minor comments.

Regarding the question about normalizing expressions, it could be done in a rule. However, really I think we can probably greatly simplify all of that logic with code gen (hopefully coming in 1.1). So, given that you have already written out all of the cases I don't think we need to do further simplification in this PR.

[mateiz]

@marmbrus@AndreSchumacher do we really want a SparkConf option for this? I'd rather minimize the number of options and add rules in the optimizer later to decide when to do this. These kind of options are esoteric and very hard for users to configure.

[marmbrus]

@mateiz good point. I agree with you long-term this decision should be up to the optimizer. However, in this case I think the right thing is probably to create a section of the sql config called hints. We don't promise to obey or support hints long term, but they can be used for immediate optimizations. This seems like the typical DB way to offset the cases where the optimization is currently poor.

[mateiz]

But are there any realistic workloads where you'd want to turn this on all the time, or turn it off all the time? It seems that in an ad-hoc query workload, you'll have some queries that can use this, and some that can't. You should just pick whether you want it as a default. Personally I'd go for it unless the cost is super high in the cases where it doesn't work, because I imagine filtering is pretty common in large schemas and I hope Parquet itself optimizes this down the line.

[mateiz]

BTW if you do add a config setting, a better name would be spark.sql.hints.parquetFilterPushdown; our other setting names don't start with org.apache. An even better option though might be to put it in the SQL statement itself, so users can do it on a per-query basis, though that probably requires nasty changes to the parser. But I'd still prefer to either always turn this on (if the penalty isn't huge) or leave it off for now and not introduce a new setting.

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AndreSchumacher]

@marmbrus@mateiz Thanks a lot for the comments and the fast response.

About the config setting: I would feel more comfortable setting a default after there has been some experience with realistic workloads and schemas. But I renamed it now, as suggested by Matei.

The bigger changes in my last commit are now to keep track of what is actually pushed and why. Then the predicates which are "completely" pushed are removed inside the Planner. Note that attempting to push "A & B" can result only in "A" being pushed because B contains anything other than a simple comparison of a column value. In this case "A & B" should be kept for now (IMHO). There is still in advantage in pushing A since hopefully there are fewer records that pass the filter to the higher level.

[AmplabJenkins]

Build triggered.

[AmplabJenkins]

Build started.

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14481/

[AmplabJenkins]

Build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14482/

[marmbrus]

However, we will never get A AND B right? As I think any conjunctive predicates will be split by using PhysicalOperation

[AndreSchumacher]

Good point, bad example. That's why I initially didn't treat ANDs at all when creating the filters from the expressions. But then I thought one could have expressions such as (A AND B) OR C which should probably be treated in the planner I guess and turned into (A OR C) AND (B OR C) but currently are not. Please correct me if I am wrong. It may be that the parser doesn't currently allow these kind of filter expressions with '(', ')' though although nothing speaks against them I guess.

[marmbrus]

Cool, thanks for renaming it.

@mateiz I don't think we should even include these hints in the docs (unless we find particularly useful ones) as I agree presenting too much complexity to users is a bad idea. However, even just for our own benchmarking, recompiling to change these settings is just not feasible and it's really hard to predict performance without actually running things. Also when I've talked about building catalyst to experienced database people, basically everyone said, "No matter how good you think your optimizer is, always make sure you have knobs to control it because it is going to be wrong."

Having these hints in the language could maybe be nice, but I really don't think that is worth the engineering effort of not only changing the parser, but also making sure they get threaded through analysis, optimization and planning correctly. Using language based hints would also would change if you are using sql, hql, or the DSL.

Having a special conf mechanism that lets you set them on a per query basis would be nice. I'm not sure how flexible the SparkConf infrastructure is in this regard, but might be something to consider. I can imagine cases where this might even be useful for standard spark jobs.

[mateiz]

Okay, it sounds good then as a hidden parameter.

[AmplabJenkins]

Merged build triggered.

[AndreSchumacher]

Jenkins, test this please

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15047/

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15048/

[AndreSchumacher]

@rxin Thanks for the note. I just rebased it.

[AndreSchumacher]

Closing this now since it got merged. Thanks everyone.