[SPARK-16669][SQL]Adding partition prunning to Metastore statistics f…

[Parth-Brahmbhatt]

What changes were proposed in this pull request?

Adding partition prunning to Metastore statistics for better join selection.

Currently the metastore statistics returns the size of entire table which results in Join selection stretagy to not use broadcast joins even when only a single partition from a large table is selected.This PR addresses that issue by only estimating the size of the partition by applying partition pruning during size estimation. Currently it only works with partition columns used with equality checks under AND,OR,IN Operators. If a partition column is used in any other operator, it defaults back to total table size. We have also introduced a configuration to enable this optimization which will be off by default. Instead of trying to calculate the path we could make a metastore query to get all the valid paths but for simplicity we are just building the path in code.

How was this patch tested?

Unit tests added.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

cc @cloud-fan and @gatorsmile - both are working on refactoring some of these code.

[cloud-fan]

If we gonna do this, I'd like to have a more general approach, which should also work for data source tables.

[gatorsmile]

Will this be part of the CBO work? The size estimation or statistics collection is being re-designed for CBO, right?

[Parth-Brahmbhatt]

@cloud-fan How do you suggest to change this? I started with Metastore as internally that is the most used datasource and will benefit from partition pruning at planning stage. I am open to any suggestions and will modify the code accordingly.

[gatorsmile]

Found a related JIRA: https://issues.apache.org/jira/browse/SPARK-17129

[Parth-Brahmbhatt]

@gatorsmile not sure if its the same issue. The issue you are pointing at talks about storing the actual partition level stats, which could be used by this PR but until its available we could rely on file system level statistics. Also given this is config driven which is disabled by default it should have no perf impact.

[gatorsmile]

How about waiting for a few days until that is delivered? Let us see whether that might simplify your PR.

[Parth-Brahmbhatt]

@gatorsmile not sure if it will simplify much in this case as most of the complexity is in figuring out what partitions can be pruned which I don't think will go away. We will rely on hive metastore instead of hdfs for size calculation whenever partition level stats are stored and available but that part of the code is not really complex.

I am fine waiting for the patch to be delivered.

[gatorsmile]

Thank you!

[gatorsmile]

@Parth-Brahmbhatt Are you still interested in this PR? Our stats refactoring has been finished in the release of 2.2. Thank you!

[Parth-Brahmbhatt]

I will re-evaluate and update or close the PR.

[gatorsmile]

@Parth-Brahmbhatt Thank you!

[wzhfy]

Seems this PR aims to solve similar problem as SPARK-15616?

[lianhuiwang]

@wzhfy Yes, I think this is same with SPARK-15616.

[Parth-Brahmbhatt]

Closing this PR given its a duplicate at this point.