[SPARK-6800][SQL] Update doc for JDBCRelation's columnPartition

[viirya]

JIRA https://issues.apache.org/jira/browse/SPARK-6800

[SparkQA]

Test build #30157 has finished for PR 5488 at commit 3e7db15.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[marmbrus]

This is not a bug, this is the intended behavior. I think the right fix here is to update the programming guide and other documentation to clearly state this. If users want to limit the scope of the data, they should add predicates which will be pushed down to the database.

[viirya]

ok. If I understand it correctly, lowerBound and upperBound are just used to decide partition stride, not for filtering. So all table rows are partitioned.

[marmbrus]

That is correct.

[SparkQA]

Test build #30296 has finished for PR 5488 at commit 3eb74d6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[marmbrus]

The parameters minValue and maxValue are advisory in that incorrect values may cause the partitioning to be poor, but no data will fail to be represented.

The sentence above already explains that the filters are only used for partitioning and that all data will always be returned. I think the best place to update would be in the SQL programming guide, in the table under the section "JDBC To Other Databases".

[viirya]

Updated.

[SparkQA]

Test build #30307 timed out for PR 5488 at commit 1dcc929 after a configured wait of 120m.

[micaelcapitao]

Hi.
The doc for the SQLContext.jdbc() method says:

@PARAM columnName the name of a column of integral type that will be used for partitioning.
@PARAM lowerBound the minimum value of columnName to retrieve
@PARAM upperBound the maximum value of columnName to retrieve
@PARAM numPartitions the number of partitions. the range minValue-maxValue will be split
evenly into this many partitions

This doc should be updated too.

How can one add predicates to limit the scope of data being pushed from the DB using the SQLContext API? Will a select limiting that scope make the table not to be pushed entirely?

[viirya]

@micaelcapitao Thanks. I updated the doc too.

I think you can use jdbc data source API to create temporary table and then use WHERE clause to add predicates.

[marmbrus]

That's correct. WHERE clause predicates and data frame filter operations
will be pushed down to the database.
On Apr 15, 2015 9:37 AM, "Liang-Chi Hsieh" notifications@github.com wrote:

@micaelcapitaohttps://github.com/micaelcapitao Thanks. I updated the
doc too.

I think you can use jdbc data source API to create temporary table and
then use WHERE clause to add predicates.

—
Reply to this email directly or view it on GitHub
#5488 (comment).

[SparkQA]

Test build #30354 has finished for PR 5488 at commit 51386c8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[marmbrus]

Thanks! Merged to master and branch-1.3