[SPARK-16429][SQL] Include StringType columns in describe()

[dongjoon-hyun]

What changes were proposed in this pull request?

Currently, Spark describe supports StringType. However, describe() returns a dataset for only all numeric columns. This PR aims to include StringType columns in describe(), describe without argument.

Background

scala> spark.read.json("examples/src/main/resources/people.json").describe("age", "name").show()
+-------+------------------+-------+|summary| age| name|+-------+------------------+-------+| count|2|3|| mean|24.5|null|| stddev|7.7781745930520225|null|| min|19|Andy|| max|30|Michael|+-------+------------------+-------+

Before

scala> spark.read.json("examples/src/main/resources/people.json").describe().show()
+-------+------------------+|summary| age|+-------+------------------+| count|2|| mean|24.5|| stddev|7.7781745930520225|| min|19|| max|30|+-------+------------------+

After

scala> spark.read.json("examples/src/main/resources/people.json").describe().show()
+-------+------------------+-------+|summary| age| name|+-------+------------------+-------+| count|2|3|| mean|24.5|null|| stddev|7.7781745930520225|null|| min|19|Andy|| max|30|Michael|+-------+------------------+-------+

How was this patch tested?

Pass the Jenkins with a update testcase.

[rxin]

private rather than private sql?

[dongjoon-hyun]

That would be better.

[dongjoon-hyun]

Thank you for fast review, @rxin . I updated it.

[SparkQA]

Test build #61929 has finished for PR 14095 at commit df2edd7.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Oh, it's a documented behavior.

Computes statistics for **numeric** columns

[SparkQA]

Test build #61930 has finished for PR 14095 at commit b6673cb.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Can you fix Python?

[dongjoon-hyun]

Oh, sure!

[rxin]

And also update the documentation.

[dongjoon-hyun]

Of course!

[dongjoon-hyun]

I fixed Python/R and the docs accordingly, and tested locally.

[rxin]

is it possible that this would fail?

[dongjoon-hyun]

Ur, this is an direct extension of line 225 of existing numericColumns.

https://github.com/apache/spark/pull/14095/files#diff-7a46f10c3cedbf013cf255564d9483cdR225

[dongjoon-hyun]

You mean the failure of resolveQuoted?

[dongjoon-hyun]

It will not fail because the names come from schema.fields.

[SparkQA]

Test build #61962 has finished for PR 14095 at commit 8915adb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @rxin .
It's ready for review again.

[SparkQA]

Test build #61965 has finished for PR 14095 at commit fa4d3b4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - merging in master.

[dongjoon-hyun]

Thank you for merging, @rxin .