[SPARK-16727][SparkR] Fix expected test output of describe and summary functions

[junyangq]

What changes were proposed in this pull request?

Fix expected test output of describe and summary functions. String columns are not summarized in the output.

How was this patch tested?

SparkR unit test.

[shivaram]

@junyangq Any idea how the tests were passing on Jenkins before this fix ?

[shivaram]

I think this is related to 142df48

cc @dongjoon-hyun

[SparkQA]

Test build #62852 has finished for PR 14357 at commit f7d02ce.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @shivaram.
Sorry, but what is the problem?

[dongjoon-hyun]

String columns works for min and max summarization.

[junyangq]

@shivaram I'm also curious...The issue arised when I ran the test locally.

[dongjoon-hyun]

> collect(describe(read.json('examples/src/main/resources/people.json')))
summaryagename1count232mean24.5<NA>3stddev7.7781745930520225<NA>4min19Andy5max30Michael

[dongjoon-hyun]

In fact, count, min, and max are supported officially for string in all Scala/Python/R.

[dongjoon-hyun]

Hi, @junyangq .
Could you give me some pointer to understand the current context? :)

[junyangq]

@dongjoon-hyun That makes sense. That's why I feel confused about the output on my local machine...

> collect(describe(read.json('examples/src/main/resources/people.json')))
summary age 1 count 2
2 mean 24.5
3 stddev 7.7781745930520225
4 min 19
5 max 30

[dongjoon-hyun]

I see. Which version is it?

[dongjoon-hyun]

I see. That seems 2.0 branch.

[dongjoon-hyun]

Hi, @shivaram .
It's due to the branch difference.
The PR is merged to the master only 17 days ago.

Thanks - merging in master.

[junyangq]

I think I ran the test under the master branch though... Let me double check :)

[dongjoon-hyun]

Thank you, @junyangq .

[junyangq]

I merged the most recent master branch, rebuilt and installed the package, but the test failed at the same place. @dongjoon-hyun

[dongjoon-hyun]

Oh, let me try this time. Thank you for double-checking, @junyangq .

[dongjoon-hyun]

@junyangq . Currently, you are at the most recent master build, right?
And, SparkR does not show the name column as a result of describe command.
I'm wondering the result of spark-shell on your systems. Could you run the following command in spark-shell?

scala> spark.read.json("examples/src/main/resources/people.json").describe().show()

[junyangq]

Hmm... It doesn't show the name column either.

[dongjoon-hyun]

At the most recent master build, I did the following things and all tests are passed.

$ git clean -fdx
$ ./build/sbt -Pyarn -Phadoop-2.3 -Pkinesis-asl -Phive-thriftserver -Phive -Psparkr package streaming-kafka-0-8-assembly/assembly streaming-flume-assembly/assembly streaming-kinesis-asl-assembly/assembly
$ R/install-dev.sh $ R/run-tests.sh ...
DONE ===========================================================================
Tests passed.

[dongjoon-hyun]

I think you did something wrong. :)
Could you close this PR?

[dongjoon-hyun]

Actually, you can see the Jenkins log, too. There is no problem with the current R testsuite.
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/62858/consoleFull

FYI, I'm using JDK 1.8.0_102 and Jenkins is using JDK 1.7.x.

[junyangq]

Yeah sure, but just wondering if the clean and additional arguments something we should normally do?

[dongjoon-hyun]

Never. I did that in order to make it sure. I don't do it frequently.
It is for you. :)

[dongjoon-hyun]

If you use -fdx, it removes IDE (like Intellij) settings, too.

[junyangq]

I see. Thank you for pointing that out :) I'll close the PR.

[dongjoon-hyun]

It's my pleasure. See you later around Apache Spark. :)