SPARK-12105 - SPARK-SQL add convenient show functions

[jbonofre]

No description provided.

[jbonofre]

Rebase

[andrewor14]

these signatures are really clunky and I don't think we want it. Also the string is already composed so it's not like we're streaming things to the writer anyway.

[andrewor14]

@jbonofre I think it might be sufficient to just expose showString. The new methods you added are kind of clunky to use. It might be good to also have an alias so the user can just call val x = df.showString().

[andrewor14]

add to whitelist

[SparkQA]

Test build #47589 has finished for PR 10130 at commit edbb87c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class ExecutorClassLoader(\n

[jbonofre]

Thanks for the update @andrewor14. It makes sense for the new functions(). Exposing the showString() is enough for most of the cases. Let me update the PR.

[SparkQA]

Test build #47594 has finished for PR 10130 at commit 219d75c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

no longer used

[SparkQA]

Test build #47720 has finished for PR 10130 at commit 694e94f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

remove ;, this is scala. I'll fix this on merge

[andrewor14]

Merged into master.

[rxin]

Sorry I reverted the patch. Please check with me on the API change in the future.

First I'm not 100% sure whether we want this (we might, just not sure yet), and then there are problems with the API (e.g. we should not use any default values for Java compatibility). Last but not least, if we want to do this, we should have it also for Python.

[jbonofre]

Hi Reynold. No problem, I just proposed the PR as least to discuss, and I think it's convenient for users. Thanks.

[jbonofre]

By the way, let me know if you want the change, I can extend to Python API as well. And what do you mean by default values about Java compatibility ?

[rxin]

Default parameter values only work in Scala, not Java.

[jjzazuet]

Hi. So it's 2018 and Dataset.scala still has

private[sql] def showString(_numRows: Int, truncate: Int = 20): String

I also vouch for having this method exposed as public to capture into logs in the driver program.

Thanks!

[dmateusp]

+1 any chance we can revive this PR ?

[tmccartan]

+1

[davis-x]

+1

[yousefpal]

+1

[loicdescotte]

+1 , it think would be very useful to get the show as string, to inject into logs, we are currently forced to use workarounds like

packageorg.apache.spark.sqlobjectAccessShowString {
defshowString[T](df: Dataset[T],
_numRows: Int, truncate: Int=20, vertical: Boolean=false):String= {
df.showString(_numRows, truncate, vertical)
}
}

or

defdataFrameToString(df: DataFrame, numRows: Int=20, truncate: Int=20):String= {
valoutput=newByteArrayOutputStream()
valprintStream=newPrintStream(output)
Console.withOut(printStream) {
df.show(numRows, truncate)
}
output.toString.trim
}

(solutions from stackoverflow)

which is bad

@rxin would it be possible to reopen the issue or consider changing the visibility ?

Thanks