[SPARK-26191][SQL] Control truncation of Spark plans via maxFields parameter by MaxGekk · Pull Request #23159 · apache/spark

MaxGekk · 2018-11-27T22:51:33Z

What changes were proposed in this pull request?

In the PR, I propose to add maxFields parameter to all functions involved in creation of textual representation of spark plans such as simpleString and verboseString. New parameter restricts number of fields converted to truncated strings. Any elements beyond the limit will be dropped and replaced by a "... N more fields" placeholder. The threshold is bumped up to Int.MaxValue for toFile().

How was this patch tested?

Added a test to QueryExecutionSuite which checks maxFields impacts on number of truncated fields in LocalRelation.

MaxGekk · 2018-11-27T22:52:29Z

ping @hvanhovell

SparkQA · 2018-11-28T02:45:24Z

Test build #99342 has finished for PR 23159 at commit e9e7893.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-11-28T20:52:16Z

@gatorsmile @cloud-fan Could you look at the changes - extracted from another PR: #22429

SparkQA · 2018-11-30T14:37:49Z

Test build #99513 has finished for PR 23159 at commit bf968be.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-11-30T19:48:18Z

Test build #99520 has finished for PR 23159 at commit a5e82f7.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-11-30T20:35:22Z

jenkins, retest this, please

SparkQA · 2018-12-01T00:23:15Z

Test build #99527 has finished for PR 23159 at commit a5e82f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-04T17:21:36Z

@HyukjinKwon @dongjoon-hyun @srowen @zsxwing Do you have any objections of this PR?

srowen · 2018-12-04T17:46:20Z

Rather than change every single call to this method, if this should generally be the value of the argument, then why not make it the default value or something?

SparkQA · 2018-12-04T21:20:43Z

Test build #99678 has finished for PR 23159 at commit b6fa959.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-04T22:38:17Z

Rather than change every single call to this method, if this should generally be the value of the argument, then why not make it the default value or something?

New parameter aims to solve the problem when there are multiple callers, and each of them needs different maximum fields. So, a feasible approach is to propagate maxFields from callers to truncatedString. Changing global SQL config does not solve the problem.

srowen

Ah OK so not every call would just pass the value SQLConf.get.maxToStringFields? It looked like it from the code here but I didn't examine the whole diff. If it were really always SQLConf.get.maxToStringFields and configured that way this could be simpler, but I suppose it isn't.

srowen · 2018-12-04T22:46:18Z

 case p if p.expressions.exists(hasGenerator) =>
 throw new AnalysisException("Generators are not supported outside the SELECT clause, but " +
- "got: " + p.simpleString)
+ "got: " + p.simpleString((SQLConf.get.maxToStringFields)))


Nit: are there extra parens here?

fixed here and other places

…ax-fields

SparkQA · 2018-12-05T15:14:49Z

Test build #99715 has finished for PR 23159 at commit e0aa626.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-05T15:23:25Z

jenkins, retest this, please

SparkQA · 2018-12-05T19:37:57Z

Test build #99735 has finished for PR 23159 at commit e0aa626.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2018-12-06T18:01:02Z

cc @cloud-fan and @gatorsmile .

HyukjinKwon · 2018-12-11T10:06:42Z

Looks okay but let me leave it to @cloud-fan, @gatorsmile and @hvanhovell

MaxGekk · 2018-12-11T10:44:44Z

@hvanhovell We all are waiting for your decision ;-). Please, review the PR.

SparkQA · 2018-12-21T03:30:11Z

Test build #100349 has finished for PR 23159 at commit 9acdf9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

MaxGekk · 2018-12-22T10:08:07Z

I am going to close the PR since nobody interested in it :-(

hvanhovell · 2018-12-24T13:14:06Z

@MaxGekk can you resolve the merge conflicts?

hvanhovell · 2018-12-24T13:14:15Z

LGTM

SparkQA · 2018-12-24T17:21:14Z

Test build #100425 has finished for PR 23159 at commit c6ad275.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class ShuffleMetrics implements MetricSet
sealed abstract class UserDefinedFunction

hvanhovell · 2018-12-27T10:12:37Z

merging to master.

…rameter ## What changes were proposed in this pull request? In the PR, I propose to add `maxFields` parameter to all functions involved in creation of textual representation of spark plans such as `simpleString` and `verboseString`. New parameter restricts number of fields converted to truncated strings. Any elements beyond the limit will be dropped and replaced by a `"... N more fields"` placeholder. The threshold is bumped up to `Int.MaxValue` for `toFile()`. ## How was this patch tested? Added a test to `QueryExecutionSuite` which checks `maxFields` impacts on number of truncated fields in `LocalRelation`. Closesapache#23159 from MaxGekk/to-file-max-fields. Lead-authored-by: Maxim Gekk <max.gekk@gmail.com> Co-authored-by: Maxim Gekk <maxim.gekk@databricks.com> Signed-off-by: Herman van Hovell <hvanhovell@databricks.com>

MaxGekk added 2 commits November 27, 2018 22:34

Added a test
7562421

Adding maxField parameter
e9e7893

MaxGekk mentioned this pull request Nov 27, 2018
[SPARK-25440][SQL] Dumping query execution info to a file #22429
Closed

Merge branch 'master' into to-file-max-fields
bf968be

Fix merge
a5e82f7

Merge branch 'master' into to-file-max-fields
b6fa959

srowen reviewed Dec 4, 2018
View reviewed changes

MaxGekk added 2 commits December 5, 2018 12:25

Removing unnecessary parentheses
b0571a2

Merge remote-tracking branch 'fork/to-file-max-fields' into to-file-m…
e0aa626
…ax-fields

Merge branch 'master' into to-file-max-fields
9acdf9d

Merge branch 'master' into to-file-max-fields
c6ad275

asfgit closed this in a1c1dd3Dec 27, 2018

MaxGekk deleted the to-file-max-fields branch August 17, 2019 13:35

Uh oh!

Conversation

MaxGekk commented Nov 27, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

MaxGekk commented Nov 27, 2018

Uh oh!

SparkQA commented Nov 28, 2018

Uh oh!

MaxGekk commented Nov 28, 2018

Uh oh!

SparkQA commented Nov 30, 2018

Uh oh!

SparkQA commented Nov 30, 2018

Uh oh!

MaxGekk commented Nov 30, 2018

Uh oh!

SparkQA commented Dec 1, 2018

Uh oh!

MaxGekk commented Dec 4, 2018

Uh oh!

srowen commented Dec 4, 2018

Uh oh!

SparkQA commented Dec 4, 2018

Uh oh!

MaxGekk commented Dec 4, 2018

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

srowenDec 4, 2018

Choose a reason for hiding this comment

Uh oh!

MaxGekkDec 5, 2018

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 5, 2018

Uh oh!

MaxGekk commented Dec 5, 2018

Uh oh!

SparkQA commented Dec 5, 2018

Uh oh!

dongjoon-hyun commented Dec 6, 2018

Uh oh!

HyukjinKwon commented Dec 11, 2018

Uh oh!

MaxGekk commented Dec 11, 2018

Uh oh!

SparkQA commented Dec 21, 2018

Uh oh!

MaxGekk commented Dec 22, 2018

Uh oh!

hvanhovell commented Dec 24, 2018

Uh oh!

hvanhovell commented Dec 24, 2018

Uh oh!

SparkQA commented Dec 24, 2018

Uh oh!

hvanhovell commented Dec 27, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MaxGekk commented Nov 27, 2018•
edited
Loading