[SPARK-14819] [SQL] Improve SET / SET -v command by bomeng · Pull Request #12583 · apache/spark

bomeng · 2016-04-21T20:21:20Z

What changes were proposed in this pull request?

Currently SET and SET -v commands are similar to Hive SET command except the following difference:

The result is not sorted;
When using SET and SET -v, in addition to the Hive related properties, it will also list all the system properties and environment properties, which is very useful in some cases.

This JIRA is trying to make the current SET command more consistent to Hive output.

How was this patch tested?

Current test cases SQLQuerySuite and HiveQuerySuite should have enough test coverage.

SparkQA · 2016-04-21T20:23:53Z

Test build #56582 has finished for PR 12583 at commit 01d3e5a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-21T21:28:20Z

You might need to rebase this in a bit because i'm refactoring all of these quite heavily for Spark 2.0.

rxin · 2016-04-21T21:28:42Z

 }

+ test("set / set -v") {
+ checkExistence(sql("set"), true, "env:", "system:")


can we add a test case that checks the value of SPARK_TESTING?

SPARK_TESTING is not in sys.env nor sys.props while running the test cases. Any suggestions to add the test cases for that?

Found out that current test cases SQLQuerySuite and HiveQuerySuite have intensive test coverage for SET command.

SparkQA · 2016-04-21T21:30:49Z

Test build #56585 has finished for PR 12583 at commit 790cfda.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T00:32:47Z

Test build #56603 has finished for PR 12583 at commit 2a077bd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T03:57:36Z

Test build #56636 has finished for PR 12583 at commit e3ba716.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T05:09:12Z

Test build #56651 has finished for PR 12583 at commit c6805e4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-04-22T06:51:03Z

Test build #56653 has finished for PR 12583 at commit 99ff8a5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-04-22T06:56:55Z

@bomeng while you are at this, do you mind implementing https://issues.apache.org/jira/browse/SPARK-14806 together? Should be pretty trivial. Thanks.

SparkQA · 2016-04-22T10:54:34Z

Test build #56688 has finished for PR 12583 at commit 6c7276b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-01-05T00:13:13Z

Hi, @bomeng .
I hit the same issue. Could you update the PR?

viirya · 2017-01-05T02:36:43Z

 (keyValueOutput, runFunc)

 // Queries all properties along with their default values and docs that are defined in the
 // SQLConf of the sqlContext.


We should update the comment regarding the system environment properties.

viirya · 2017-01-05T02:36:53Z

@@ -138,17 +138,18 @@ case class SetCommand(kv: Option[(String, Option[String])]) extends RunnableComm
 // Queries all key-value pairs that are set in the SQLConf of the sqlContext.


viirya · 2017-01-05T02:37:51Z

 override def run(sqlContext: SQLContext): Seq[Row] = runFunc(sqlContext)

+ /**
+ * get the system environment properties as a sequence


Nit: "get" -> "Gets", "as a sequence" -> "as a sequence of rows".

dongjoon-hyun · 2017-01-13T18:54:48Z

Hmm. This PR seems to be stale. Actually, what I wanted was a sorted result of SET and SET -v.

dongjoon-hyun · 2017-01-13T19:08:34Z

I made another #16579 for SET for a sorted result.

HyukjinKwon · 2017-02-03T12:44:59Z

@dongjoon-hyun, then, do you mind if I ask if this PR is not useful in favour of #16579?

dongjoon-hyun · 2017-02-03T17:02:52Z

Hi, @bomeng , do you have something to refresh here?
I think we can close this for now.

improve SET / SET -v command
01d3e5a

remove unused imports
790cfda

rxin reviewed Apr 21, 2016
View reviewed changes

bomeng added 3 commits April 21, 2016 14:53

Merge remote-tracking branch 'upstream/master' intoSPARK-14819
a0de01e

Merge remote-tracking branch 'upstream/master' intoSPARK-14819
392327a

improve the codes and fix test failure
2a077bd

fix the test failure
e3ba716

bomeng added 3 commits April 21, 2016 22:05

remove unnecessary test case
84243a0

roll back
c6805e4

roll back
99ff8a5

merge upstream
6c7276b

viirya reviewed Jan 5, 2017
View reviewed changes

HyukjinKwon mentioned this pull request Feb 15, 2017
[BUILD] Close stale PRs #16937
Closed

asfgit closed this in ed338f7Feb 17, 2017

		@@ -138,17 +138,18 @@ case class SetCommand(kv: Option[(String, Option[String])]) extends RunnableComm
		// Queries all key-value pairs that are set in the SQLConf of the sqlContext.

Uh oh!

Conversation

bomeng commented Apr 21, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Apr 21, 2016

Uh oh!

rxin commented Apr 21, 2016

Uh oh!

rxinApr 21, 2016

Choose a reason for hiding this comment

Uh oh!

bomengApr 21, 2016

Choose a reason for hiding this comment

Uh oh!

bomengApr 22, 2016

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Apr 21, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

rxin commented Apr 22, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Apr 22, 2016

Uh oh!

dongjoon-hyun commented Jan 5, 2017

Uh oh!

viiryaJan 5, 2017

Choose a reason for hiding this comment

Uh oh!

viiryaJan 5, 2017

Choose a reason for hiding this comment

Uh oh!

viiryaJan 5, 2017

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 13, 2017

Uh oh!

dongjoon-hyun commented Jan 13, 2017

Uh oh!

HyukjinKwon commented Feb 3, 2017

Uh oh!

dongjoon-hyun commented Feb 3, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bomeng commented Apr 21, 2016•
edited
Loading

rxin commented Apr 22, 2016•
edited
Loading