[SPARK-12629] [SparkR] Fixes for DataFrame saveAsTable method by NarineK · Pull Request #10580 · apache/spark

NarineK · 2016-01-04T21:49:26Z

I've tried to solve some of the issues mentioned in: https://issues.apache.org/jira/browse/SPARK-12629
Please, let me know what do you think.
Thanks!

SparkQA · 2016-01-04T22:21:15Z

Test build #48692 has finished for PR 10580 at commit 62872f1.

This patch fails R style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-01-05T00:04:12Z

Test build #48697 has finished for PR 10580 at commit d55cc13.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2016-01-05T00:57:51Z

I think the direction we are heading is to have multiple sqlContext sessions and be able to specify which one to use, instead of automatically picking one?

sun-rui · 2016-01-05T02:26:17Z

I think the same problem exists in write.df()? Could you extract the following logic into a util function, and update write.df() also to use it?

See #9192 :)

@felixcheung , @sun-rui , @shivaram : that's a good idea, I can use "getSqlContext()" method proposed in https://github.com/apache/spark/pull/9192/files#diff-b11442485f6b77bf47b58b4747321638R36
I guess I need to wait that pull request to go in and then I can merge it with mine .

@felixcheung, it seems that you have it for saveAsTable too. After merge I can add my test cases ...

sun-rui · 2016-01-05T02:30:16Z

@felixcheung, as we discussed before, I don't think it makes sense to support multiple sessions in SparkR now. Let's keep the current way until there is need to support it.

sun-rui · 2016-01-05T02:34:31Z

Please use df.write.saveAsTable() instead of deprecated df.saveAsTable().

this is done in #10584.

Yes, let's merge with PR #10584 to unblock SparkR

…avaAsRable

shivaram · 2016-01-06T06:33:34Z

@NarineK FYI #10584 has been merged. If you rebase / merge with master you should be able to see the changes.

…avaAsRable

NarineK · 2016-01-06T19:09:07Z

Thanks for letting us know @shivaram
How about the #9192 ?
I was thinking to reuse getSqlContext() utility method

sun-rui · 2016-01-07T02:19:53Z

I think we may need more discussion on #9192, so let's merge this first.

SparkQA · 2016-01-07T17:48:53Z

Test build #48951 has finished for PR 10580 at commit 7de1894.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-07T18:22:53Z

Test build #48952 has finished for PR 10580 at commit faa4971.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

NarineK · 2016-01-07T18:41:37Z

I see. I've pushed all my changes and left the piece of code which retrieves the context in case source is not passed by the argument. Later, you can replace it with the getContext utility method.

shivaram · 2016-01-19T22:44:19Z

LGTM. Thanks @NarineK -- @sun-rui Do you have any more comments on this ?

felixcheung · 2016-01-20T00:25:58Z

looks fine to me

sun-rui · 2016-01-20T05:18:52Z

@NarineK, could you update the write.df() function as well?

NarineK · 2016-01-21T01:38:25Z

updated write.df and saveDF

SparkQA · 2016-01-21T01:42:01Z

Test build #49840 has finished for PR 10580 at commit fdd38c5.

This patch fails R style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2016-01-21T20:12:29Z

Test build #49884 has finished for PR 10580 at commit 3c24314.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

NarineK · 2016-01-21T21:22:16Z

added ! Let me know what do you think!

sun-rui · 2016-01-22T01:38:27Z

LGTM

shivaram · 2016-01-22T18:33:15Z

Merging this to master.

shivaram · 2016-02-02T18:52:23Z

@sun-rui @felixcheung I'm going to backport this to branch-1.6 so that the bug fix is available in 1.6.1. Let me know if you think there is a problem with it

I've tried to solve some of the issues mentioned in: https://issues.apache.org/jira/browse/SPARK-12629 Please, let me know what do you think. Thanks! Author: Narine Kokhlikyan <narine.kokhlikyan@gmail.com> Closes#10580 from NarineK/sparkrSavaAsRable. (cherry picked from commit 8a88e12) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>

Some fixes for saveAsTable method
62872f1

Fix merge conflicts
d55cc13

sun-rui reviewed Jan 5, 2016
View reviewed changes

Merge branch 'master' ofhttps://github.com/apache/sparkinto sparkrS…
8948f98
…avaAsRable

NarineK added 2 commits January 6, 2016 10:12

change the signature
86c1432

Merge branch 'master' ofhttps://github.com/apache/sparkinto sparkrS…
7de1894
…avaAsRable

fixed style check errors
faa4971

Added the changes to write.df and saveDF too
fdd38c5

NarineK added 2 commits January 21, 2016 11:30

fixing merge conflict
8abf4c1

fixed stylecheck error
3c24314

asfgit closed this in 8a88e12Jan 22, 2016

Uh oh!

Conversation

NarineK commented Jan 4, 2016

Uh oh!

SparkQA commented Jan 4, 2016

Uh oh!

SparkQA commented Jan 5, 2016

Uh oh!

felixcheung commented Jan 5, 2016

Uh oh!

sun-ruiJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

felixcheungJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

NarineKJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

NarineKJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

sun-rui commented Jan 5, 2016

Uh oh!

sun-ruiJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

sun-ruiJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

felixcheungJan 5, 2016

Choose a reason for hiding this comment

Uh oh!

shivaram commented Jan 6, 2016

Uh oh!

NarineK commented Jan 6, 2016

Uh oh!

sun-rui commented Jan 7, 2016

Uh oh!

SparkQA commented Jan 7, 2016

Uh oh!

SparkQA commented Jan 7, 2016

Uh oh!

NarineK commented Jan 7, 2016

Uh oh!

shivaram commented Jan 19, 2016

Uh oh!

felixcheung commented Jan 20, 2016

Uh oh!

sun-rui commented Jan 20, 2016

Uh oh!

NarineK commented Jan 21, 2016

Uh oh!

SparkQA commented Jan 21, 2016

Uh oh!

SparkQA commented Jan 21, 2016

Uh oh!

NarineK commented Jan 21, 2016

Uh oh!

sun-rui commented Jan 22, 2016

Uh oh!

shivaram commented Jan 22, 2016

Uh oh!

shivaram commented Feb 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants