Uh oh!
There was an error while loading. Please reload this page.
[SPARK-12063][SQL] Use number in group by clause to refer to columns - #10052
[SPARK-12063][SQL] Use number in group by clause to refer to columns#10052dereksabryfb wants to merge 7 commits into
Conversation
86d7e93 to
74c42aeComparemarmbrus
commented
Dec 1, 2015
Thanks for working on this. This seems resonable to support, but I have two suggestions:
|
rxin
commented
Dec 1, 2015
@dereksabryfb you should also add the email you used in your git commit to your github profile so it shows up on github. |
dereksabryfb
commented
Dec 3, 2015
Thanks for your feedback! The change is now implemented as a rule in the Analyzer and a unit test has been added. Let me know if there are any other changes I need to make. |
There was a problem hiding this comment.
I'm not sure we want to match on the pretty string as I think this would also trigger for things like "1", instead I'd consider matching on a Literal of NumericType.
marmbrus
commented
Dec 7, 2015
Are there other cases that we should handle here? Can you do the same thing in ORDER BY normally? |
dereksabryfb
commented
Dec 8, 2015
Thanks for the feedback! I'm making the changes you suggested. With respect to the 'ORDER BY' clause, this looks to be semi-handled by the ResolveSortReferences in the Analyzer; because in standard SQL literals are allowed in the order by clause, this rule just sorts by the literal value '1'; In HiveQL, '1' is interpreted as a column in the same way it is in the group by clause. I can add the case for a Sort() with an IntegerType. I assume there is no one currently relying on how 'sort by 1' currently functions. |
marmbrus
commented
Dec 8, 2015
We'll have to note the change in the release notes, but since its a no-op to sort by a constant I think we can safely change behavior here. |
dereksabryfb
commented
Dec 10, 2015
Added a case for sort |
marmbrus
commented
Dec 10, 2015
ok to test |
SparkQA
commented
Dec 10, 2015
Test build #47534 has finished for PR 10052 at commit
|
dereksabryfb
commented
Dec 10, 2015
Apologies, I haven't been able to run ./dev/run-tests, getting the following exception: http://pastebin.com/L0p0sjtJ so I wasn't able to pick up the style issues, and I'm not sure if there's more that the build doesn't flag. |
marmbrus
commented
Dec 10, 2015
I'd try the following locally Each of those commands can be run separately too and you can use ~ to rerun whenever something changes to iterate more quickly |
SparkQA
commented
Dec 10, 2015
Test build #47538 has finished for PR 10052 at commit
|
dereksabryfb
commented
Dec 11, 2015
Looks like it fails on the query "SELECT a, count(2) FROM testData2 GROUP BY a, 2" because 2 now refers to the column count(2) which is not valid syntax for a group by clause in hive. How should I go about resolving this? The purpose of the test is to see that literals in the group by clause don't modify the results, but the purpose of this patch is to do the opposite of that. I could modify the offending case, but I feel the whole test may be irrelevant with this patch. |
marmbrus
commented
Dec 11, 2015
We should probably throw an Regarding the test, we can probably remove it. |
dereksabryfb
commented
Dec 11, 2015
I removed the offending test. I found that there was no Thanks again for your feedback. |
SparkQA
commented
Dec 11, 2015
Test build #47566 has finished for PR 10052 at commit
|
SparkQA
commented
Dec 11, 2015
Test build #47565 has finished for PR 10052 at commit
|
There was a problem hiding this comment.
case Literal(index: Int) => is easier. It also eliminates the need for group.toString.toInt
hvanhovell
commented
Jan 18, 2016
@dereksabryfb are you still working on this? |
srowen
commented
May 6, 2016
Ping @dereksabryfb -- are you working on this? or else close it |
hvanhovell
commented
May 6, 2016
I think this one can be closed, it has been implemented in #11846 |
JoshRosen
commented
Sep 7, 2016
Yep, it looks like this PR has been subsumed by #11846. @dereksabryfb, could you please close this pull request? Thanks! |
If there is a number n in a group by clause, the nth column in the select clause is used to modify the group by clause to refer to this column instead of the number.
eg.
select a,b from c group by 1,2
becomes
select a,b from c group by a,b