Uh oh!
There was an error while loading. Please reload this page.
[SPARK-28286][SQL][PYTHON][TESTS] Convert and port 'pivot.sql' into UDF test base - #25122
[SPARK-28286][SQL][PYTHON][TESTS] Convert and port 'pivot.sql' into UDF test base#25122chitralverma wants to merge 10 commits into
Conversation
@HyukjinKwon I've raised this PR as a WIP till I incorporate your comments. I had some doubts regarding the tests in pivot.sql and was hoping you could clear it for me. While porting 'pivot.sql', I ran the command below on the original sql and it fails when running for configs
On inspection it seems like there is some discrepancy while handling the This error persists in the port also. As per the guide, I tried looking for a related Jira but couldn't find one, so I thought I'd run this by you first before creating one. Stacktrace: Any help will be appreciated. Thanks, |
HyukjinKwon
commented
Jul 12, 2019
ok to test |
HyukjinKwon
commented
Jul 12, 2019
add to whitelist |
HyukjinKwon
commented
Jul 12, 2019
The |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
HyukjinKwon
commented
Jul 12, 2019
SparkQA
commented
Jul 12, 2019
Test build #107571 has finished for PR 25122 at commit
|
@HyukjinKwon Thanks for your review comments, I've replied to them. I feel the suggested workaround(this and this) will not work in this case as certain udf's like As per the guide in SPARK-27921 , do you suggest I create a separate Jira for this and comment the problematic test cases for now ? |
Uh oh!
There was an error while loading. Please reload this page.
SparkQA
commented
Jul 12, 2019
Test build #107592 has finished for PR 25122 at commit
|
HyukjinKwon
commented
Jul 18, 2019
@chitralverma, #25130 is merged. Can you rebase and sync against the current master? |
HyukjinKwon
commented
Jul 18, 2019
retest this please |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
HyukjinKwon
commented
Jul 18, 2019
Looks fine otherwise if the tests pass. I will take another look before merging it in. |
SparkQA
commented
Jul 18, 2019
Test build #107813 has finished for PR 25122 at commit
|
chitralverma
commented
Jul 18, 2019
@HyukjinKwon I'm merging the changes and testing. will ping you once its done. thanks |
chitralverma
commented
Jul 18, 2019
retest this please |
SparkQA
commented
Jul 18, 2019
Test build #107838 has finished for PR 25122 at commit
|
@HyukjinKwon you can review this now. thanks. I've also updated the diff in the OP |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| ) | ||
| PIVOT ( | ||
| udf(sum(earnings)) | ||
| FOR course IN ('dotNET', 'Java') |
There was a problem hiding this comment.
I can test it by myself but I thought you know the results - just out of curiosity, what do we get if we do FOR udf(course)?
There was a problem hiding this comment.
FOR udf(course) in line 29 will result in a org.apache.spark.sql.catalyst.parser.ParseException
SparkQA
commented
Jul 18, 2019
Test build #107839 has finished for PR 25122 at commit
|
HyukjinKwon
commented
Jul 18, 2019
Merged to master. Thanks for working on this, @chitralverma |
chitralverma
commented
Jul 18, 2019
Thanks for merging this @HyukjinKwon. This was my first contribution, looking forward to doing more. :D |
SparkQA
commented
Jul 18, 2019
Test build #107849 has finished for PR 25122 at commit
|
SparkQA
commented
Jul 18, 2019
Test build #107851 has finished for PR 25122 at commit
|
HyukjinKwon
commented
Jul 19, 2019
Thank YOU @chitralverma for staying focused on each diff here and writing a PR even without nits :D. Nowdays, those details are a key. |
What changes were proposed in this pull request?
This PR adds some tests converted from pivot.sql to test UDFs following the combination guide in SPARK-27921.
Diff comparing to 'pivot.sql'
How was this patch tested?
Tested as guided in SPARK-27921.