Uh oh!
There was an error while loading. Please reload this page.
[SPARK-22603][SQL] Fix 64KB JVM bytecode limit problem with FormatString - #19817
[SPARK-22603][SQL] Fix 64KB JVM bytecode limit problem with FormatString#19817kiszk wants to merge 2 commits into
Conversation
| val argList = ctx.freshName("argLists") | ||
| val numArgLists = argListGen.length | ||
| val argListCode = argListGen.zipWithIndex.map { case(v, index) => | ||
| v._2.code + s"\n$argList[$index] = " + |
There was a problem hiding this comment.
would it be possible to use the syntax:
s"""
...
"""
as it is done in many other places? I think it would be more readable. What do you think?
SparkQA
commented
Nov 24, 2017
Test build #84173 has finished for PR 19817 at commit
|
SparkQA
commented
Nov 25, 2017
Test build #84180 has finished for PR 19817 at commit
|
mgaido91
commented
Nov 25, 2017
LGTM |
| arguments = ("InternalRow", ctx.INPUT_ROW) :: ("Object[]", argList) :: Nil) | ||
| } else { | ||
| argListCode.mkString("\n") | ||
| } |
There was a problem hiding this comment.
Could you create a splitExpressions in CodegenContext for avoiding the duplicate codes, like
if (ctx.INPUT_ROW!=null&& ctx.currentVars ==null) {
ctx.splitExpressions(...)
} else {
inputs.mkString("\n")
}There was a problem hiding this comment.
Sure, is it better to do this in this PR? Or, should I create another PR?
There was a problem hiding this comment.
I wanted to submit a PR for that, but I was waiting for all the PRs related (this one should be the last one) to be merged. Let me know if I can do that or you are doing it. My suggestion would be: can't we just change the existing method to include this check?
There was a problem hiding this comment.
cloud-fan
commented
Nov 27, 2017
LGTM, merging to master/2.2! |
## What changes were proposed in this pull request? This PR changes `FormatString` code generation to place generated code for expressions for arguments into separated methods if these size could be large. This PR passes variable arguments by using an `Object` array. ## How was this patch tested? Added new test cases into `StringExpressionSuite` Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes#19817 from kiszk/SPARK-22603. (cherry picked from commit 2dbe275) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
## What changes were proposed in this pull request? This PR changes `FormatString` code generation to place generated code for expressions for arguments into separated methods if these size could be large. This PR passes variable arguments by using an `Object` array. ## How was this patch tested? Added new test cases into `StringExpressionSuite` Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closesapache#19817 from kiszk/SPARK-22603. (cherry picked from commit 2dbe275) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
This PR changes
FormatStringcode generation to place generated code for expressions for arguments into separated methods if these size could be large.This PR passes variable arguments by using an
Objectarray.How was this patch tested?
Added new test cases into
StringExpressionSuite