[SPARK-17702][SQL] Code generation including too many mutable states exceeds JVM size limit.

[ueshin]

What changes were proposed in this pull request?

Code generation including too many mutable states exceeds JVM size limit to extract values from references into fields in the constructor.
We should split the generated extractions in the constructor into smaller functions.

How was this patch tested?

I added some tests to check if the generated codes for the expressions exceed or not.

[SparkQA]

Test build #66020 has finished for PR 15275 at commit 858a3ec.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66025 has finished for PR 15275 at commit 80b9435.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Can you add more inline comments explaining what's going on?

[SparkQA]

Test build #66090 has finished for PR 15275 at commit 3c4b765.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

LGTM. cc @davies to take a look too.

[kiszk]

How about exposing this API as public? This API looks more flexible and reusable.Is there any reason to declare as private?
(I replaced "non-public" with "public" in the first statement).

[ueshin]

I didn't have a special reason.
Should we make it public? @rxin, @davies

[kiszk]

As far as I know, there are two possible use cases of this API
https://issues.apache.org/jira/browse/SPARK-16223
https://github.com/apache/spark/pull/15219/files#diff-8bcc5aea39c73d4bf38aef6f6951d42cL587 in #15219

[SparkQA]

Test build #66267 has finished for PR 15275 at commit 3c4b765.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Merging in master. Thanks.

[viirya]

This affects many places in code generation. I think this should be a common issue for the users of previous versions. Should we make a backport to branch-2.0 and branch-1.6? cc @rxin@ueshin@kiszk

[viirya]

One important reason to prepare a backport for this is, as we deprecate the settings to turn off unsafe and codegen since 1.6, we can't turn them off to avoid JVM size limitation issue if users encounter that.

[ueshin]

It seems we can cherry-pick into branch-2.0 without conflicts, but can't into branch-1.6.
Please let me know if we need to backport to branch-1.6.

[rezasafi]

Sorry to bother, it seems that the backport to 1.6 is not clean and easy. Any update on the decision of whether this fix will be tried to be backported to 1.6? thanks.

[xs2rajni]

When will this fix be backported to spark 2.0?