Uh oh!
There was an error while loading. Please reload this page.
[SPARK-18091] [SQL] [BACKPORT-1.6] Deep if expressions cause Generated SpecificUnsafeProjection code to exceed JVM code size limit - #16146
Closed
kapilsingh5050 wants to merge 4 commits into
Conversation
…gh to exceed JVM method code size limit
cloud-fan
commented
Dec 6, 2016
Contributor
ok to test |
SparkQA
commented
Dec 6, 2016
Test build #69707 has finished for PR 16146 at commit
|
cloud-fan
commented
Dec 12, 2016
Contributor
hi @kapilsingh5050 can you also include #16244? It does fix the maven tests. |
kapilsingh5050
commented
Dec 12, 2016
Author
Yes, I'll do that but the test failures here are different. I'm still to figure out the root cause. |
…with nested wide schema The wide schema, the expression of fields will be splitted into multiple functions, but the variable for loopVar can't be accessed in splitted functions, this PR change them as class member. Added regression test. Author: Davies Liu <davies@databricks.com> Closesapache#12338 from davies/nested_row. Conflicts: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
## What changes were proposed in this pull request? After apache#15620 , all of the Maven-based 2.0 Jenkins jobs time out consistently. As I pointed out in apache#15620 (comment) , it seems that the regression test is an overkill and may hit constants pool size limitation, which is a known issue and hasn't been fixed yet. Since apache#15620 only fix the code size limitation problem, we can simplify the test to avoid hitting constants pool size limitation. ## How was this patch tested? test only change Author: Wenchen Fan <wenchen@databricks.com> Closesapache#16244 from cloud-fan/minor.
kapilsingh5050
commented
Dec 12, 2016
Author
ExpressionEncoderSuite and RowEncoderSuite tests were failing because of following fix missing in branch-1.6: |
SparkQA
commented
Dec 12, 2016
Test build #70015 has finished for PR 16146 at commit
|
kapilsingh5050force-pushed
the
SPARK-18091-IfCodegenFix-BACKPORT-1.6
branch
from
December 15, 2016 09:56
73f1231 to
708c847CompareSparkQA
commented
Dec 15, 2016
Test build #70185 has finished for PR 16146 at commit
|
Author
@cloud-fan The last test failure is because of following error: |
cloud-fan
commented
Dec 15, 2016
Contributor
retest this please |
SparkQA
commented
Dec 15, 2016
Test build #70189 has finished for PR 16146 at commit
|
kapilsingh5050
commented
Dec 16, 2016
Author
@cloud-fan please review and merge |
rezasafi
commented
Feb 7, 2017
Contributor
Hi, will this pull request merge to branch-1.6? |
zzcclp
commented
Feb 15, 2017
Contributor
ping @cloud-fan , will this pr be merged into branch-1.6? |
maropu added a commit
to maropu/spark
that referenced
this pull request
Apr 23, 2017
Closesapache#11785Closesapache#13027Closesapache#13614Closesapache#13761Closesapache#15197Closesapache#14006Closesapache#12576Closesapache#15447Closesapache#13259Closesapache#15616Closesapache#14473Closesapache#16638Closesapache#16146Closesapache#17269Closesapache#17313Closesapache#17418Closesapache#17485Closesapache#17551Closesapache#17463Closesapache#17625Closesapache#10739Closesapache#15193Closesapache#15344Closesapache#14804Closesapache#16993Closesapache#17040Closesapache#15180Closesapache#17238
peter-toth pushed a commit
to peter-toth/spark
that referenced
this pull request
Oct 6, 2018
This pr proposed to close stale PRs. Currently, we have 400+ open PRs and there are some stale PRs whose JIRA tickets have been already closed and whose JIRA tickets does not exist (also, they seem not to be minor issues). // Open PRs whose JIRA tickets have been already closed Closesapache#11785Closesapache#13027Closesapache#13614Closesapache#13761Closesapache#15197Closesapache#14006Closesapache#12576Closesapache#15447Closesapache#13259Closesapache#15616Closesapache#14473Closesapache#16638Closesapache#16146Closesapache#17269Closesapache#17313Closesapache#17418Closesapache#17485Closesapache#17551Closesapache#17463Closesapache#17625 // Open PRs whose JIRA tickets does not exist and they are not minor issues Closesapache#10739Closesapache#15193Closesapache#15344Closesapache#14804Closesapache#16993Closesapache#17040Closesapache#15180Closesapache#17238 N/A Author: Takeshi Yamamuro <yamamuro@apache.org> Closesapache#17734 from maropu/resolved_pr. Change-Id: Id2e590aa7283fe5ac01424d30a40df06da6098b5
zifeif2 pushed a commit
to zifeif2/spark
that referenced
this pull request
Nov 22, 2025
## What changes were proposed in this pull request? This pr proposed to close stale PRs. Currently, we have 400+ open PRs and there are some stale PRs whose JIRA tickets have been already closed and whose JIRA tickets does not exist (also, they seem not to be minor issues). // Open PRs whose JIRA tickets have been already closed Closesapache#11785Closesapache#13027Closesapache#13614Closesapache#13761Closesapache#15197Closesapache#14006Closesapache#12576Closesapache#15447Closesapache#13259Closesapache#15616Closesapache#14473Closesapache#16638Closesapache#16146Closesapache#17269Closesapache#17313Closesapache#17418Closesapache#17485Closesapache#17551Closesapache#17463Closesapache#17625 // Open PRs whose JIRA tickets does not exist and they are not minor issues Closesapache#10739Closesapache#15193Closesapache#15344Closesapache#14804Closesapache#16993Closesapache#17040Closesapache#15180Closesapache#17238 ## How was this patch tested? N/A Author: Takeshi Yamamuro <yamamuro@apache.org> Closesapache#17734 from maropu/resolved_pr.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix for SPARK-18091 which is a bug related to large if expressions causing generated SpecificUnsafeProjection code to exceed JVM code size limit.
This PR changes if expression's code generation to place its predicate, true value and false value expressions' generated code in separate methods in context so as to never generate too long combined code.
How was this patch tested?
Added a unit test and also tested manually with the application (having transformations similar to the unit test) which caused the issue to be identified in the first place.