[SPARK-13121] [Streaming] java mapWithState mishandles scala Option

[sparkyengine]

java mapwithstate with Function3 has wrong conversion of java Optional to scala Option, fixed code uses same conversion used in the mapwithstate call that uses Function4 as an input. Optional.fromNullable(v.get) fails if v is None, better to use JavaUtils.optionToOptional(v) instead.

[holdenk]

Hi @gabrielenizzoli - generally Spark require's that there is a JIRA associated the proposed change as well as having it mentioned in the pull request title. This is also just merging against the 1.6 branch but generally most changes are applied to master and then explicitly back ported (looking in master it seems to have changed as part of the migration away from having guava in the public API to Optional.ofNullable). The guide at https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark can be quite useful for structuring your pull request.

[zsxwing]

Jenkins, test this please

[zsxwing]

@gabrielenizzoli Thanks, good catch. Please follow the instructions in https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark

[SparkQA]

Test build #50501 has finished for PR 11007 at commit 78c0323.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

retest this please

[SparkQA]

Test build #50516 has finished for PR 11007 at commit 78c0323.

• This patch fails from timeout after a configured wait of 250m.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

retest this please

[SparkQA]

Test build #50544 has finished for PR 11007 at commit 78c0323.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

This sounds correct to me.

[zsxwing]

retest this please

[SparkQA]

Test build #50571 has finished for PR 11007 at commit 78c0323.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

LGTM. Merging to master and 1.6. Thanks @gabrielenizzoli

[zsxwing]

@gabrielenizzoli I have merged this PR to branch 1.6. Could you open a new PR to fix the master branch and close this one? Thanks!

[sparkyengine]

@zsxwing yes, I will close this PR and create a new one for master. Should I assign the same jira issue number to the PR for master?

[zsxwing]

@zsxwing yes, I will close this PR and create a new one for master. Should I assign the same jira issue number to the PR for master?

Just make the new PR has the same title of this one.