[SPARK-29924][DOCS] Document Apache Arrow JDK11 requirement

[dongjoon-hyun]

What changes were proposed in this pull request?

This adds a note for additional setting for Apache Arrow library for Java 11.

Why are the changes needed?

Since Apache Arrow 0.14.0, an additional setting is required for Java 9+.

• https://issues.apache.org/jira/browse/ARROW-3191

It's explicitly documented at Apache Arrow 0.15.0.

• https://issues.apache.org/jira/browse/ARROW-6206

However, there is no plan to handle that inside Apache Arrow side.

• https://issues.apache.org/jira/browse/ARROW-7223

In short, we need to document this for the users who is using Arrow-related feature on JDK11.

For dev environment, we handle this via SPARK-29923 .

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Generated document and see the pages.

[dongjoon-hyun]

cc @HyukjinKwon , @BryanCutler , @srowen .

[SparkQA]

Test build #117369 has finished for PR 27356 at commit ed7f624.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Thank you, @srowen . Merged to master.

[BryanCutler]

LGTM, thanks @dongjoon-hyun !

[dongjoon-hyun]

Thank you, @BryanCutler !

[HyukjinKwon]

LGTM

[dongjoon-hyun]

Thanks, @HyukjinKwon . It's a holiday in South Korea, isn't it? 😉

[chogg]

Will the io.netty.tryReflectionSetAccessible=true be set automatically at any point in the future, or will users always need to set this manually? Is there a downside to always setting this to true?

Thanks for your help here :)

[dongjoon-hyun]

@chogg . Unfortunately, Apache Spark cannot add it by default with the same reason why Netty and Apache Arrow project didn't add it automatically.

Could you ask that to the Netty or Apache Arrow community instead?

[romainx]

Hello.

For information we observed that this setting seems to not be required anymore with new versions of the libraries. See jupyter/docker-stacks#1256 for detail.

Best.

[HyukjinKwon]

Thanks for heads up @romainx. Would you be interested in fixing it? See also http://spark.apache.org/contributing.html

[romainx]

@HyukjinKwon sure I will have a look and I will try to draft a PR, it could take some time since it's my first contribution here. Thanks for the proposal.

[HyukjinKwon]

Sure, thanks.