[Spark 3922] Refactor spark-core to use Utils.UTF_8

[zsxwing]

A global UTF8 constant is very helpful to handle encoding problems when converting between String and bytes. There are several solutions here:

1. Add val UTF_8 = Charset.forName("UTF-8") to Utils.scala
2. java.nio.charset.StandardCharsets.UTF_8 (require JDK7)
3. io.netty.util.CharsetUtil.UTF_8
4. com.google.common.base.Charsets.UTF_8
5. org.apache.commons.lang.CharEncoding.UTF_8
6. org.apache.commons.lang3.CharEncoding.UTF_8

IMO, I prefer option 1) because people can find it easily.

This is a PR for option 1) and only fixes Spark Core.

[zsxwing]

/cc @rxin, @JoshRosen

[AmplabJenkins]

Can one of the admins verify this patch?

[SparkQA]

QA tests have started for PR 2781 at commit 80f4af8.

• This patch merges cleanly.

[srowen]

I vote for com.google.common.base.Charsets.UTF_8 now, and java.nio.charset.StandardCharsets.UTF_8 when Spark moves to Java 7+. No need to define this constant yet again.

[SparkQA]

QA tests have finished for PR 2781 at commit 80f4af8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21681/
Test FAILed.

[vanzin]

Like @srowen, I prefer to use existing constants.

Also, it might be worth it to import the constant directly and just use UTF_8 (instead of something like Utils.UTF_8) so that switching to the JDK7 version later is less noisy.

(Edit: just fixed typo.)

[SparkQA]

QA tests have started for PR 2781 at commit 80f4af8.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2781 at commit 80f4af8.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2781 at commit 80f4af8.

• This patch merges cleanly.

[zsxwing]

Also, it might be worth it to import the constant directory and just use UTF_8 (instead of something like Utils.UTF_8) so that switching to the JDK7 version later is less noisy.

Good point. I updated to use com.google.common.base.Charsets.UTF_8 directly in the spark core. I override my previous commits because they were redundant.

[SparkQA]

QA tests have finished for PR 2781 at commit 80f4af8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 2781 at commit 2d27423.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2781 at commit 2d27423.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zsxwing]

retest this please

[rxin]

Jenkins, retest this please.

[SparkQA]

Test build #22344 has started for PR 2781 at commit f974edd.

• This patch merges cleanly.

[SparkQA]

Test build #22344 has finished for PR 2781 at commit f974edd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22344/
Test PASSed.

[rxin]

LGTM. @JoshRosen any further comment?

[JoshRosen]

This looks good to me. Thanks!

[rxin]

Merging in master. Thanks.