[SPARK-8962] Add Scalastyle rule to ban direct use of Class.forName; fix existing uses

[JoshRosen]

This pull request adds a Scalastyle regex rule which fails the style check if Class.forName is used directly. Class.forName always loads classes from the default / system classloader, but in a majority of cases, we should be using Spark's own Utils.classForName instead, which tries to load classes from the current thread's context classloader and falls back to the classloader which loaded Spark when the context classloader is not defined.

[JoshRosen]

/cc @yhuai@tdas

[JoshRosen]

This pull request might become intractable to review due to how many files it touches, so I recommend using the Reviewable link to keep track of changes.

[SparkQA]

Test build #37065 has finished for PR 7350 at commit 62882ee.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shivaram]

The RBackendHandler change looks good to me. I do find it weird that we have scala style rules for not using specific functions, but I guess its hard to track this during code review in a large codebase.

[JoshRosen]

@shivaram, in this case I think the style rule is justified even if it's inconvenient or a hassle to implement because this Class.forName issue has led to a number of separate bugs over the years.

[rxin]

@JoshRosen looks like you missed a few places. Can you fix them so we can merge this?

[SparkQA]

Test build #1053 has finished for PR 7350 at commit 62882ee.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #37158 has finished for PR 7350 at commit d707ba7.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #37159 has finished for PR 7350 at commit c0b7885.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Unfortunately another pr created a conflict again ...

[JoshRosen]

Argh... will fix now.

[SparkQA]

Test build #37190 has finished for PR 7350 at commit e3e96f7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #37247 has finished for PR 7350 at commit e3e96f7.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #37260 has finished for PR 7350 at commit e3e96f7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

Thanks - merging this in.