[SPARK-14679] [UI] Fix UI DAG visualization OOM.

[rdblue]

What changes were proposed in this pull request?

The DAG visualization can cause an OOM when generating the DOT file.
This happens because clusters are not correctly deduped by a contains
check because they use the default equals implementation. This adds a
working equals implementation.

How was this patch tested?

This adds a test suite that checks the new equals implementation.

[SparkQA]

Test build #55986 has finished for PR 12437 at commit e0bda13.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Do we need canEqual since that is already known to be a RDDOperationCluster?

[rdblue]

This check ensures that equality is symmetric if subclasses have a different definition of equals. As I understand scala best practices, it should be there if the class can be extended. Right now this isn't extended, but I thought it may be in the future so I included it. I'm fine either way.

[srowen]

OK this is another way or saying "this.getClass == that.getClass", essentially? I think it is OK.

[rdblue]

It's similar, but it's calling a method on the other object to see if it would reject equality. There's a good explanation of canEqual on StackOverflow.

[joan38]

The only benefit you get with canEqual over this.getClass == that.getClass is that it's a little bit less strict.
See https://www.artima.com/lejava/articles/equality.html (look for "getClass" in the text to avoid reading the whole thing)

[SaintBacchus]

@rdblue Can this PR fix the case like this:

2016-02-2415:40:20,260 | ERROR | [qtp1927776715-4120] | Failedtomakedotfileofstage619 | org.apache.spark.Logging$class.logError(Logging.scala:96)
java.lang.OutOfMemoryError: RequestedarraysizeexceedsVMlimitatjava.util.Arrays.copyOf(Arrays.java:3332)
atjava.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)

[SparkQA]

Test build #56123 has finished for PR 12437 at commit 9764b43.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rdblue]

@SaintBacchus, I think that error message is essentially the same as that we were seeing, it's just hitting a different limitation when expanding:

[SparkQA]

Test build #56125 has finished for PR 12437 at commit f05edc1.

• This patch fails build dependency tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rdblue]

The test failure looks unrelated to my changes.

[SparkQA]

Test build #2828 has finished for PR 12437 at commit f05edc1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/1.6

[rdblue]

Thank you @srowen!