[SPARK-6796][Streaming][WebUI] Add "Active Batches" and "Completed Batches" lists to StreamingPage

[zsxwing]

This PR adds two lists, Active Batches and Completed Batches. Here is the screenshot:

Due to SPARK-6766, I need to merge #5414 in my local machine to get the above screenshot.

[zsxwing]

cc @tdas and @JoshRosen

[SparkQA]

Test build #29922 has finished for PR 5434 at commit 8286a26.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[tdas]

This is great start. But we should be adding unit tests for this. Recently we have started writing unit tests for UI using selenium. Take a look at how this is written.

https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/streaming/UISeleniumSuite.scala

Try adding unit test to that do some basic testing of this feature.

[tdas]

Move this to the top of the function, i think more consistent.

[zsxwing]

I added some tests to verify the title titles and column names.

[zsxwing]

retest this please

[SparkQA]

Test build #29986 has finished for PR 5434 at commit 57f7663.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[tdas]

LGTM. Though, let me try running this locally and seeing the UI myself.

[zsxwing]

Rebased on master.

[SparkQA]

Test build #30060 has finished for PR 5434 at commit b248787.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[JoshRosen]

You could make this class abstract and leave this method undefined, then override it in the concrete classes.

[zsxwing]

Fixed. Made BatchTableBase abstract.

[SparkQA]

Test build #30125 has finished for PR 5434 at commit 86b5e7f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[tdas]

One small nit that I noticed when i ran it myself. This says events where as else where in the UI it says records. Lets change it to events to keep things consistent.

[tdas]

Another points I noticed in the UI, in the Active Batches (XXX), the value is always = number of batches retained. Its looks like that only that many batches have completed whereas 1000 batches may have completed and we showing only 100 batches. I think we should show it as Active Batches (last XX out of YY) where YY is the sum of all completed batches.

[tdas]

Also, could you make the stats at the top of the page "Waiting batches" and "Processed batches" links to the corresponding sections? And the names should be consistent, so please rename them here, keep them as "Active Batches" and "Completed Batches".

[zsxwing]

@tdas I fixed as per your suggestion. I also made "startTime" more readable. Here is the new screenshot:

[SparkQA]

Test build #30172 has finished for PR 5434 at commit 51b792e.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[SparkQA]

Test build #30174 has finished for PR 5434 at commit be50fc6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[tdas]

I am merging this. Thanks!