[WIP][SPARK-37359][K8S] Cleanup the Spark Kubernetes Integration tests

[holdenk]

What changes were proposed in this pull request?

Drop remove reason stats check to make the test less flaky & disable Spark R test [hasn't passed for a long time]
Try and avoid PV/PVC delete/create race condition

Why are the changes needed?

Or K8s test suite is broken so people ignore it. This is not good.

Listener bus message is not always delivered and printed
SparkR tests have been broken for a long time and I don't see any interest in fixing them
PV/PVC creation/deletion can have a race condition during integration tests.

Does this PR introduce any user-facing change?

Test only change

How was this patch tested?

WIP (waiting on CI for k8s int).

[SparkQA]

Test build #145348 has finished for PR 34636 at commit 71d7bf4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49819/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49819/

[SparkQA]

Test build #145349 has finished for PR 34636 at commit bee9d63.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49820/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49820/

[SparkQA]

Test build #145351 has finished for PR 34636 at commit a73b27c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49822/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49822/

[SparkQA]

Test build #145400 has finished for PR 34636 at commit 12a4096.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49873/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49873/

[SparkQA]

Test build #145404 has finished for PR 34636 at commit e1e61f2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49877/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49877/

[SparkQA]

Test build #145520 has finished for PR 34636 at commit 14022c0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49992/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49992/

[Yikun]

retest this please

[SparkQA]

Test build #146393 has finished for PR 34636 at commit 14022c0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50868/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50868/

[holdenk]

Closed for now we can revisit this later when we re-enable K8s testing.