[SPARK-48137][INFRA] Run yarn test only in PR builders and Daily CIs

[dongjoon-hyun]

What changes were proposed in this pull request?

We have been providing a dedicated test environment for yarn and connect module because they are flaky.

• [SPARK-47051][INFRA] Create a new test pipeline for yarn and connect #45107

However, they are still flaky. So, this PR aims to run yarn test only in PR builders (if needed) and Daily CIs (always).

• Reduce the irrelevant re-tries by triggering YARN CI only when we need to test YARN module.
• Protect YARN CI from connect flakiness by providing an independent GitHub Action environment in PR Builders and Daily CIs.
• Lastly, commit builder will offload YARN module tests to the daily CIs

Why are the changes needed?

• PR builders provide an extensive test coverage with YARN testing.

• Daily CIs with YARN tests

  • NON-ANSI CI: https://github.com/apache/spark/actions/workflows/build_non_ansi.yml (1AM)
  • Java 21 SBT CI: https://github.com/apache/spark/actions/workflows/build_java21.yml (4AM)
  • RockDB UI CI: https://github.com/apache/spark/actions/workflows/build_rockdb_as_ui_backend.yml (6AM)
  • Maven Java 17 CI: https://github.com/apache/spark/actions/workflows/build_maven.yml (1PM)
  • Maven Java 21 CI: https://github.com/apache/spark/actions/workflows/build_maven_java21.yml (2PM)
  • Maven Java 21 on AppleSilicon CI: https://github.com/apache/spark/actions/workflows/build_maven_java21_macos14.yml (8PM every two days)

• YARN CI has been flaky in GitHub Action environment and requires irrelevant re-tries very frequently.

  • https://github.com/apache/spark/actions/runs/8962451417/job/24611353908 (2024-05-05)
  • https://github.com/apache/spark/actions/runs/8962440192/job/24611326971 (2024-05-05)

[info] *** 6 TESTS FAILED ***
[error] Failed tests:
[error] org.apache.spark.deploy.yarn.YarnClusterSuite
[error] (yarn / Test / test) sbt.TestsFailedException: Tests unsuccessful

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

WDYT, @HyukjinKwon ?

[dongjoon-hyun]

Alternatively, we can ignore YarnClusterSuite completely. However, I'm not sure who is going to take on that JIRA issue in the future. So, I choose this path.

[dongjoon-hyun]

Could you review this PR too, @LuciferYang ?

[LuciferYang]

LGTM

[dongjoon-hyun]

Thank you so much! Merged to master~

[LuciferYang]

Sorry, I just reviewed modules.py and have a small question: if the common/network-yarn module has changed, should yarn also be set to true?

[dongjoon-hyun]

Yes, source_file_regexes has common/network-yarn and I believe it makes sense.

spark/dev/sparktestsupport/modules.py

Lines 1447 to 1460 in 7c728b2

	yarn=Module(
	name="yarn",
	dependencies=[],
	source_file_regexes=[
	"resource-managers/yarn/",
	"common/network-yarn/",
	],
	build_profile_flags=["-Pyarn"],
	sbt_test_goals=[
	"yarn/test",
	"network-yarn/test",
	],
	test_tags=["org.apache.spark.tags.ExtendedYarnTest"],
	)