[SPARK-46283][INFRA] Remove streaming-kinesis-asl module from MODULES_TO_TEST for branch-3.x daily tests

[LuciferYang]

What changes were proposed in this pull request?

After the merge of #43736, the master branch began testing the streaming-kinesis-asl module.

At the same time, because the daily test will reuse build_and_test.yml, the daily test of branch-3.x also began testing streaming-kinesis-asl.

However, in branch-3.x, the env ENABLE_KINESIS_TESTS is hard-coded as 1 in dev/sparktestsupport/modules.py:

spark/dev/sparktestsupport/modules.py

Lines 332 to 346 in 1321b4e

	streaming_kinesis_asl = Module(
	name="streaming-kinesis-asl",
	dependencies=[tags, core],
	source_file_regexes=[
	"connector/kinesis-asl/",
	"connector/kinesis-asl-assembly/",
	],
	build_profile_flags=[
	"-Pkinesis-asl",
	],
	environ={"ENABLE_KINESIS_TESTS": "1"},
	sbt_test_goals=[
	"streaming-kinesis-asl/test",
	],
	)

which leads to the failure of the daily test of branch-3.x:

• branch-3.3: https://github.com/apache/spark/actions/runs/7111246311
• branch-3.4: https://github.com/apache/spark/actions/runs/7098435892
• branch-3.5: https://github.com/apache/spark/actions/runs/7099811235

[info] org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite *** ABORTED *** (1 second, 14 milliseconds)
[info]   java.lang.Exception: Kinesis tests enabled using environment variable ENABLE_KINESIS_TESTS
[info] but could not find AWS credentials. Please follow instructions in AWS documentation
[info] to set the credentials in your system such that the DefaultAWSCredentialsProviderChain
[info] can find the credentials.
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils$.getAWSCredentials(KinesisTestUtils.scala:258)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient$lzycompute(KinesisTestUtils.scala:58)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.kinesisClient(KinesisTestUtils.scala:57)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.describeStream(KinesisTestUtils.scala:168)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.findNonExistentStreamName(KinesisTestUtils.scala:181)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.$anonfun$beforeAll$1(KinesisStreamSuite.scala:61)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite.runIfTestsEnabled$(KinesisFunSuite.scala:39)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.runIfTestsEnabled(KinesisStreamSuite.scala:42)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.beforeAll(KinesisStreamSuite.scala:59)
[info]   at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)
[info]   at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
[info]   at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.org$scalatest$BeforeAndAfter$$super$run(KinesisStreamSuite.scala:42)
[info]   at org.scalatest.BeforeAndAfter.run(BeforeAndAfter.scala:273)
[info]   at org.scalatest.BeforeAndAfter.run$(BeforeAndAfter.scala:271)
[info]   at org.apache.spark.streaming.kinesis.KinesisStreamTests.run(KinesisStreamSuite.scala:42)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:321)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:517)
[info]   at sbt.ForkMain$Run.lambda$runTest$1(ForkMain.java:414)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[info]   at java.lang.Thread.run(Thread.java:750)
[info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilderOldApi started
[info] Test org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite.testJavaKinesisDStreamBuilder started
[info] Test run org.apache.spark.streaming.kinesis.JavaKinesisInputDStreamBuilderSuite finished: 0 failed, 0 ignored, 2 total, 0.244s
[info] ScalaTest
[info] Run completed in 8 seconds, 542 milliseconds.
[info] Total number of tests run: 31
[info] Suites: completed 4, aborted 4
[info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0
[info] *** 4 SUITES ABORTED ***
[error] Error: Total 37, Failed 0, Errors 4, Passed 33
[error] Error during tests:
[error]   org.apache.spark.streaming.kinesis.WithoutAggregationKinesisBackedBlockRDDSuite
[error]   org.apache.spark.streaming.kinesis.WithAggregationKinesisBackedBlockRDDSuite
[error]   org.apache.spark.streaming.kinesis.WithAggregationKinesisStreamSuite
[error]   org.apache.spark.streaming.kinesis.WithoutAggregationKinesisStreamSuite
[error] (streaming-kinesis-asl / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 13 s, completed Dec 5, 2023 12:03:53 PM

This PR adds a conditional judgment for the build task:

if [[ "$MODULES_TO_TEST" == *"streaming-kinesis-asl"* ]] && [[ "${{ inputs.branch }}" =~ ^branch-3 ]]; then 
  MODULES_TO_TEST=${MODULES_TO_TEST//streaming-kinesis-asl, /}
fi

When MODULES_TO_TEST contains streaming-kinesis-asl and the test branch is not master, it removes the streaming-kinesis-asl, substring from MODULES_TO_TEST and reassigns it to MODULES_TO_TEST. This avoids the testing of the streaming-kinesis-asl module in branch-3.x.

Why are the changes needed?

Prevent the daily tests of branch-3.x from testing the streaming-kinesis-asl module.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Monitor GA after merged

Was this patch authored or co-authored using generative AI tooling?

No

[LuciferYang]

cc @HyukjinKwon @dongjoon-hyun @zhengruifeng

If we don't want to backport #43736 to branch-3.x, I think this PR could be a possible solution.

[junyuc25]

Thank you for fixing this :).

[LuciferYang]

After reconsidering, I have changed this part to match branches that start with branch-3. This way, there is no need to change this conditional expression again after cutting branch-4.x

[dongjoon-hyun]

+1, LGTM.

[LuciferYang]

Thanks @dongjoon-hyun @HyukjinKwon @LuciferYang @junyuc25 ~

[LuciferYang]

The recent daily test has run successfully:

• branch-3.3: https://github.com/apache/spark/actions/runs/7124833869
• branch-3.4: https://github.com/apache/spark/actions/runs/7125949868
• branch-3.5: https://github.com/apache/spark/actions/runs/7127294228

[dongjoon-hyun]

Great! Thank you for the updates, @LuciferYang .