[SPARK-26856][PYSPARK][FOLLOWUP] Fix UT failure due to wrong patterns for Kinesis assembly

[dongjoon-hyun]

What changes were proposed in this pull request?

After SPARK-26856, Kinesis Python UT fails with Found multiple JARs exception due to a wrong pattern.

• https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/104171/console

Exception: Found multiple JARs: .../spark-streaming-kinesis-asl-assembly-3.0.0-SNAPSHOT.jar,
.../spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT.jar;
please remove all but one

It's because the pattern was changed in a wrong way.

Original

kinesis_asl_assembly_dir, "target/scala-*/%s-*.jar"%name_prefix))
kinesis_asl_assembly_dir, "target/%s_*.jar"%name_prefix))

After SPARK-26856

project_full_path, "target/scala-*/%s*.jar"%jar_name_prefix))
project_full_path, "target/%s*.jar"%jar_name_prefix))

The actual kinesis assembly jar files look like the followings.

SBT Build

-rw-r--r-- 1 dongjoon staff 87459461 Apr 1 19:01 spark-streaming-kinesis-asl-assembly-3.0.0-SNAPSHOT.jar
-rw-r--r-- 1 dongjoon staff 309 Apr 1 18:58 spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT-tests.jar
-rw-r--r-- 1 dongjoon staff 309 Apr 1 18:58 spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT.jar

MAVEN Build

-rw-r--r-- 1 dongjoon staff 8.6K Apr 1 18:55 spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT-sources.jar
-rw-r--r-- 1 dongjoon staff 8.6K Apr 1 18:55 spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT-test-sources.jar
-rw-r--r-- 1 dongjoon staff 8.7K Apr 1 18:55 spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT-tests.jar
-rw-r--r-- 1 dongjoon staff 21M Apr 1 18:55 spark-streaming-kinesis-asl-assembly_2.12-3.0.0-SNAPSHOT.jar

In addition, after SPARK-26856, the utility function search_jar is shared to find avro jar files which are identical for both sbt and mvn. To sum up, The current jar pattern parameter cannot handle both kinesis and avro jars. This PR splits the single pattern into two patterns.

How was this patch tested?

Manual. Please note that this will remove only Found multiple JARs exception. Kinesis tests need more configurations to run locally.

$ build/sbt -Pkinesis-asl test:package streaming-kinesis-asl-assembly/assembly
$ export ENABLE_KINESIS_TESTS=1
$ python/run-tests.py --python-executables python2.7 --module pyspark-streaming

[dongjoon-hyun]

cc @HyukjinKwon and @gaborgsomogyi .

[HyukjinKwon]

LGTM

[HyukjinKwon]

Thanks for catching this.

[dongjoon-hyun]

Thank you for review and approval, @HyukjinKwon .

[SparkQA]

Test build #104178 has finished for PR 24268 at commit 690c136.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Hey @dongjoon-hyun, just to doubly sure, can you check Avro ones as well?

[dongjoon-hyun]

Ah, thank you so much, @HyukjinKwon . Avro has different pattern. I'll update PR once more with that.

[SparkQA]

Test build #104181 has finished for PR 24268 at commit d738df7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #104182 has finished for PR 24268 at commit 0a03c16.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

Thank you for review and merging, @HyukjinKwon !

[gaborgsomogyi]

Late LGTM, thank you @dongjoon-hyun for fixing this.