Skip to content

[SPARK-53068][SQL][TESTS] Mark TransformWith*Suite as SlowSQLTest in sql/core - #51782

Closed
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-53068
Closed

[SPARK-53068][SQL][TESTS] Mark TransformWith*Suite as SlowSQLTest in sql/core#51782
dongjoon-hyun wants to merge 1 commit into
apache:masterfrom
dongjoon-hyun:SPARK-53068

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Aug 1, 2025

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR aims to move TransformWith*Suite tests from sql - other tests pipeline to sql - slow tests pipeline for rebalancing the test workload and avoiding the AS-IS GitHub Action test timeout (120m).

$ find sql/core -name 'TransformWith*Suite.scala'
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateClusterSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateUnsafeRowSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateChainingSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateAvroSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithStateInitialStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithListStateTTLSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithValueStateTTLSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateTTLSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithListStateSuite.scala
sql/core/src/test/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServerSuite.scala

Why are the changes needed?

This PR reduces sql - other tests running time from 101 mins to 84 mins.

BEFORE

Screenshot 2025-08-01 at 12 30 19

AFTER

Does this PR introduce any user-facing change?

No because this is a test change.

How was this patch tested?

Pass the CIs with the rebalanced tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun

Copy link
Copy Markdown
MemberAuthor

Could you review this GitHub Action test rebalancing PR, @huaxingao ?

@dongjoon-hyun

Copy link
Copy Markdown
MemberAuthor

All tests passed.

@huaxingaohuaxingao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dongjoon-hyun

Copy link
Copy Markdown
MemberAuthor

Thank you so much, @huaxingao . 😄

@dongjoon-hyun
dongjoon-hyun deleted the SPARK-53068 branch August 1, 2025 22:23
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@dongjoon-hyun@huaxingao