[SPARK-37670][FOLLOWUP][SQL][TESTS][3.2] Update TPCDS golden files

[dongjoon-hyun]

What changes were proposed in this pull request?

This is a followup of #34929 to update TPCDS plan test golden file due to ID number changes.

Why are the changes needed?

Currently, branch-3.2 is broken.

• https://github.com/apache/spark/commits/branch-3.2

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

[dongjoon-hyun]

cc @maryannxue , @cloud-fan , @HyukjinKwon , @sunchao

[HyukjinKwon]

👍

[dongjoon-hyun]

Thank you, @HyukjinKwon !

[sunchao]

LGTM pending CI

[cloud-fan]

thanks!

[dongjoon-hyun]

Thank you, @sunchao and @cloud-fan .

[dongjoon-hyun]

q5 failure is fixed but q4 seems to be indeterministic for some reasons in branch-3.2. The result seems to be oscillated with new values and old values. Let me take a look at more.

- check simplified (tpcds-v1.4/q4) *** FAILED *** (1 second, 36 milliseconds)
...
*** 1 TEST FAILED ***
Failed: Total 9599, Failed 1, Errors 0, Passed 9598, Ignored 29

[dongjoon-hyun]

Here is the summary.

• The original PR proposed Make CTE IDs more deterministic by starting from 0 for each query.
• However, the current status generate indeterministic IDs in the following two commands.

SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite -- -z (tpcds-v1.4/q4)"
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite"

However, the query plan structure itself looks identical. Only different ID is the root cause of failure here.

[cloud-fan]

Imagine that there is a global counter for CTE IDs, running a query along will have different IDs with running many queries. I think we need to regen the golden files with SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStabilitySuite" to match Github Action.

[dongjoon-hyun]

That golden files in branch-3.2 are the one already generated by that method, @cloud-fan . And, it fails in GitHub Action environment.

[dongjoon-hyun]

Let me comment on the original PR.

[dongjoon-hyun]

Here

• [SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs #34929 (comment)
• [SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs #34929 (comment)

[dongjoon-hyun]

As an alternative, reverting PR is also under testing.

• Revert "[SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs" #36819

[dongjoon-hyun]

Thank you everyone. This is closed according to @cloud-fan 's approach.

• [SPARK-37670][SQL] Support predicate pushdown and column pruning for de-duped CTEs #34929 (comment)