[SPARK-42049][SQL][FOLLOWUP] Always filter away invalid ordering/partitioning

[cloud-fan]

What changes were proposed in this pull request?

This is a follow-up of #37525 . When the project list has aliases, we go to the projectExpression branch which filters away invalid partitioning/ordering that reference non-existing attributes in the current plan node. However, this filtering is missing when the project list has no alias, where we directly return the child partitioning/ordering.

This PR fixes it.

Why are the changes needed?

to make sure we always return valid output partitioning/ordering.

Does this PR introduce any user-facing change?

no

How was this patch tested?

new tests

[cloud-fan]

cc @peter-toth

[dongjoon-hyun]

cc @sunchao , @huaxingao , @viirya , too.

[peter-toth]

Good catch! Thanks for this follow-up!

But, I'm not sure this part is correct as sortOrder.children should be filtered with _.references.subsetOf(outputSet) separately.

E.g. this test (is a bit artifical though) fails now:

 test("SPARK-42049: Improve AliasAwareOutputExpression - no alias but still prune expressions 2") {
withSQLConf(SQLConf.OPTIMIZER_EXCLUDED_RULES.key ->
Seq(CollapseProject.ruleName, ColumnPruning.ruleName).mkString(",")) {
val df = spark.range(2).select($"id" as "a", $"id" as "b").select($"a")
val outputOrdering = df.queryExecution.optimizedPlan.outputOrdering
assert(outputOrdering.size == 1)
assert(outputOrdering.head.child.asInstanceOf[Attribute].name == "a")
assert(outputOrdering.head.sameOrderExpressions.size == 0)
}
}

[cloud-fan]

good point!

[viirya]

Hmm, the failed test seems a related one.

[dongjoon-hyun]

Oh is it still failing?

[viirya]

I think it was failing due to latest commit.

[cloud-fan]

The failed test checks invalid ordering and I've updated it.

[dongjoon-hyun]

Merged to master/3.4.
Thank you, @cloud-fan , @peter-toth , @viirya .