[SPARK-47129][CONNECT][SQL][3.5] Make ResolveRelations cache connect plan properly

[zhengruifeng]

What changes were proposed in this pull request?

Make ResolveRelations handle plan id properly

cherry-pick bugfix #45214 to 3.5

Why are the changes needed?

bug fix for Spark Connect, it won't affect classic Spark SQL

before this PR:

from pyspark.sql import functions as sf
spark.range(10).withColumn("value_1", sf.lit(1)).write.saveAsTable("test_table_1")
spark.range(10).withColumnRenamed("id", "index").withColumn("value_2", sf.lit(2)).write.saveAsTable("test_table_2")
df1 = spark.read.table("test_table_1")
df2 = spark.read.table("test_table_2")
df3 = spark.read.table("test_table_1")
join1 = df1.join(df2, on=df1.id==df2.index).select(df2.index, df2.value_2)
join2 = df3.join(join1, how="left", on=join1.index==df3.id)
join2.schema

fails with

AnalysisException: [CANNOT_RESOLVE_DATAFRAME_COLUMN] Cannot resolve dataframe column "id". It's probably because of illegal references like `df1.select(df2.col("a"))`. SQLSTATE: 42704

That is due to existing plan caching in ResolveRelations doesn't work with Spark Connect

=== Applying Rule org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations ===
'[#12]Join LeftOuter, '`==`('index, 'id) '[#12]Join LeftOuter, '`==`('index, 'id)
!:- '[#9]UnresolvedRelation [test_table_1], [], false :- '[#9]SubqueryAlias spark_catalog.default.test_table_1
!+- '[#11]Project ['index, 'value_2] : +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false
! +- '[#10]Join Inner, '`==`('id, 'index) +- '[#11]Project ['index, 'value_2]
! :- '[#7]UnresolvedRelation [test_table_1], [], false +- '[#10]Join Inner, '`==`('id, 'index)
! +- '[#8]UnresolvedRelation [test_table_2], [], false :- '[#9]SubqueryAlias spark_catalog.default.test_table_1
! : +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false
! +- '[#8]SubqueryAlias spark_catalog.default.test_table_2
! +- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_2`, [], false
Can not resolve 'id with plan 7

[#7]UnresolvedRelation [test_table_1], [], false was wrongly resolved to the cached one

:- '[#9]SubqueryAlias spark_catalog.default.test_table_1
+- 'UnresolvedCatalogRelation `spark_catalog`.`default`.`test_table_1`, [], false

Does this PR introduce any user-facing change?

yes, bug fix

How was this patch tested?

added ut

Was this patch authored or co-authored using generative AI tooling?

ci

[dongjoon-hyun]

Please update the JIRA information first because we cannot backport an improvement, @zhengruifeng .

[zhengruifeng]

thanks @dongjoon-hyun for reminder, I have updated the Type and versions

[dongjoon-hyun]

+1, LGTM. Thank you, @zhengruifeng .

[zhengruifeng]

the failed java linter seems unrelated, it was already broken in branch-3.5 https://github.com/apache/spark/actions/runs/8886943128/job/24404782422

[zhengruifeng]

thanks @dongjoon-hyun so much!

merged to branch-3.5