[SPARK-28441][SQL][Python] Fix error when non-foldable expression is used in correlated scalar subquery

[viirya]

What changes were proposed in this pull request?

In SPARK-15370, We checked the expression at the root of the correlated subquery, in order to fix count bug. If a PythonUDF in in the checking path, evaluating it causes the failure as we can't statically evaluate PythonUDF. The Python UDF test added at SPARK-28277 shows this issue.

If we can statically evaluate the expression, we intercept NULL values coming from the outer join and replace them with the value that the subquery's expression like before, if it is not, we replace them with the PythonUDF expression, with statically evaluated parameters.

After this, the last query in udf-except.sql which throws java.lang.UnsupportedOperationException can be run:

SELECT t1.k
FROM t1
WHERE t1.v <= (SELECT udf(max(udf(t2.v)))
FROM t2
WHERE udf(t2.k) = udf(t1.k))
MINUS
SELECT t1.k
FROM t1
WHERE udf(t1.v) >= (SELECT min(udf(t2.v))
FROM t2
WHERE t2.k = t1.k)
-- !query 2 schema
struct<k:string>
-- !query 2 output
two

Note that this issue is also for other non-foldable expressions, like rand. As like PythonUDF, we can't call eval on this kind of expressions in optimization. The evaluation needs to defer to query runtime.

How was this patch tested?

Added tests.