[SPARK-33621][SQL] Add a way to inject data source rewrite rules

[aokolnychyi]

What changes were proposed in this pull request?

This PR adds a way to inject data source rewrite rules.

Why are the changes needed?

Right now SparkSessionExtensions allow us to inject optimization rules but they are added to operator optimization batch. There are cases when users need to run rules after the operator optimization batch (e.g. cases when a rule relies on the fact that expressions have been optimized). Currently, this is not possible.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

This PR comes with a new test.

[aokolnychyi]

cc @holdenk@dbtsai@dongjoon-hyun@rdblue@cloud-fan@sunchao@viirya@HyukjinKwon

[aokolnychyi]

We should probably wait for clarity on the discussion here before moving on with this one.

[dongjoon-hyun]

cc @gatorsmile

[rdblue]

This looks good to me, but I agree that we should change the name if anyone comes up with a better one.

[SparkQA]

Test build #132072 has finished for PR 30577 at commit a76f4c2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[aokolnychyi]

I'll wait to update this until we resolve the name issue.

[dongjoon-hyun]

@aokolnychyi . Could you address the existing comments first without waiting for the others? They are orthogonal. You don't need to wait. For example,

• This PR breaks Scala 2.13 build.
• Test case prefix.

[aokolnychyi]

Will do today, @dongjoon-hyun!

[dongjoon-hyun]

Gentle ping, @aokolnychyi .

Will do today, @dongjoon-hyun!

[maropu]

Could you update the description above, too?

spark/sql/core/src/main/scala/org/apache/spark/sql/SparkSessionExtensions.scala

Lines 37 to 48 in 29096a8

	* This current provides the following extension points:
	*
	* <ul>
	* <li>Analyzer Rules.</li>
	* <li>Check Analysis Rules.</li>
	* <li>Optimizer Rules.</li>
	* <li>Planning Strategies.</li>
	* <li>Customized Parser.</li>
	* <li>(External) Catalog listeners.</li>
	* <li>Columnar Rules.</li>
	* <li>Adaptive Query Stage Preparation Rules.</li>
	* </ul>

[aokolnychyi]

Done, thanks for catching this!

[aokolnychyi]

Sorry for the delay, @dongjoon-hyun! I've updated the PR now.

[aokolnychyi]

Gentle ping @gatorsmile on the name suggestion discussed here.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36961/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36961/

[SparkQA]

Test build #132361 has finished for PR 30577 at commit 6b38928.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[aokolnychyi]

Test failures are in HiveThriftHttpServerSuite.

[dongjoon-hyun]

+1, LGTM. Thank you for updating, @aokolnychyi .
Merged to master for Apache Spark 3.2.0.