[SPARK-30128][DOCS][PYTHON][SQL] Document/promote 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC

[HyukjinKwon]

What changes were proposed in this pull request?

This PR adds and exposes the options, 'recursiveFileLookup' and 'pathGlobFilter' in file sources 'mergeSchema' in ORC, into documentation.

• recursiveFileLookup at file sources: [SPARK-27990][SQL][ML] Provide a way to recursively load data from datasource #24830 (SPARK-27627)
• pathGlobFilter at file sources: [SPARK-27627][SQL] Make option "pathGlobFilter" as a general option for all file sources #24518 (SPARK-27990)
• mergeSchema at ORC: [SPARK-11412][SQL] Support merge schema for ORC #24043 (SPARK-11412)

Note thattimeZone option was not moved from DataFrameReader.options as I assume it will likely affect other datasources as well once DSv2 is complete.

Why are the changes needed?

To document available options in sources properly.

Does this PR introduce any user-facing change?

In PySpark, pathGlobFilter can be set via DataFrameReader.(text|orc|parquet|json|csv) and DataStreamReader.(text|orc|parquet|json|csv).

How was this patch tested?

Manually built the doc and checked the output. Option setting in PySpark is rather a logical change. I manually tested one only:

$ ls -al tmp
...
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 aa
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 ab
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 ac
-rw-r--r-- 1 hyukjin.kwon staff 3 Dec 20 12:19 cc

>>>spark.read.text("tmp", pathGlobFilter="*c").show()

+-----+
|value|
+-----+
| ac|
| cc|
+-----+

[HyukjinKwon]

Let me just cc everybody involved in those PRs and JIRAs.

@nchammas, @gengliangwang, @WeichenXu123, @WangGuangxin, @dongjoon-hyun, @gatorsmile, @cloud-fan, @Ngone51, @mengxr

[nchammas]

I'm confused by the title of the PR. It looks like you're only working with pathGlobFilter. And didn't we address recursiveFileLookup and mergeSchema in #26718, #26730, and #26755?

[HyukjinKwon]

@nchammas, I wanted to mention this in Scala side too in the PR title.

[SparkQA]

Test build #115602 has finished for PR 26958 at commit dde76ab.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #115600 has finished for PR 26958 at commit d5ff1e4.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

retest this please

[SparkQA]

Test build #115615 has finished for PR 26958 at commit dde76ab.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Let me merge this in few days if you guys are fine since it's just a doc change.

[dongjoon-hyun]

+1, LGTM.

[HyukjinKwon]

Thanks all. Merged to master.