[SPARK-30665][DOCS][BUILD][PYTHON] Eliminate pypandoc dependency

[nchammas]

What changes were proposed in this pull request?

This PR removes any dependencies on pypandoc. It also makes related tweaks to the docs README to clarify the dependency on pandoc (not pypandoc).

Why are the changes needed?

We are using pypandoc to convert the Spark README from Markdown to ReST for PyPI. PyPI now natively supports Markdown, so we don't need pypandoc anymore. The dependency on pypandoc also sometimes causes issues when installing Python packages that depend on PySpark, as described in #18981.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually:

python -m venv venv
source venv/bin/activate
pip install -U pip
cd python/
python setup.py sdist
pip install dist/pyspark-3.0.0.dev0.tar.gz pyspark --version

I also built the PySpark and R API docs with jekyll and reviewed them locally.

It would be good if a maintainer could also test this by creating a PySpark distribution and uploading it to Test PyPI to confirm the README looks as it should.

[nchammas]

cc @holdenk

[SparkQA]

Test build #117496 has finished for PR 27376 at commit 4098acf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Oh, thank you for making a PR for this, @nchammas . pandoc bites us frequently.

[dongjoon-hyun]

cc @HyukjinKwon

[SparkQA]

Test build #117515 has finished for PR 27376 at commit 16d42c3.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #117518 has finished for PR 27376 at commit 70f3b88.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

BTW, this improvement should not go to branch-2.4. I updated Affected Version in JIRA, @nchammas .

[HyukjinKwon]

Looks fine except https://github.com/apache/spark/pull/27376/files#r372704609

[HyukjinKwon]

LGTM

[HyukjinKwon]

@nchammas, mind updating the PR description accordingly as well?

[nchammas]

Updated the description. Also made some tweaks to the docs README to separate R prerequisites from general API doc prerequisites. Happy to make any further changes if necessary.

[SparkQA]

Test build #117533 has finished for PR 27376 at commit 6fa5b8c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #117540 has finished for PR 27376 at commit 71182fe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Merged to master.

[SparkQA]

Test build #117542 has finished for PR 27376 at commit 98e0a31.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.