[SPARK-19570][PYSPARK] Allow to disable hive in pyspark shell

[zjffdu]

What changes were proposed in this pull request?

SPARK-15236 do this for scala shell, this ticket is for pyspark shell. This is not only for pyspark itself, but can also benefit downstream project like livy which use shell.py for its interactive session. For now, livy has no control of whether enable hive or not.

How was this patch tested?

I didn't find a way to add test for it. Just manually test it.
Run bin/pyspark --master local --conf spark.sql.catalogImplementation=in-memory and verify hive is not enabled.

[SparkQA]

Test build #72796 has finished for PR 16906 at commit 431bcf8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zjffdu]

@holdenk Please help review

[holdenk]

Let me take a look tomorrow.

[holdenk]

Ah interesting, thanks for working on this. I've been working on a JIRA which also requires "stand alone" style testing, I'll make the (WIP) PR for that and maybe you can take a look at it. Perhaps we could generalize that one a bit and make something for testing things like this (provided the overhead isn't too high).

[felixcheung]

I think we need to lowercase the conf value as in #16907

[zjffdu]

Yeah, make sense. Fixed it.

[SparkQA]

Test build #74944 has finished for PR 16906 at commit ba10538.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #75428 has finished for PR 16906 at commit 5322189.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[felixcheung]

LGTM, @holdenk do you have any comment on this?

[holdenk]

I think this looks reasonable, although it would maybe make sense to add a warning if the user has explicitly requested hive support and we are falling through to non-hive support (e.g. in the except side of the try block).

[felixcheung]

+1 on that, we do have the log on the R side.

[SparkQA]

Test build #75562 has finished for PR 16906 at commit 63942c1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zjffdu]

Thanks @holdenk@felixcheung for review, I just added the warning log

[zjffdu]

Kindly ping @holdenk

[holdenk]

LGTM, thanks for working on this @zjffdu

[holdenk]

Merged to master