[SPARK-32130][SQL] Disable the JSON option inferTimestamp by default

[MaxGekk]

What changes were proposed in this pull request?

Set the JSON option inferTimestamp to false if an user don't pass it as datasource option.

Why are the changes needed?

To prevent perf regression while inferring schemas from JSON with potential timestamps fields.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

• Modified existing tests in JsonSuite and JsonInferSchemaSuite.
• Regenerated results of JsonBenchmark in the environment:

Item	Description
Region	us-west-2 (Oregon)
Instance	r3.xlarge
AMI	ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 (ami-06f2f779464715dc5)
Java	OpenJDK 64-Bit Server VM 1.8.0_252 and OpenJDK 64-Bit Server VM 11.0.7+10

[MaxGekk]

I am running JSONBenchmark now. If it will show significantly different numbers, I will push the changes to this PR.

[gengliangwang]

Shall we add migration guide as well?

[MaxGekk]

If it will show significantly different numbers ...

Some numbers are different on JDK 8, I am generating benchmark results on JDK 11, and will push the changes soon.

[dongjoon-hyun]

Thank you, @MaxGekk !

[MaxGekk]

I am going to add this to the SQL migration guide:

  - In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference.

The question is about the version from which we are going to disable the inference - Spark 3.1 or Spark 3.0.1.

[dongjoon-hyun]

I set the target version to 3.0.1. This should go to 3.0.1.
This is a regression due to SPARK-26246, isn't it?

[cloud-fan]

Yea this should go to branch-3.0 to fix the perf regression.

AFAIK migration guide is only for major/feature releases, so there is no migration guide for 3.0.1. Ideally 3.0.1 should be Spark 3.0 when it's released, so we can simply remove that item from the 3.0 migration guide.

[MaxGekk]

@cloud-fan There are items for minor versions too Upgrading from Spark SQL 2.4 to 2.4.1. See at docs/sql-migration-guide.md. I am going to add a separate section Upgrading from Spark SQL 3.0 to 3.0.1

[SparkQA]

Test build #124770 has finished for PR 28966 at commit 277a679.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

This #28966 (comment) took 7 hr 37 min :-(

[dongjoon-hyun]

Sorry, but this test case seems to be supposed to run infer timestamp always. Why does this become faster?

- infer timestamps from files                       84621          84939         526          0.1        8462.1       0.0X
+ infer timestamps from files                       18102          18120          28          0.6        1810.2       0.1X

[dongjoon-hyun]

It seems that we need to update JsonBenchmark together.

readBench.addCase("infer timestamps from files", numIters) { _ =>
    spark.read.json(timestampDir).noop()
}

[dongjoon-hyun]

Never mind. Let's update the benchmark suite later as a follow-up.

[dongjoon-hyun]

+1, LGTM. Merged to master/3.0. Thank you all.
(All UTs including R already passed in the on-going run.)

[SparkQA]

Test build #124800 has finished for PR 28966 at commit 43fc9aa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

It takes 7 hours ..? I think we should really revisit the test and cut down. Or maybe it's time to think about splitting and migrating to Github Actions completely ..