[SPARK-32686][PYTHON] Un-deprecate inferring DataFrame schema from list of dict

[nchammas]

What changes were proposed in this pull request?

As discussed in #29491 (comment) and in SPARK-32686, this PR un-deprecates Spark's ability to infer a DataFrame schema from a list of dictionaries. The ability is Pythonic and matches functionality offered by Pandas.

Why are the changes needed?

This change clarifies to users that this behavior is supported and is not going away in the near future.

Does this PR introduce any user-facing change?

Yes. There used to be a UserWarning for this, but now there isn't.

How was this patch tested?

I tested this manually.

Before:

>>>spark.createDataFrame(spark.sparkContext.parallelize([{'a': 5}]))
/Users/nchamm/Documents/GitHub/nchammas/spark/python/pyspark/sql/session.py:388: UserWarning: UsingRDDofdicttoinferSchemaisdeprecated. Usepyspark.sql.Rowinsteadwarnings.warn("Using RDD of dict to inferSchema is deprecated. "DataFrame[a: bigint]
>>>spark.createDataFrame([{'a': 5}])
.../python/pyspark/sql/session.py:378: UserWarning: inferringschemafromdictisdeprecated,pleaseusepyspark.sql.Rowinsteadwarnings.warn("inferring schema from dict is deprecated,"DataFrame[a: bigint]

After:

>>>spark.createDataFrame(spark.sparkContext.parallelize([{'a': 5}]))
DataFrame[a: bigint] >>>spark.createDataFrame([{'a': 5}])
DataFrame[a: bigint]

[nchammas]

cc @HyukjinKwon

[SparkQA]

Test build #127755 has finished for PR 29510 at commit 4e5c365.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Looks good assuming from the history:

• Python 2.7's dict does not keep the order, which is fixed in Python 3 and PySpark at SPARK-29748
• It was deprecated when Row API was introduced at 51aa135 in SPARK-2010. I think we now target Python usability and only allowing Row doesn't look Python friendly.

[HyukjinKwon]

cc @BryanCutler fyi

[nchammas]

Yeah, this deprecation has been around for ~6 years. I wonder if we should check with dev@ to make sure there are no surprising consequences of un-deprecating it?

[SparkQA]

Test build #127845 has finished for PR 29510 at commit 37ceb56.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BryanCutler]

LGTM

[SparkQA]

Test build #127851 has finished for PR 29510 at commit 5136829.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[BryanCutler]

merged to master, thanks @nchammas !