[branch-48] Set the default value of datafusion.execution.collect_statistics to true #16447

[blaginin]

Backport #16447 by @AdamGS to #16486

[alamb]

This is a somewhat subtle issue so I will try and summarize:

In DataFusion 47 and earlier,

1. Calling DataFrame::register_parquet collected statistics for the table at create time (slower to create table, potentially faster quer)
2. Calling CREATE EXTERNAL TABLE did not collect statistics (faster to create table, but potentially slower query)

There are more details about this on the ticket from @davisp here:

• Mismatched configuration between SessionContext::register_parquet and CREATE EXTERNAL TABLE #15908

In DataFusion 48.0.0:

1. Make SessionContext::register_parquet obey collect_statistics config #16080 made DataFrame::register_parquet and CREATE EXTERNAL TABLE DID NOT collect statistics.

However this means that users who were relying on statistics, such as @AdamGS , saw queries get slower (see #16444)

Thus this PR proposes changing DataFusion 48.0.1 so

1. Both DataFrame::register_parquet and CREATE EXTERNAL TABLEWILL collect statistics.

Note that this is consistent with the behavior on the latest main (what will be released in DataFusion 49.0.0):

• Default to collecting statistics when creating LIstingTables #16158

Since we have already made this change, the thinking is that by changing 48.0.1 we'll avoid people full migrating to the "no default statistics" behavior only to have to change back again in 49

[alamb]

Thank you @blaginin

[alamb]

I plan to merge this in tomorrow unless there is any more feedback

[alamb]

Thanks agian @dmitriibugakov and @blaginin