[C++][Dataset] Support null -> other type promotion in Dataset scanning

[asfimport]

With regarding schema evolution / normalization, we support inserting nulls for a missing column or changing nullability, or normalizing column order, but we do not yet seem to support promotion of null type to any other type.

Small python example:

In [11]: df = pd.DataFrame({"col": np.array([None, None, None, None], dtype='object')})
    ...: df.to_parquet("test_filter_schema.parquet", engine="pyarrow")
    ...:
    ...: import pyarrow.dataset as ds
    ...: dataset = ds.dataset("test_filter_schema.parquet", format="parquet", schema=pa.schema([("col", pa.int64())]))
    ...: dataset.to_table()
...
~/scipy/repos/arrow/python/pyarrow/_dataset.pyx in pyarrow._dataset.Dataset.to_table()
~/scipy/repos/arrow/python/pyarrow/_dataset.pyx in pyarrow._dataset.Scanner.to_table()
~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowTypeError: fields had matching names but differing types. From: col: null To: col: int64

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Ben Kietzman / @bkietz

Related issues:

• [Python] More graceful reading of empty String columns in ParquetDataset (is depended upon by)
• [Python][Parquet][C++] Null values in a single partition of Parquet dataset, results in invalid schema on read (is depended upon by)

PRs and other links:

• GitHub Pull Request #8343

_{Note: This issue was originally created as ARROW-9147. Please see the migration documentation for further details.}

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
This one will require more design since it's akin to do a cast.

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 8343
#8343

[asfimport]

Ji Xu:
May I know if there's plan to support other type promotion? For example:

From: col: int32 To: col: int64

[asfimport]

Gabriel Bassett:
I received the following error with arrow 2.0.0 (R):

Error in dataset___Scanner__ToTable(self): Type error: fields had matching names but differing types. From: <column_name>: bool To: <column_name>: null

Should this be fixed? Is it possible the order in which the null is encountered is handled differently?