[C++][Python][Parquet] We should allow for overriding to types by providing a schema

[asfimport]

The following shouldn't throw

>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> import pyarrow.dataset as ds
>>> pa.__version__
'2.0.0'
>>> schema = pa.schema([pa.field("utf8", pa.utf8())])
>>> table = pa.Table.from_pydict(\{"utf8": ["foo", "bar"]}, schema)
>>> pq.write_table(table, "/tmp/example.parquet")
>>> large_schema = pa.schema([pa.field("utf8", pa.large_utf8())])
>>> ds.dataset("/tmp/example.parquet", schema=large_schema,
format="parquet").to_table()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/_dataset.pyx", line 405, in
pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 2262, in
pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 122, in
pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 107, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: fields had matching names but differing types.
From: utf8: string To: utf8: large_string

Reporter: Micah Kornfield / @emkornfield

Related issues:

• [C++][Dataset] Schema evolution in Dataset scanning (is related to)

_{Note: This issue was originally created as ARROW-11353. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Note that in general, we don't have any functionality exposed in the python parquet bindings to override the type used when reading specific columns (I don't know to what extent this actually exists in the C++ layer). So this is not an issue specific to the large types. Now for the large types, it might be that this can be done more efficiently in the C++ layer than "just" casting afterwards?

For the Dataset API (what the example above is using): this is a known limitation right now that the schema has to match exactly, and almost no schema evolution is allowed (only from null -> any type). See ARROW-11003 for an umbrella issue for this (eg we also want to allow to read int32 columns as int64).

[asfimport]

Micah Kornfield / @emkornfield:
Changed the title to make this more general.  I think doing this at load time would save work.  Not sure how much of a performance impact it would have vs casting.

[thisisnic]

No longer reproducible