ARROW-13086: [Python] Expose Parquet ArrowReaderProperties::coerce_int96_timestamp_unit_

[isichei]

No description provided.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13086

[pitrou]

@isichei Can you take a look at the CI failures?

[isichei]

@pitrou - will take a look over the next few days. Bit confused as I am seeing tests pass on my side that are failing on CI (for example the tests I added) 🤔

Will try to replicate the docker CI build on my machine and get back to you

[jorisvandenbossche]

You migth have built Arrow/pyarrow without DATASET enabled?

The failures are from _dataset.pyx, you will need to add the new option to ParquetReadOptions / _PARQUET_READ_OPTIONS, I think.

[isichei]

I've gotten stuck on the dataset API and not sure what is missing:

• pq.ParquetFile definitely works with the new coerce_int96_timestamp_unit parameter.
• I've added the new parameter the ParquetScanOptions as in _dataset.pyx that class has access to the ArrowReaderProperties cpp class which has the setter for the parameter.
• I have also added the parameter to the _ParquetDatasetV2 (in parquet.py) allowing it to be passed down to ParquetFileFormat and ParquetScanOptions.
• I've added a test (in test_dataset.py::test_parquet_scan_options) to check that this is actually being set properly and as far as I can tell it is being set.
• However, when i call pq.read_table which uses _ParquetDatasetV2 the test fails at it looks like coerce_int96_timestamp_unit is not being set on reading the parquet file and I can't figure out why (see parquet/test_datetime.py::test_coerce_int96_timestamp_overflow[read_table]).

### Next Steps

If someone else doesn't have time to look at this in more detail. It might be beneficial for me to just make this PR expose the parameter to ParquetFile and drop it from the Dataset API (so it is ready to merge for V5 release). Then I can create a new feature request on JIRA to expose the Dataset API to the new parameter.

Let me know what you think.

[jorisvandenbossche]

@isichei I will take a look

[jorisvandenbossche]

I think the option needed to be added to ParquetReadOptions instead of ParquetFragmentScanOptions, because it is an option that determines the schema of the resulting dataset, and not just how an individual fragment gets scanned.
To do that, it required a small addition in the C++ code for the Parquet Dataset.

[isichei]

@jorisvandenbossche thanks for the changes. I added a minor addition and test. The test might be a little redudent so happy to take out.

I've noted the failing Pandas test will push a fix later this evening.

[isichei]

OK, think tests are all now passing as far as I can tell and is ready to merge?

My apologies to your CI/CD resources there...

[jorisvandenbossche]

@isichei I rebased this and did a small clean-up of the test (the reason Windows was failing was because of a wrong format string (%Y-%m-%s instead of %Y-%m-%d), but in the end I removed the comparison as strings alltogether).

[isichei]

Thanks @jorisvandenbossche much appreciated!

[kszucs]

Thanks @isichei!