[C++][Dataset] Parquet schema lost on dataset write

[beryan]

Describe the enhancement requested

As a user of the Arrow Dataset API I would like to write partitioned data and preserve Parquet schema information.

For example, I may have an arrow::Table containing Parquet INTERVAL data stored in it's physical type representation, a fixed_len_byte_array of length 12. Because no arrow::Schema types are a direct match I use a arrow::FixedSizeBinaryBuilder to create the table. Existing properties and arrow::dataset::FileSystemDataset::Write() don't support providing a native schema for the output file format. As a result, the Parquet logical types of written data that do not have an arrow::schema equivalent are lost.

Some Parquet logical types affected:

• interval
• uuid
• enum
• json
• bson

Current behavior: When using the Arrow Dataset API, data type roundtripping is limited by the types arrow::schema can represent

Desired behavior: Provide a parquet schema that allows the user to specify a target schema.

Component(s)

C++

[pitrou]

This would probably need to be added to the Arrow-Parquet C++ APIs first. I see two possible kinds of API:

1. expose an additional parameter to pass the full Parquet schema (we would need to validate the Parquet schema is compatible with the provided Arrow data)
2. allow specifying the Parquet logical type of a Arrow column, using dedicated key-value metadata: we need to devise a binary encoding for a Parquet node description (can we use Thrift?)

@wgtmac @mapleFU

[mapleFU]

Emmm can this be solved by direct mapping arrow type / extension type to specific Parquet Logical Type (like json and uuid) ?

I'm ok to add the api to pass the full parquet schema, but maybe it's a bit complex for just these schemas?

[wgtmac]

expose an additional parameter to pass the full Parquet schema

I'm in favor of this because it is simple.

However, these logical types have different issues out there:

• json and uuid: both have canonical extension types. json has been supported already while uuid has an in-progress PR: GH-43807: [C++][Python] Add UUID extension type conversion support to/from Parquet #45866
• bson: it does not have canonical extension type but it is similar to json.
• interval: IIRC, this type does not have a perfect mapping to Arrow's interval types.
• enum: AFAIK, this type is only supported by parquet-java. Not sure if we need to support it on our end.

[beryan]

I appreciate the discussion and consideration!

I'll add that I wasn't able to find a suitable workaround before realizing an enhancement would be ideal.

• I tried writing with the uuid extension type. Since that didn't work I assumed extension types wouldn't work in general. I didn't realize there were extra steps to cover the to/from Parquet process. Thanks for linking that PR.
• I tried writing custom key value metadata, but that also didn't work. I noticed there are some issues (e.g. The metadata info will lost in parquet file schema after writing the parquet file by calling the FileSystemDataset::Write() method. #33335) for metadata loss so that seems currently expected.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

[beryan]

I still think this is a worthwhile enhancement! This means library users have type support limitations when writing partitioned parquet datasets.