AvroSchemaConverter silently drops typed_value when converting shredded variant schemas

[nssalian]

Describe the bug, including details regarding any error messages, version, and platform.

I noticed this on the parquet-cli while checking the schema for a shredded parquet file.

Essentially, given this Parquet schema:

message table {
 required group data (VARIANT(1)) {
 required binary metadata;
 optional binary value;
 optional group typed_value {
 required group name {
 optional binary value;
 optional binary typed_value (STRING);
 }
 required group age {
 optional binary value;
 optional int32 typed_value (INTEGER(8,true));
 }
 }
 }
}

AvroSchemaConverter.convert() produces:

{
 "type": "record",
 "name": "data",
 "fields": [
 {"name": "metadata", "type": "bytes"},
 {"name": "value", "type": "bytes"}
 ]
}

typed_value is missing

Expected: The converter should convert all children of the VARIANT group (including typed_value when present)

Version: 1.18.0-SNAPSHOT

I can add a fix to this myself if there are no objections. If we want to keep this behavior, we should make a note.

Component(s)

Avro

[nssalian]

CC: @emkornfield @aihuaxu @steveloughran

[emkornfield]

I think the avro schema converter might be overloaded between CLI and actual avro schema conversion then? I think other systems depend on avro not being shredded. It does seem useful at the CLI level to be able to observed shredded columns though.

[nssalian]

I think the avro schema converter might be overloaded between CLI and actual avro schema conversion then? I think other systems depend on avro not being shredded. It does seem useful at the CLI level to be able to observed shredded columns though.

Agreed. I do see that the converter serves both the CLI (parquet-cli/Schemas.java) for schema display and the Avro read/write paths (AvroReadSupport, AvroWriteSupport, AvroVariantConverter). I'll patch those as well. @emkornfield , do you mind assigning this to me so I can post a PR? Thanks.

[emkornfield]

Agreed. I do see that the converter serves both the CLI (parquet-cli/Schemas.java) for schema display and the Avro read/write paths (AvroReadSupport, AvroWriteSupport, AvroVariantConverter). I'll patch those as well. @emkornfield , do you mind assigning this to me so I can post a PR? Thanks.

Just to clarify, I'm not sure the Avro core functionality should change here (we probably don't want to return shredded for Avro). Assigned

[nssalian]

Just to clarify, I'm not sure the Avro core functionality should change here (we probably don't want to return shredded for Avro). Assigned

Thanks for clarifying and assigning. I'll patch the CLI functionality or maybe clean up the doc to say the parquet schema works better.

[steveloughran]

should it convert or should it unshred the variant and attach that data?