Parquet-avro fails to decode array of record with a single field name "element" correctly

[asfimport]

Found this issue while investigating SPARK-16344.

For the following Parquet schema

message root {
  optional group f (LIST) {
    repeated group list {
      optional group element {
        optional int64 element;
      }
    }
  }
}

parquet-avro decodes it as something like this:

record SingleElement {
  int element;
}

record NestedSingleElement {
  SingleElement element;
}

record Spark16344Wrong {
  array<NestedSingleElement> f;
}

while correct interpretation should be:

record SingleElement {
  int element;
}

record Spark16344 {
  array<SingleElement> f;
}

The reason is that the element syntactic group for LIST in

<list-repetition> group <name> (LIST) {
  repeated group list {
    <element-repetition> <element-type> element;
  }
}

is recognized as a record field named element. The problematic code lies in AvroRecordConverter.isElementType(). We should probably check the standard 3-level layout first before falling back to the legacy 2-level layout.

Reporter: Cheng Lian / @liancheng
Assignee: Ryan Blue / @rdblue

Related issues:

• Avro's isElementType() change breaks the reading of some parquet(1.8.1) files (is caused by)
• Array of struct with a single field name "element" can't be decoded from Parquet files written by Spark 1.6+ (is related to)

PRs and other links:

• PR #352

_{Note: This issue was originally created as PARQUET-651. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
Fixed in PR #352 by converting the file's element schema to Avro and comparing it with the expected schema. Thanks to [~lian cheng], who came up with the solution for SPARK-16344!

[asfimport]

Ryan Blue / @rdblue:
Merged #352. Thanks for reviewing, [~lian cheng]!