Skip to content

[SPARK-53535][SQL][FOLLOWUP] Fix findCheapestGroupField returning invalid Map - #52758

Closed
ZiyaZa wants to merge 1 commit into
apache:masterfrom
ZiyaZa:fix-missing-struct-with-map
Closed

[SPARK-53535][SQL][FOLLOWUP] Fix findCheapestGroupField returning invalid Map#52758
ZiyaZa wants to merge 1 commit into
apache:masterfrom
ZiyaZa:fix-missing-struct-with-map

Conversation

@ZiyaZa

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This PR fixes a bug from #52557, where we are reading an additional field if all the requested fields of a struct are missing from the Parquet file. We used to always pick the cheapest leaf column of the struct. However, if this leaf was inside a Map column, then we'd generate an invalid Map type like the following:

optional group _1 (MAP) {
repeated group key_value {
required boolean key;
}
}

Since there is no value field in this group, we'd fail later when trying to convert this Parquet type to a Spark type. This PR changes the additional field selection logic to enforce selecting a field from both the key and the value of the map, which can now give us a type like following:

optional group _1 (MAP) {
repeated group key_value {
required boolean key;
optional group value {
optional int32 _2;
}
}
}

Why are the changes needed?

To fix a critical bug where we would throw an exception when reading a Parquet file.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

@ZiyaZaZiyaZa changed the title [SPARK-53535] Fix findCheapestGroupField returning invalid Map[SPARK-53535][SQL] Fix findCheapestGroupField returning invalid MapOct 28, 2025
@cloud-fancloud-fan changed the title [SPARK-53535][SQL] Fix findCheapestGroupField returning invalid Map[SPARK-53535][SQL][FOLLOWUP] Fix findCheapestGroupField returning invalid MapOct 28, 2025
@cloud-fan

Copy link
Copy Markdown
Contributor

thanks, merging to master!

huangxiaopingRD pushed a commit to huangxiaopingRD/spark that referenced this pull request Nov 25, 2025
…alid Map
### What changes were proposed in this pull request?
This PR fixes a bug from apache#52557, where we are reading an additional field if all the requested fields of a struct are missing from the Parquet file. We used to always pick the cheapest leaf column of the struct. However, if this leaf was inside a Map column, then we'd generate an invalid Map type like the following:
```
optional group _1 (MAP) {
repeated group key_value {
required boolean key;
}
}
```
Since there is no `value` field in this group, we'd fail later when trying to convert this Parquet type to a Spark type. This PR changes the additional field selection logic to enforce selecting a field from both the key and the value of the map, which can now give us a type like following:
```
optional group _1 (MAP) {
repeated group key_value {
required boolean key;
optional group value {
optional int32 _2;
}
}
}
```
### Why are the changes needed?
To fix a critical bug where we would throw an exception when reading a Parquet file.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New unit tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closesapache#52758 from ZiyaZa/fix-missing-struct-with-map.
Authored-by: Ziya Mukhtarov <ziya5muxtarov@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@ZiyaZa@cloud-fan