test: cover date maps in Parquet temporal fuzz tests - #5877
Conversation
sunchao
left a comment
There was a problem hiding this comment.
Checked 28c29062 against base 3810936b, preserving the existing approval. There have been no source changes since that approval. I found no new or remaining verified P1/P2 issue.
The added map<date,date> projection correctly avoids ORDER BY. The answer helper compares unordered rows and map contents. When INT96 timestamp conversion is off, the test also requires a Comet scan. When it is on, the extension's existing global-disable guard explains why only Spark answer equality is required. I checked these paths against the maintained Spark 3.5/4.0 sources.
At 2026-09-12 21:51:12 UTC, CI had 41 successful and 11 skipped checks, including Required Checks. All nine temporal variants passed in both the Spark 3.5 scans job and Spark 4.0 scans job. Both checked out dd5c1403, whose parents are exactly this base and head and whose changed test file matches the head.
Coverage remains limited to the generator's single-entry, non-null date maps. The three timestamp-output settings do not make these DATE fields exercise timestamp encodings. No separate local JVM/native suite or performance benchmark was run. Maintained Spark 3.4/4.1 source gaps remain.
Which issue does this PR close?
Closes #2945.
Rationale for this change
The temporal Parquet fuzz tests cover date columns in scalar, array and struct form but omit maps.
What changes are included in this PR?
Add a
map<date,date>column to the existing test matrix. Compare maps withoutORDER BYand assert native scans where Comet is enabled. Preserve the Spark answer checks when INT96 timestamp conversion disables Comet.How are these changes tested?
All nine temporal variants pass locally on Spark 4.1.3. Fork CI passed, including the scans suites on Spark 3.4–4.2 and macOS Spark 4.0.