Skip to content

test: cover date maps in Parquet temporal fuzz tests - #5877

Queued
rich7420 wants to merge 1 commit into
apache:mainfrom
rich7420:test/2945-temporal-map-coverage
Queued

test: cover date maps in Parquet temporal fuzz tests#5877
rich7420 wants to merge 1 commit into
apache:mainfrom
rich7420:test/2945-temporal-map-coverage

Conversation

@rich7420

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2945.

Rationale for this change

The temporal Parquet fuzz tests cover date columns in scalar, array and struct form but omit maps.

What changes are included in this PR?

Add a map<date,date> column to the existing test matrix. Compare maps without ORDER BY and assert native scans where Comet is enabled. Preserve the Spark answer checks when INT96 timestamp conversion disables Comet.

How are these changes tested?

All nine temporal variants pass locally on Spark 4.1.3. Fork CI passed, including the scans suites on Spark 3.4–4.2 and macOS Spark 4.0.

@github-actions github-actions Bot added enhancement New feature or request test Testing related labels Sep 12, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rich7420

@sunchao
sunchao added this pull request to the merge queue Sep 12, 2026

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 28c29062 against base 3810936b, preserving the existing approval. There have been no source changes since that approval. I found no new or remaining verified P1/P2 issue.

The added map<date,date> projection correctly avoids ORDER BY. The answer helper compares unordered rows and map contents. When INT96 timestamp conversion is off, the test also requires a Comet scan. When it is on, the extension's existing global-disable guard explains why only Spark answer equality is required. I checked these paths against the maintained Spark 3.5/4.0 sources.

At 2026-09-12 21:51:12 UTC, CI had 41 successful and 11 skipped checks, including Required Checks. All nine temporal variants passed in both the Spark 3.5 scans job and Spark 4.0 scans job. Both checked out dd5c1403, whose parents are exactly this base and head and whose changed test file matches the head.

Coverage remains limited to the generator's single-entry, non-null date maps. The three timestamp-output settings do not make these DATE fields exercise timestamp encodings. No separate local JVM/native suite or performance benchmark was run. Maintained Spark 3.4/4.1 source gaps remain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request test Testing related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

testParquetTemporalTypes fuzz test does not include MapType

3 participants