Add shredded Variant reader cases with variant logical type by aihuaxu · Pull Request #91 · apache/parquet-testing

aihuaxu · 2025-07-30T17:46:20Z

This is generated from apache/iceberg#13708 which includes the following changes on top of #90:

Add variant logical type
Update the decimal test data to align with the spec. E.g., the value with precision in [1,9] uses decimal, so 123456.7890 should uses decimal8 instead of decimal4. Update to use 12345.6789 for decimal4 test.
Mark some tests as invalid since they don't align with the spec and the readers can choose different implementation.

I have tested from Parquet-Java against this data (https://github.com/apache/parquet-java/pull/3258/files).
Here shows an example of the schema with variant logical type annotated.

Schema:
message table {
required int32 id = 1;
optional group var (VARIANT(1)) = 2 {
required binary metadata;
optional binary value;
optional group typed_value (LIST) {
repeated group list {
required group element {
optional binary value;
optional binary typed_value (STRING);
}
}
}
}
}

aihuaxu · 2025-07-30T18:09:21Z

cc @rdblue, @zeroshade. This is the same data set but I have included parquet logical type in the schema.

xxubai · 2025-08-01T09:49:40Z

Do we have any python script to generate shredding variants

aihuaxu · 2025-08-02T16:12:39Z

Do we have any python script to generate shredding variants

We don't have right now. We generated from Iceberg test cases.

xxubai · 2025-08-02T16:36:44Z

We don't have right now. We generated from Iceberg test cases.

OK. Anyway, this is very helpful for shredding variant testing in other projects.

zeroshade · 2025-08-14T23:45:59Z

@aihuaxu You updated the descriptions in shredded_variant/cases.json but the files that actually have the variant logical type in them are in data/shredded_variant/ can we consolidate to either one directory or the other?

…e notes.

alamb · 2025-08-15T15:49:08Z

My understanding is that this PR would replace #90 , is that correct?

alamb

Thank you @aihuaxu

I can't test with these values as we are still working on the shredding implementation in arrow-rs (see apache/arrow-rs#8150 and related tickets) but they look reasonable to me

I defer to @zeroshade to check with the Go implementation

cc @carpecodeum and @mprammer and @scovich

alamb · 2025-08-15T15:50:40Z

+ "parquet_file" : "case-127.parquet",
+ "error_message" : "Unsupported shredded value type: INTEGER(32,false)"
+}, {
+ "case_number" : 128,


This error message looks clear to me

aihuaxu · 2025-08-15T15:53:15Z

My understanding is that this PR would replace #90 , is that correct?

That is correct. I build on top of #90 to add variant logical type and made small changes (update decimal test cases and change test case with INVALID).

aihuaxu · 2025-08-15T22:50:11Z

@zeroshade Can you check with GO implementation? I have tested with Java implementation and it works fine.

zeroshade · 2025-08-18T15:24:38Z

Hey @aihuaxu I've filed #94 with the files generated by Go, also fixing the issues you identified.

aihuaxu · 2025-08-18T17:33:28Z

Hey @aihuaxu I've filed #94 with the files generated by Go, also fixing the issues you identified.

@zeroshade I have verified from parquet-java and it works. Thanks.

aihuaxu · 2025-08-18T17:35:56Z

@zeroshade and @zeroshade can we merge this PR? Seems we have done enough cross-language check.

carpecodeum · 2025-08-18T19:27:41Z

arrow-rs is also going ahead with these files as the core test cases - apache/arrow-rs#8104

alamb · 2025-08-18T21:08:12Z

@zeroshade and @zeroshade can we merge this PR? Seems we have done enough cross-language check.

I think it is a good idea. However, I suspect that @zeroshade may not have the permissions to do so. I think it will require someone from this group (@emkornfield @julienledem @rdblue etc)

https://people.apache.org/phonebook.html?unix=parquet

wgtmac · 2025-08-21T05:37:52Z

Thanks all for validation! Let me merge this.

alamb · 2025-08-21T19:00:59Z

Thank you @wgtmac and @aihuaxu and @zeroshade

@carpecodeum

…ing` data (#8325) # Which issue does this PR close? We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. - Closes#8084 # Rationale for this change Now that we have merged the upstream parquet-variant tests: - apache/parquet-testing#91 We can test how far we are from the rust variant implementation working for all the values This PR updates the test harness added #8104 by @carpecodeum to use the final parquet files and the currnet APIs # What changes are included in this PR? 1. Update parquet-testing pin 2. Update the test harness to use the standard rust test runner (`#[test]`) rather than a custom main function 3. Added links to follow on tickets You can run this test manually like this: ```shell cargo test --all-features --test variant_integration ... running 138 tests test test_variant_integration_case_106 ... ok test test_variant_integration_case_107 ... ok test test_variant_integration_case_109 ... ok test test_variant_integration_case_110 ... ok .. test test_variant_integration_case_90 ... ok test test_variant_integration_case_91 ... ok test test_variant_integration_case_93 ... ok test test_variant_integration_case_83 - should panic ... ok test test_variant_integration_case_84 - should panic ... ok ``` # Are these changes tested? Yes this is all tests # Are there any user-facing changes? No

rdblue added 2 commits July 23, 2025 16:42

Add shredded variant reader test cases.
31286ef

Add parquet_file for error cases.
3c86c73

aihuaxu mentioned this pull request Jul 30, 2025
ci(parquet/pqarrow): integration tests for reading shredded variants apache/arrow-go#455
Merged

aihuaxu force-pushed the variant-shared-testing branch 2 times, most recently from cbb19c1 to 560a76eCompare August 14, 2025 20:55

aihuaxu force-pushed the variant-shared-testing branch 2 times, most recently from 0af1223 to 0f5bf3bCompare August 15, 2025 03:53

Add variant logical type. Update decimal values. Add invalid test cas…
f422987
…e notes.

aihuaxu force-pushed the variant-shared-testing branch from 0f5bf3b to f422987Compare August 15, 2025 04:17

alamb reviewed Aug 15, 2025
View reviewed changes

alamb mentioned this pull request Aug 15, 2025
Add shredded Variant reader test cases #90
Closed

This was referenced Aug 16, 2025
Test Variant implementation with external test cases apache/parquet-java#3258
Closed
Generate variant tests in GO language apache/arrow-go#476
Closed

wgtmac approved these changes Aug 21, 2025
View reviewed changes

wgtmac merged commit a3d96a6 into apache:masterAug 21, 2025

zeroshade mentioned this pull request Aug 21, 2025
Shredded Variant Test files written from Go #94
Closed

alamb mentioned this pull request Sep 11, 2025
Update variant_integration test to use final approved parquet-testing data apache/arrow-rs#8325
Merged

sdf-jkl mentioned this pull request Jul 10, 2026
GH-590: Allow VARIANT value to be omitted apache/parquet-format#591
Closed

Uh oh!

Conversation

aihuaxu commented Jul 30, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aihuaxu commented Jul 30, 2025

Uh oh!

xxubai commented Aug 1, 2025

Uh oh!

aihuaxu commented Aug 2, 2025

Uh oh!

xxubai commented Aug 2, 2025

Uh oh!

zeroshade commented Aug 14, 2025

Uh oh!

alamb commented Aug 15, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambAug 15, 2025

Choose a reason for hiding this comment

Uh oh!

aihuaxu commented Aug 15, 2025

Uh oh!

aihuaxu commented Aug 15, 2025

Uh oh!

zeroshade commented Aug 18, 2025

Uh oh!

aihuaxu commented Aug 18, 2025

Uh oh!

aihuaxu commented Aug 18, 2025

Uh oh!

carpecodeum commented Aug 18, 2025

Uh oh!

alamb commented Aug 18, 2025

Uh oh!

wgtmac commented Aug 21, 2025

Uh oh!

alamb commented Aug 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

aihuaxu commented Jul 30, 2025•
edited
Loading