Uh oh!
There was an error while loading. Please reload this page.
GH-112 Harden variant decoding: example data - #113
Conversation
Test files are added to parquet-format project with commentary.
e1eff0f to
6333d3fCompareAll parquet readers must reject this.
...from discussion in the iceberg PR.
| Robust implementations of variant decoders SHOULD reject these. | ||
| | File | Malformed Structure | | ||
| |---------------------------------------------------------------|----------------------------------------------------------------------------| |
There was a problem hiding this comment.
Would be great to add the sample code that helped generate these individual cases in the README as well.
There was a problem hiding this comment.
until the relevant PR is merged, that code doesn't really exist, does it?
There was a problem hiding this comment.
Agreed but those are in the tests of the Java repo. The data/README here has a snippet of the code that was used to generate the files. I'd prefer that be here rather than link to another repo. I'm okay reusing the same code but having it here is better for users to reproduce the artifacts. If you are planning to do that after the code merges, that's great.
There was a problem hiding this comment.
I think it is ok not to include reproducer of malformed data -- a link to the relevant PR (or perhaps a gist) would be enough
There was a problem hiding this comment.
Looks good to me -- thank you @steveloughran@laskoviymishka and @nssalian
steveloughran
commented
Aug 12, 2026
Thanks @alamb...can someone with merge permissions merge this? The cross-project pr ref can go in the commit message. |
alamb
commented
Aug 12, 2026
🚀 |
# Which issue does this PR close? - Follow-up to #10786, which updated the `parquet-testing` revision. - No tracking issue yet; one can be added if needed. # Rationale for this change The updated `parquet-testing` revision included: - [apache/parquet-testing#113](apache/parquet-testing#113): malformed and edge-case Variant files. - [apache/parquet-testing#117](apache/parquet-testing#117): mark four shredded Variant cases as invalid because they omit required `value` columns. - [apache/parquet-testing#118](apache/parquet-testing#118): JSON and BSON logical-type files. - [apache/parquet-testing#119](apache/parquet-testing#119): an extended ALP fixture. #10786 made these fixtures available to Arrow Rust, but did not exercise the JSON, BSON, or malformed Variant files. The same revision also added an extended ALP fixture. That fixture is covered by the ALP encoder/decoder work in #9372, where the required decoding support exists. # What changes are included in this PR? - Read the JSON logical-type fixture and verify all decoded string values. - Read the BSON logical-type fixture and verify all decoded binary values. - Validate all 14 files under `bad_data/variants`, including the one valid duplicate-offset case and 13 malformed cases. The four shredded Variant cases renamed as invalid by the same `parquet-testing` update are already exercised by the existing Variant integration harness. # Are these changes tested? Yes. The Parquet and Variant integration tests pass locally, along with formatting and clippy checks. The PR's CI checks are also green. # Are there any user-facing changes? No. This PR only adds integration-test coverage for existing `parquet-testing` fixtures. # AI usage This PR was prepared with OpenAI Codex and reviewed by a human. The integration tests, formatting, and clippy checks described above were run against the final branch, and the GitHub CI checks passed. --------- Co-authored-by: cetra3 <cetra3@hotmail.com>
Add parquet files with malformed variant data.
This is generated from the tests of apache/parquet-java#3562 when configured to not delete output during test runs