Uh oh!
There was an error while loading. Please reload this page.
Add JSON and BSON logical type test files - #118
Merged
Conversation
nssalian
commented
Jul 30, 2026
ContributorAuthor
@emkornfield@alamb PTAL |
alamb
approved these changes
Aug 3, 2026
Contributor
There was a problem hiding this comment.
I verified the contents of this with datafusion-cli:
$ datafusion-cli -c "SELECT * from 'data/json.parquet'"
DataFusion CLI v54.1.0+------------------+
| json_field |
+------------------+
| {"a":1} |
| {"a":1,"b":null} |
| [1,null,3] |
| NULL |
+------------------+4 row(s) fetched.
Elapsed 0.039 seconds.Contributor
There was a problem hiding this comment.
Metadata is good too
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/json.parquet
Metadata for file: data/json.parquet
version: 1
num of rows: 4
created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message json_fixture {
OPTIONAL BYTE_ARRAY json_field (JSON);
}Contributor
There was a problem hiding this comment.
Contents look good:
datafusion-cli -c "SELECT * from 'data/bson.parquet'"
DataFusion CLI v54.1.0+--------------------------------+
| bson_field |
+--------------------------------+
| 0c0000001061000100000000 |
| 0f000000106100010000000a620000 |
| NULL |
+--------------------------------+3 row(s) fetched.I decoded into bson and it matches the contents:
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ uvrun--withpymongo--withpyarrowpython3-c "
importbson, pyarrow.parquetaspqt=pq.read_table('data/bson.parquet')
forvint.column('bson_field'):
print('NULL'ifnotv.is_validelsebson.decode(v.as_py()))
"
{'a': 1}{'a': 1, 'b': None}NULLAs does metadata
andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/bson.parquet
Metadata for file: data/bson.parquet
version: 1
num of rows: 3
created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message bson_fixture {
OPTIONAL BYTE_ARRAY bson_field (BSON);
}nssalian
commented
Aug 3, 2026
ContributorAuthor
Thank you for the review and testing it out @alamb |
alamb
commented
Aug 6, 2026
Contributor
Thanks again @nssalian |
Jefffrey pushed a commit
to apache/arrow-rs
that referenced
this pull request
Sep 2, 2026
# Which issue does this PR close? - Follow-up to #10786, which updated the `parquet-testing` revision. - No tracking issue yet; one can be added if needed. # Rationale for this change The updated `parquet-testing` revision included: - [apache/parquet-testing#113](apache/parquet-testing#113): malformed and edge-case Variant files. - [apache/parquet-testing#117](apache/parquet-testing#117): mark four shredded Variant cases as invalid because they omit required `value` columns. - [apache/parquet-testing#118](apache/parquet-testing#118): JSON and BSON logical-type files. - [apache/parquet-testing#119](apache/parquet-testing#119): an extended ALP fixture. #10786 made these fixtures available to Arrow Rust, but did not exercise the JSON, BSON, or malformed Variant files. The same revision also added an extended ALP fixture. That fixture is covered by the ALP encoder/decoder work in #9372, where the required decoding support exists. # What changes are included in this PR? - Read the JSON logical-type fixture and verify all decoded string values. - Read the BSON logical-type fixture and verify all decoded binary values. - Validate all 14 files under `bad_data/variants`, including the one valid duplicate-offset case and 13 malformed cases. The four shredded Variant cases renamed as invalid by the same `parquet-testing` update are already exercised by the existing Variant integration harness. # Are these changes tested? Yes. The Parquet and Variant integration tests pass locally, along with formatting and clippy checks. The PR's CI checks are also green. # Are there any user-facing changes? No. This PR only adds integration-test coverage for existing `parquet-testing` fixtures. # AI usage This PR was prepared with OpenAI Codex and reviewed by a human. The integration tests, formatting, and clippy checks described above were run against the final branch, and the GitHub CI checks passed. --------- Co-authored-by: cetra3 <cetra3@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two fixtures for the JSON and BSON logical types along with the README update.
{"a":1}, {"a":1,"b":null}, [1,null,3], and a NULL row.{"a":1}, {"a":1,"b":null}, and a NULL row.Both are a single optional BYTE_ARRAY column and were generated by parquet-mr 1.18.0-SNAPSHOT
Verified readable by both parquet-java's reader and pyarrow, and the BSON payloads decode with a BSON library. data/README.md has a "JSON and BSON logical types" section with the generator programs and the read-back.