Skip to content

Add JSON and BSON logical type test files - #118

Merged
alamb merged 1 commit into
apache:masterfrom
nssalian:json-bson
Aug 6, 2026
Merged

Add JSON and BSON logical type test files#118
alamb merged 1 commit into
apache:masterfrom
nssalian:json-bson

Conversation

@nssalian

@nssaliannssalian commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Adds two fixtures for the JSON and BSON logical types along with the README update.

  • data/json.parquet (4 rows): {"a":1}, {"a":1,"b":null}, [1,null,3], and a NULL row.
  • data/bson.parquet (3 rows): BSON {"a":1}, {"a":1,"b":null}, and a NULL row.
  • data/README.md has a "JSON and BSON logical types"

Both are a single optional BYTE_ARRAY column and were generated by parquet-mr 1.18.0-SNAPSHOT
Verified readable by both parquet-java's reader and pyarrow, and the BSON payloads decode with a BSON library. data/README.md has a "JSON and BSON logical types" section with the generator programs and the read-back.

@nssalian

Copy link
Copy Markdown
ContributorAuthor

@emkornfield@alamb PTAL

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nssalian -- I double checked the contents of these files and they make sense to me

Comment threaddata/bson.parquet

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified the contents of this with datafusion-cli:

$ datafusion-cli -c "SELECT * from 'data/json.parquet'"
DataFusion CLI v54.1.0+------------------+
| json_field |
+------------------+
| {"a":1} |
| {"a":1,"b":null} |
| [1,null,3] |
| NULL |
+------------------+4 row(s) fetched.
Elapsed 0.039 seconds.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metadata is good too

andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/json.parquet
Metadata for file: data/json.parquet
version: 1
num of rows: 4
created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message json_fixture {
OPTIONAL BYTE_ARRAY json_field (JSON);
}

Comment threaddata/json.parquet

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contents look good:

datafusion-cli -c "SELECT * from 'data/bson.parquet'"
DataFusion CLI v54.1.0+--------------------------------+
| bson_field |
+--------------------------------+
| 0c0000001061000100000000 |
| 0f000000106100010000000a620000 |
| NULL |
+--------------------------------+3 row(s) fetched.

I decoded into bson and it matches the contents:

andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ uvrun--withpymongo--withpyarrowpython3-c "
importbson, pyarrow.parquetaspqt=pq.read_table('data/bson.parquet')
forvint.column('bson_field'):
print('NULL'ifnotv.is_validelsebson.decode(v.as_py()))
"
{'a': 1}{'a': 1, 'b': None}NULL

As does metadata

andrewlamb@Andrews-MacBook-Pro-3:~/Software/parquet-testing$ ~/Software/arrow-rs/target/release/parquet-schema data/bson.parquet
Metadata for file: data/bson.parquet
version: 1
num of rows: 3
created by: parquet-mr version 1.18.0-SNAPSHOT (build 1a9e455655604acf09cdd45b4e2958661d38281c)
metadata:
writer.model.name: example
message bson_fixture {
OPTIONAL BYTE_ARRAY bson_field (BSON);
}

@alambalamb changed the title GH-101: Add JSON and BSON logical type test files[GH-101]: Add JSON and BSON logical type test filesAug 3, 2026
@alambalamb changed the title [GH-101]: Add JSON and BSON logical type test filesAdd JSON and BSON logical type test filesAug 3, 2026
@nssalian

Copy link
Copy Markdown
ContributorAuthor

Thank you for the review and testing it out @alamb

@alamb
alamb merged commit e7bea05 into apache:masterAug 6, 2026
@alamb

alamb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Thanks again @nssalian

@nssalian
nssalian deleted the json-bson branch August 9, 2026 03:28
Jefffrey pushed a commit to apache/arrow-rs that referenced this pull request Sep 2, 2026
# Which issue does this PR close?
- Follow-up to #10786, which updated the `parquet-testing` revision.
- No tracking issue yet; one can be added if needed.
# Rationale for this change
The updated `parquet-testing` revision included:
-
[apache/parquet-testing#113](apache/parquet-testing#113):
malformed and edge-case Variant files.
-
[apache/parquet-testing#117](apache/parquet-testing#117):
mark four shredded Variant cases as invalid because they omit required
`value` columns.
-
[apache/parquet-testing#118](apache/parquet-testing#118):
JSON and BSON logical-type files.
-
[apache/parquet-testing#119](apache/parquet-testing#119):
an extended ALP fixture.
#10786 made these fixtures available to Arrow Rust, but did not exercise
the JSON, BSON, or malformed Variant files.
The same revision also added an extended ALP fixture. That fixture is
covered by the ALP encoder/decoder work in #9372, where the required
decoding support exists.
# What changes are included in this PR?
- Read the JSON logical-type fixture and verify all decoded string
values.
- Read the BSON logical-type fixture and verify all decoded binary
values.
- Validate all 14 files under `bad_data/variants`, including the one
valid duplicate-offset case and 13 malformed cases.
The four shredded Variant cases renamed as invalid by the same
`parquet-testing` update are already exercised by the existing Variant
integration harness.
# Are these changes tested?
Yes. The Parquet and Variant integration tests pass locally, along with
formatting and clippy checks. The PR's CI checks are also green.
# Are there any user-facing changes?
No. This PR only adds integration-test coverage for existing
`parquet-testing` fixtures.
# AI usage
This PR was prepared with OpenAI Codex and reviewed by a human. The
integration tests, formatting, and clippy checks described above were
run against the final branch, and the GitHub CI checks passed.
---------
Co-authored-by: cetra3 <cetra3@hotmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add examples for JSON and BSON logical types

2 participants

@nssalian@alamb