Uh oh!
There was an error while loading. Please reload this page.
Core, OpenAPI: Preserve avg value sizes in REST JSON - #17855
Conversation
Keep average value sizes in REST content-file JSON round trips and drop stale path-size statistics when rewrite_table_path rewrites position deletes. Generated-by: Cursor
| IntegerMap: | ||
| type: object | ||
| properties: | ||
| keys: | ||
| type: array | ||
| items: | ||
| $ref: '#/components/schemas/IntegerTypeValue' | ||
| description: "List of integer column ids for each corresponding value" | ||
| values: | ||
| type: array | ||
| items: | ||
| $ref: '#/components/schemas/IntegerTypeValue' | ||
| description: "List of integer values, matched to 'keys' by index" | ||
| example: | ||
| { | ||
| "keys": [ 1, 2 ], | ||
| "values": [ 8, 16 ] | ||
| } |
There was a problem hiding this comment.
can't we use ValueMap ? like what we use for other stats
iceberg/open-api/rest-catalog-open-api.yaml
Line 5126 in ef8a69d
There was a problem hiding this comment.
updated the OpenAPI to match v4 field statistics (content-stats + avg-value-size-in-bytes)
| avg-value-sizes: | ||
| allOf: | ||
| - $ref: '#/components/schemas/IntegerMap' | ||
| description: | ||
| Map of column id to average value size in memory (uncompressed) in | ||
| bytes over non-null values |
There was a problem hiding this comment.
would this be part of data file ? like lower / upper bound are ?
Leave REST JSON round-trip changes here; rewrite_table_path handling moves to a separate PR. Generated-by: Cursor
huan233usc
commented
Aug 28, 2026
Hi @singhpk234 for quick review -- I just realize that this PR was not marked as draft when I am double checking the generated code vs spec. Will asked for another review when I finish my self validation. Sry for confusion |
avg_value_size_in_bytes lives in v4 content_stats field structs, not a v1-v3 metrics map. REST JSON now uses content-stats keyed by field ID. Generated-by: Cursor
Use the spec wording for content_stats and avg_value_size_in_bytes. Generated-by: Cursor
Match other optional ContentFileParser fields: explicit JSON null is missing, and invalid field IDs get a contextual error. Generated-by: Cursor
Satisfy checkstyle by creating the parsed content stats map with the relocated Maps utility. Generated-by: Cursor
Generated-by: Cursor
Generated-by: Cursor
huan233usc
commented
Sep 1, 2026
Updated the PR, PTAL @singhpk234@nastra@szehon-ho Thanks |
| content_stats: dict[str, FieldStatistics] | None = Field( | ||
| None, | ||
| alias='content-stats', | ||
| description='Container struct for per-field metrics structs', |
There was a problem hiding this comment.
Could we document that content-stats keys are decimal string field IDs, ideally with an example? The parser requires numeric keys, while the OpenAPI schema currently allows arbitrary strings. Fine to address in a follow-up.
Uh oh!
There was an error while loading. Please reload this page.
szehon-ho
commented
Sep 4, 2026
Merged, thanks @huan233usc and @singhpk234 for additional early review ! |
Summary
Follow-up to #17451. Average value sizes were dropped when content files crossed the planned REST scan JSON boundary.
Preserve
avgValueSizesfor data and delete files inContentFileParserJSON round trips. REST JSON follows v4 field statistics: optionalcontent-statskeyed by field ID, withavg-value-size-in-byteson each per-field struct. Document the same shape in the REST OpenAPI schema.This is file-level REST metadata only. It does not add manifest-level aggregation of average sizes.
Test plan
./gradlew :iceberg-core:test --tests org.apache.iceberg.TestContentFileParser --tests org.apache.iceberg.TestFileScanTaskParser --tests org.apache.iceberg.TestDataTaskParser./gradlew :iceberg-core:spotlessCheckAI Disclosure