Uh oh!
There was an error while loading. Please reload this page.
GH-3327: Bug fix incorrect compressed size reported by DataPageV1 - #3326
Conversation
ArnavBalyan
commented
Sep 23, 2025
cc @gszadovszky could you PTAL :) |
gszadovszky
commented
Sep 24, 2025
@ArnavBalyan, I have a feeling that this change breaks the contract of Since in parquet-cli we only want to show page statistics, I would suggest implementing a separate code path that would only read the page headers to provide the required values instead of decompressing (and decrypting) the pages for no reason. |
eefd756 to
b5d3769Compareb5d3769 to
40b20eeCompareArnavBalyan
commented
Sep 24, 2025
Updated thanks, the CLI directly reads the page header to provide the bytes. The overhead of the header read should be negligible especially for the CLI only usecase |
gszadovszky
left a comment
There was a problem hiding this comment.
Thanks, @ArnavBalyan. LGTM
Uh oh!
There was an error while loading. Please reload this page.
Rationale for this change
ColumnChunkPageReadStoresets decompressed bytes when creating datapagev1.page.getCompressedSize()ShowPagesCommandactually reports wrong compressedSize.Are these changes tested?
Are there any user-facing changes?