Uh oh!
There was an error while loading. Please reload this page.
GH-3522: Eliminate unnecessary page-size copies in compressed page assembly and CRC checksums - #3536
GH-3522: Eliminate unnecessary page-size copies in compressed page assembly and CRC checksums#3536iemejia wants to merge 1 commit into
Conversation
…age assembly and CRC checksums Two sources of unnecessary full-page-size byte[] allocations: 1. BAOSBytesInput.writeInto(ByteBuffer) called toByteArray() which copies the entire internal buffer. Replace with writeTo() using a thin ByteBufferBackedOutputStream adapter that writes directly from the internal buf[] without allocation. 2. CRC32 checksum computation in ColumnChunkPageWriteStore called toByteArray() on each BytesInput to pass to crc.update(byte[]). Replace with writeAllTo(CRC32OutputStream) that streams bytes directly to crc.update() without intermediate copies. For a typical 1MB page, this eliminates 1-3 full-page allocations per page during write (one per CRC computation + one for the ByteBuffer assembly).
006519e to
1032712Compareiemejia
commented
May 17, 2026
Closing — this will be superseded by a forthcoming PR for column I/O and page assembly optimizations (v2-par7-column-io), which is not yet opened because it depends on #3565 (PLAIN) and #3568 (RLE) being merged first. I initially submitted a series of small, focused PRs thinking they'd be easier to review. In practice the sheer number (~16 PRs, with more pending) made things harder to follow — even for me. I've regrouped the changes by encoding type / performance area so that each PR is self-contained with its own benchmarks and test coverage, which should make review and performance analysis much more straightforward. Apologies for the churn. The replacement PR will be opened once its dependencies land. Thank you. |
Summary
toByteArray()copy inBAOSBytesInput.writeInto(ByteBuffer)by streaming through aByteBufferBackedOutputStreamadaptertoByteArray()copy in CRC32 checksum computation by streaming through aCRC32OutputStreamadapterDetails
Two sources of unnecessary full-page-size
byte[]allocations:BAOSBytesInput.writeInto(ByteBuffer)calledtoByteArray()which copies the entire internal buffer. Replace withwriteTo()using a thinByteBufferBackedOutputStreamadapter that writes directly from the internalbuf[]without allocation.CRC32 checksum computation in
ColumnChunkPageWriteStorecalledtoByteArray()on eachBytesInputto pass tocrc.update(byte[]). Replace withwriteAllTo(CRC32OutputStream)that streams bytes directly tocrc.update()without intermediate copies.For a typical 1MB page, this eliminates 1-3 full-page allocations per page during write (one per CRC computation + one for the ByteBuffer assembly). The benefit is reduced GC pressure and peak memory rather than steady-state throughput.
All TestBytesInput tests pass. Compiles clean.