Skip to content

Optimize PlainValuesReader by reading directly from ByteBuffer (12x decode speedup) #3493

Description

@iemejia

Describe the enhancement requested

PlainValuesReader (used for PLAIN-encoded INT32, INT64, FLOAT, and DOUBLE columns) currently
reads each value through a LittleEndianDataInputStream wrapper around a ByteBufferInputStream.
Per value, readInt() performs 4 separate virtual in.read() calls and manually reassembles the
result with bit shifts:

publicfinalintreadInt() throwsIOException {
intch4 = in.read();
intch3 = in.read();
intch2 = in.read();
intch1 = in.read();
if ((ch1 | ch2 | ch3 | ch4) < 0) thrownewEOFException();
return ((ch1 << 24) + (ch2 << 16) + (ch3 << 8) + (ch4 << 0));
}

The LittleEndianDataInputStream.readInt() method itself carries a TODO comment from years ago
suggesting exactly this kind of replacement (a ByteBuffer-backed implementation).

Proposal

In PlainValuesReader.initFromPage(), obtain the page data as a single contiguous ByteBuffer
via stream.slice(stream.available()) with ByteOrder.LITTLE_ENDIAN, and call the corresponding
ByteBuffer accessor directly per value:

  • readInteger()buffer.getInt()
  • readLong()buffer.getLong()
  • readFloat()buffer.getFloat()
  • readDouble()buffer.getDouble()
  • skip(n)buffer.position(buffer.position() + n * typeSize)

ByteBuffer.getInt() with the appropriate byte order is a HotSpot intrinsic that compiles to a
single unaligned load instruction on x86/ARM — no virtual dispatch, no per-byte assembly, no
checked IOException on the per-value path.

ByteBufferInputStream.slice() already handles both single-buffer (zero-copy view) and
multi-buffer (single contiguous copy) cases transparently. In practice page data is almost
always a single contiguous buffer.

Benchmark results

IntEncodingBenchmark.decodePlain (100,000 INT32 values per invocation, JMH -wi 3 -i 5 -f 1):

PatternBefore (ops/s)After (ops/s)Speedup
SEQUENTIAL92,918,2971,143,149,23512.3x
RANDOM92,126,8881,147,547,09312.5x
LOW_CARDINALITY93,005,4511,142,666,76012.3x
HIGH_CARDINALITY93,312,5961,144,681,87612.3x

The speedup is consistent across data patterns because the bottleneck is entirely in the
per-value dispatch overhead, not the data itself. All four numeric plain reader types
(int, long, float, double) benefit equally.

End-to-end on ReadBenchmarks.readFile (6-column schema, 100k rows), this contributes to the
overall read-path improvement of 5–10% measured when combined with related decode optimizations.

Side effects

After this change, LittleEndianDataInputStream has no remaining production usages in the
codebase — only one test reference (TestColumnChunkPageWriteStore) and two TODO comments.
A follow-up could remove or deprecate the class.

Validation

All 573 parquet-column tests pass with the change applied.

Component(s)

Core

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions