Skip to content

Cache hashCode() for non-reused Binary instances (huge dictionary-encode speedup) #3499

Description

@iemejia

Describe the enhancement requested

PLAIN_DICTIONARY encoding of BINARY columns repeatedly hashes Binary keys during dictionary map lookups (Object2IntMap.put / getInt calls inside DictionaryValuesWriter.PlainBinaryDictionaryValuesWriter). Each call to Binary.hashCode() recomputes the FNV-style hash byte-by-byte across the full backing buffer:

privatestaticfinalinthashCode(byte[] array, intoffset, intlength) {
intresult = 1;
for (inti = offset; i < offset + length; i++) {
byteb = array[i];
result = 31 * result + b;
}
returnresult;
}

For columns with many repeated values, the same Binary instance is hashed many times — once per dictionary lookup attempt across the entire page. As string length grows, the cost is dominated by these recomputations.

JMH (BinaryEncodingBenchmark.encodeDictionary, 100k values per invocation, JDK 18, JMH -wi 5 -i 10 -f 3) on master:

cardinalitystringLengthops/s
LOW1013.2M
LOW1003.0M
LOW1000300K
HIGH10848K
HIGH100418K
HIGH100072.5K

The 1000-byte LOW case spending 3 ms to encode 100k values is essentially all hashing work.

Proposal

Cache hashCode() per Binary instance using the java.lang.String.hashCode() idiom — a single int field with sentinel 0 meaning "not yet computed":

@OverridepublicinthashCode() {
inth = cachedHashCode;
if (h != 0) {
returnh;
}
returncacheHashCode(Binary.hashCode(value, offset, length));
}

Properties:

  • Race-safe without volatile: the value computed is a deterministic function of immutable bytes, so any two threads that race on the first call produce the same int and either ordering is correct (per java.lang.String).
  • Reused (mutable-buffer) Binary instances do not cache, preserving the current contract that mutating the backing array between calls produces a different hash.
  • Hash that genuinely equals 0 is recomputed every call — acceptably rare and still semantically correct.

Expected speedup (JMH same configuration, 30 samples per row):

cardinalitystringLengthBeforeAfterΔ
LOW1013.2M20.2M+53%
LOW1003.0M18.0M+511%
LOW1000300K21.9M+7193% (72.9x)
HIGH10848K1.34M+58%
HIGH100418K1.32M+216%
HIGH100072.5K1.30M+1687% (17.9x)

Scope

  • Single file change to parquet-column/src/main/java/org/apache/parquet/io/api/Binary.java.
  • Adds two unit tests in TestBinary for cache correctness (constant cached, reused not cached).
  • No public-API change. Adds one package-private field (transient int cachedHashCode) and one package-private helper (cacheHashCode(int)).
  • No allocation-rate change (cache is one extra int field on existing instances).

Relation

This is part of a small series of focused PRs upstreaming optimizations from work in parquet-perf. Previous PRs in the series: #3494 (PlainValuesReader), #3496 (PlainValuesWriter).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions