Skip to content

AVRO-4267: [java] Compare bytes as unsigned - #3972

Open
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:avro-4267-unsigned-order
Open

AVRO-4267: [java] Compare bytes as unsigned#3972
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:avro-4267-unsigned-order

Conversation

@1fanwang

Copy link
Copy Markdown

What is the purpose of the change

Java users who sort generic Avro records containing bytes can get a different order before and after serialization. Before this change, 0x7f sorted after 0xff in memory but before it in encoded data. After this change, both paths use Avro's unsigned byte order.

GenericData.compare now compares byte buffers over their remaining ranges with unsigned values and absolute reads. This preserves buffer positions and works with heap, sliced, read-only, and direct buffers. Converted logical types keep the existing Comparable behavior.

Fixes https://issues.apache.org/jira/browse/AVRO-4267

Verifying this change

This change adds coverage for unsigned values, prefixes, nonzero positions, sliced buffers, read-only buffers, direct buffers, position preservation, and converted logical types.

Raw logs

Before, on a9b88f538c74d2347444fac60d7289abcf231a50 with only the regression test applied:

$ JAVA_HOME=$(/usr/libexec/java_home -v 21) /opt/homebrew/bin/mvn -pl lang/java/avro -Dtest=TestCompare#genericBytesCompareMatchesBinaryUnsignedByteOrder test -DskipITs -DskipTests=false[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0[ERROR] TestCompare.genericBytesCompareMatchesBinaryUnsignedByteOrder:75 expected: <true> but was: <false>[INFO] BUILD FAILURE

After:

$ JAVA_HOME=$(/usr/libexec/java_home -v 21) /opt/homebrew/bin/mvn -Dmaven.build.cache.enabled=false -pl lang/java/avro clean test -DskipITs[INFO] Spotless.Java is keeping 315 files clean - 0 needs changes to be clean, 315 were already clean, 0 were skipped because caching determined they were already clean[INFO] Tests run: 3463, Failures: 0, Errors: 0, Skipped: 2[INFO] You have 0 Checkstyle violations.[INFO] Tests run: 3463, Failures: 0, Errors: 0, Skipped: 2[INFO] Tests run: 3463, Failures: 0, Errors: 0, Skipped: 2[INFO] BUILD SUCCESS

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

GenericData used Java signed-byte ordering for byte buffers while BinaryData followed the Avro unsigned-byte order. The same values could therefore sort differently in memory and after encoding.
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@github-actionsgithub-actionsBot added the Java Pull Requests for Java binding label Aug 25, 2026

@prashantpiyush1111prashantpiyush1111 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Verified compareByteBuffers() — uses absolute get(index) reads so buffer positions are preserved (heap/sliced/direct/read-only, matches the tests), and the unsigned-compare logic correctly mirrors BinaryData.compareBytes(). Logical types (e.g. decimal) still fall through to the old Comparable path via the instanceof ByteBuffer guard. Good test coverage. No concerns, thanks!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

JavaPull Requests for Java binding

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@1fanwang@prashantpiyush1111