Uh oh!
There was an error while loading. Please reload this page.
GH-709: Correct length calculation of value buffers of variable-sized arrays - #707
Merged
Conversation
pepijnve
requested review from
jbonofre, laurentgo, lidavidm and wgtmac
as code ownersApril 10, 2025 14:50
This comment has been minimized.
This comment has been minimized.
pepijnve
commented
Apr 10, 2025
ContributorAuthor
This PR should have the 'bug-fix' label, but I don't seem to be able to apply that myself. |
lidavidm
commented
Apr 10, 2025
Member
arrow-java doesn't support slicing in general, so for offset-based arrays, other code downstream may not work properly if the first offset is nonzero. I think the longer term fix is to either detect this and copy or decide to properly implement slicing. That said fixing an out-of-bounds is always good. |
lidavidm
approved these changes
Apr 11, 2025
lidavidm
left a comment
Member
There was a problem hiding this comment.
LGTM.
Do you mind opening a new issue to link this PR to? The original issue has more discussion. In particular I think the cross-language integration tests need to be improved to cover this case
pepijnve
commented
Apr 11, 2025
ContributorAuthor
I've created a new issue and update the summary and description of this PR. |
lidavidm
commented
Apr 14, 2025
Member
Thanks, the CI failures here should not be related but let me dig into what's going on - I may ask you to rebase |
dongjoon-hyun pushed a commit
to apache/spark
that referenced
this pull request
May 15, 2025
### What changes were proposed in this pull request? This pr aims to upgrade `arrow-java` from 18.2.0 to 18.3.0. ### Why are the changes needed? The new version bring some bug fixes, like: - apache/arrow-java#627 - apache/arrow-java#654 - apache/arrow-java#656 - apache/arrow-java#693 - apache/arrow-java#705 - apache/arrow-java#707 - apache/arrow-java#722 In addition, the new version introduces a cascading upgrade for flatbuffers-java([ from 24.3.25 to 25.1.24 ](apache/arrow-java#600)) the full release note as follows: - https://github.com/apache/arrow-java/releases/tag/v18.3.0 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Acitons ### Was this patch authored or co-authored using generative AI tooling? No Closes#50892 from LuciferYang/arrow-java-18.3.0. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
timhurskidremio pushed a commit
to timhurskidremio/dremio-arrow-java
that referenced
this pull request
Dec 5, 2025
…-sized arrays (apache#707) ## What's Changed For variable-size binary layout arrays, BufferImportTypeVisitor currently derives the length of the value buffer by calculating the difference between the last and first offset. When the first offset is not zero, this is actually incorrect and leads to out of bounds errors when attempting to read values from the imported array. Instead, BufferImportTypeVisitor should simply use the last offset value as the length of the value buffer. This PR makes that change. Just FYI, I bumped into this issue when attempting to import an array originating from DataFusion. A test query of the form `SELECT column1 FROM VALUES ('a'), ('b'), ('c'), ('d') LIMIT 2 OFFSET 1;` returns a slice of the full set of values. The values buffer contains all the original values, and the offsets buffer contains 1 and 2 as values to handle the offset from the query. Closesapache#709.
This was referenced Jul 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's Changed
For variable-size binary layout arrays, BufferImportTypeVisitor currently derives the length of the value buffer by calculating the difference between the last and first offset. When the first offset is not zero, this is actually incorrect and leads to out of bounds errors when attempting to read values from the imported array.
Instead, BufferImportTypeVisitor should simply use the last offset value as the length of the value buffer. This PR makes that change.
Just FYI, I bumped into this issue when attempting to import an array originating from DataFusion. A test query of the form
SELECT column1 FROM VALUES ('a'), ('b'), ('c'), ('d') LIMIT 2 OFFSET 1;returns a slice of the full set of values. The values buffer contains all the original values, and the offsets buffer contains 1 and 2 as values to handle the offset from the query.Closes#709 .