Describe the bug, including details regarding any error messages, version, and platform.
In the main branch, DecimalVector and Decimal256Vector are losing nullability information during a zero-copy transfer. It currently forces a nullable FieldType onto the "to" ValueVector.
The deficiency in the code is in DecimalVector.TransferImpl class when trying to get a transfer getTransferPair(String, BufferAllocator) creating a DecimalVector with a nullable FieldType (although the "from" ValueVector's FieldType is non-nullable):
| privateclassTransferImplimplementsTransferPair { |
| DecimalVectorto; |
| |
| publicTransferImpl(Stringref, BufferAllocatorallocator) { |
| to = |
| newDecimalVector(ref, allocator, DecimalVector.this.precision, DecimalVector.this.scale); |
| } |
Note that all the other (at least the primitive) ValueVectors are transferred without losing nullability information by using the "from" ValueVector's field. E.g.:
| privateclassTransferImplimplementsTransferPair { |
| BigIntVectorto; |
| |
| publicTransferImpl(Stringref, BufferAllocatorallocator) { |
| to = newBigIntVector(ref, field.getFieldType(), allocator); |
| } |
To reproduce the issue: add the following test in src/test/java/org/apache/arrow/vector/TestDecimalVector.java to see it fail.
@TestpublicvoidtestGetTransferPairWithoutFieldNonNullable() {
finalFieldTypedecimalNonNullableType =
newFieldType(false, newArrowType.Decimal(10, scale), null);
finalDecimalVectorfromVector =
newDecimalVector("decimal", decimalNonNullableType, allocator);
finalTransferPairtransferPair =
fromVector.getTransferPair(fromVector.getField().getName(), allocator);
finalDecimalVectortoVector = (DecimalVector) transferPair.getTo();
// These asserts failassertSame(fromVector.getField().isNullable(), toVector.getField().isNullable());
assertSame(fromVector.getField().getFieldType(), toVector.getField().getFieldType());
}
Describe the bug, including details regarding any error messages, version, and platform.
In the main branch,
DecimalVectorandDecimal256Vectorare losing nullability information during a zero-copy transfer. It currently forces a nullableFieldTypeonto the "to"ValueVector.The deficiency in the code is in
DecimalVector.TransferImplclass when trying to get a transfergetTransferPair(String, BufferAllocator)creating aDecimalVectorwith a nullableFieldType(although the "from" ValueVector's FieldType is non-nullable):arrow-java/vector/src/main/java/org/apache/arrow/vector/DecimalVector.java
Lines 563 to 569 in 48a20ba
Note that all the other (at least the primitive)
ValueVectors are transferred without losing nullability information by using the "from"ValueVector's field. E.g.:arrow-java/vector/src/main/java/org/apache/arrow/vector/BigIntVector.java
Lines 319 to 324 in 48a20ba
To reproduce the issue: add the following test in
src/test/java/org/apache/arrow/vector/TestDecimalVector.javato see it fail.