Skip to content

[arrow-vector] Zero-copy transfer of DecimalVector and Decimal256Vector is losing nullability information #692

Description

@bodduv

Describe the bug, including details regarding any error messages, version, and platform.

In the main branch, DecimalVector and Decimal256Vector are losing nullability information during a zero-copy transfer. It currently forces a nullable FieldType onto the "to" ValueVector.

The deficiency in the code is in DecimalVector.TransferImpl class when trying to get a transfer getTransferPair(String, BufferAllocator) creating a DecimalVector with a nullable FieldType (although the "from" ValueVector's FieldType is non-nullable):

privateclassTransferImplimplementsTransferPair {
DecimalVectorto;
publicTransferImpl(Stringref, BufferAllocatorallocator) {
to =
newDecimalVector(ref, allocator, DecimalVector.this.precision, DecimalVector.this.scale);
}

Note that all the other (at least the primitive) ValueVectors are transferred without losing nullability information by using the "from" ValueVector's field. E.g.:

privateclassTransferImplimplementsTransferPair {
BigIntVectorto;
publicTransferImpl(Stringref, BufferAllocatorallocator) {
to = newBigIntVector(ref, field.getFieldType(), allocator);
}


To reproduce the issue: add the following test in src/test/java/org/apache/arrow/vector/TestDecimalVector.java to see it fail.

@TestpublicvoidtestGetTransferPairWithoutFieldNonNullable() {
finalFieldTypedecimalNonNullableType =
newFieldType(false, newArrowType.Decimal(10, scale), null);
finalDecimalVectorfromVector =
newDecimalVector("decimal", decimalNonNullableType, allocator);
finalTransferPairtransferPair =
fromVector.getTransferPair(fromVector.getField().getName(), allocator);
finalDecimalVectortoVector = (DecimalVector) transferPair.getTo();
// These asserts failassertSame(fromVector.getField().isNullable(), toVector.getField().isNullable());
assertSame(fromVector.getField().getFieldType(), toVector.getField().getFieldType());
}

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions