[Python] Inconsistent cast behavior between array and scalar for int64

[rohanjain101]

Describe the bug, including details regarding any error messages, version, and platform.

>>> scal = pa.scalar(6312878760374611856, type=pa.int64())
>>> scal.cast(pa.float64())
<pyarrow.DoubleScalar: 6.312878760374612e+18>
>>> arr = pa.array([6312878760374611856], type=pa.int64())
>>> arr.cast(pa.float64())
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "pyarrow\array.pxi", line 926, in pyarrow.lib.Array.cast
 File "C:\pandas2_ps_04323\lib\site-packages\pyarrow\compute.py", line 391, in cast
 return call_function("cast", [arr], options)
 File "pyarrow\_compute.pyx", line 560, in pyarrow._compute.call_function
 File "pyarrow\_compute.pyx", line 355, in pyarrow._compute.Function.call
 File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
 File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 6312878760374611856 not in range: -9007199254740992 to 9007199254740992
>>>

Behavior is not consistent in casting between array and scalar. The array behavior of raising does not seem correct, as it seems an int64 should always be able to be casted to float64.

Component(s)

Python

[danepitkin]

Hi @rohanjain101,

I am able to reproduce this example on pyarrow v11 using macos 13.3.

What you are experiencing is the difference between safe vs unsafe casting, since the number you chose probably can not be fully represented in the new type. It is not true that all int64 values can be safely converted to float64. Due to the way precision works in floating point, there are numbers that may be skipped that could otherwise be represented by int64. See https://en.wikipedia.org/wiki/Double-precision_floating-point_format, which states: only Integers from −253 to 253 (−9,007,199,254,740,992 to 9,007,199,254,740,992) can be exactly represented. My guess is the underlying implementation enforces this hard limit, since technically I believe there are some int64 numbers that can be larger and still represented safely when cast to float64 (such as 18,014,398,509,481,984, but not 18,014,398,509,481,983).

>>> arr = pa.array([18014398509481984], type=pa.int64())
>>> arr.cast(pa.float64())
Traceback (most recent call last):
...
pyarrow.lib.ArrowInvalid: Integer value 18014398509481984 not in range: -9007199254740992 to 9007199254740992

It appears the scalar cast defaults to allow unsafe casting, while the array defaults to safe casting. You can allow unsafe casting in the array like this:

>>> arr.cast(pa.float64(), safe=False)
<pyarrow.lib.DoubleArray object at 0x126a40ee0>
[
 6.312878760374612e+18
]

There are no options to choose safe vs unsafe cast in scalar APIs at the moment. The documentation does state the scalar will perform a safe cast, though, which it is not doing: https://arrow.apache.org/docs/python/generated/pyarrow.Int64Scalar.html#pyarrow.Int64Scalar

This is either a bug in scalar safe casting or the documentation is wrong. Ideally, Scalars can also allow you to choose safe vs unsafe casting with an option. Either way, some more investigation is still needed.

[rohanjain101]

@danepitkin thank you for the clarification. In numpy however, the cast succeeds, it seems as if full value is preserved:

np.array([18014398509481984]).astype("float64")
array([1.80143985e+16])

Is their an internal difference in how double values are stored between arrow and numpy that would cause the difference?

[danepitkin]

@danepitkin thank you for the clarification. In numpy however, the cast succeeds, it seems as if full value is preserved:

np.array([18014398509481984]).astype("float64")
array([1.80143985e+16])

Is their an internal difference in how double values are stored between arrow and numpy that would cause the difference?

In your example, 18,014,398,509,481,984 can be converted to float64 safely according to the floating point specification so it is not a good example to use. Instead let's try 18,014,398,509,481,983, which is not a multiple of 2 (required by integers between 2^53 and 2^54 for safe conversion).

You will lose data in this numpy cast. (And yes, my guess is they adhere to the floating point spec slightly differently purely based on the different behavior).

>>> np.array([18014398509481983]).astype("float64").astype("int64") == 18014398509481983
array([False])

>>> np.array([18014398509481983]).astype("float64").astype("int64")
array([18014398509481984])

Numpy defaults to unsafe casting (https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html), but it seems it also doesn't perform safety checks properly all of the time.

>>> type(np.array([18014398509481984])[0])
<class 'numpy.int64'>


# Bug? No safety error for int64 -> float64
>>> np.array([18014398509481983]).astype("float64", casting="safe").astype("int64") == 18014398509481983
array([False])


# Good? Errors out on float64 -> int64, but the bug happened in the int64 -> float64 and was somehow propagated..
>>> np.array([18014398509481983]).astype("float64", casting="safe").astype("int64", casting="safe") == 18014398509481983
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: Cannot cast array data from dtype('float64') to dtype('int64') according to the rule 'safe'

[danepitkin]

For pyarrow, we should probably:

1. Allow both safe and unsafe conversion options for scalar APIs (feature)
2. Default to safe conversion for scalars, which appears is not happening (bug)
3. Look into allowing safe conversion from int <-> float for valid numbers larger than 2^53 (feature)

@AlenkaF @jorisvandenbossche what do you think?

[rohanjain101]

But in the example where the cast is safe, for 18,014,398,509,481,984, shouldn't that then succeed in pyarrow if it can be done safely? In my example, the array case is still raising even if the cast is safe. Should it only raise for 18,014,398,509,481,983?

[danepitkin]

But in the example where the cast is safe, for 18,014,398,509,481,984, shouldn't that then succeed in pyarrow if it can be done safely? In my example, the array case is still raising even if the cast is safe. Should it only raise for 18,014,398,509,481,983?

If pyarrow were to follow the floating point specification exactly, then yes it would. Right now, it seems to be a limitation of the implementation. You could argue that option (3) above should be a bug instead of a feature.

[danepitkin]

I'd recommend filing an issue with numpy about this, too:

>>> import numpy as np
>>> np.__version__
'1.24.2'

# Bug: No safety error for initial int64 -> float64 conversion
>>> np.array([18014398509481983]).astype("float64", casting="safe").astype(str)
array(['1.8014398509481984e+16'], dtype='<U32')

[danepitkin]

Thanks for raising this issue by the way. I don't think I expressed that earlier. Your contributions are appreciated!

[jorisvandenbossche]

Numpy defaults to unsafe casting (https://numpy.org/doc/stable/reference/generated/numpy.ndarray.astype.html), but it seems it also doesn't perform safety checks properly all of the time.

Sidenote: numpy doesn't really have the same concept of "safe" casting as how we use this in pyarrow. In pyarrow this depends on the values, while in numpy this is just a property of a cast between two dtypes. So to say if a cast from one dtype to another is safe or not, numpy needs to make some generalization/assumption, and so it seems it decided that casting int to float is generally safe (indeed, except for large ints) and casting floats to ints is generally not safe (indeed, except if you have rounded floats):

>>> np.can_cast(np.int64(), np.float64(), casting="safe")
True
>>> np.can_cast(np.float64(), np.int64(), casting="safe")
False

[jorisvandenbossche]

For pyarrow, we should probably:

1. Allow both safe and unsafe conversion options for scalar APIs (feature)
2. Default to safe conversion for scalars, which appears is not happening (bug)

Yes, fully agreed, I opened a separate issue for specifically this aspect: #35040

3. Look into allowing safe conversion from int <-> float for valid numbers larger than 2^53 (feature)

For checking the safety of casting int to float, we indeed use this fixed range:

arrow/cpp/src/arrow/compute/kernels/scalar_cast_numeric.cc

Lines 171 to 250 in e488942

	// ----------------------------------------------------------------------
	// Implement fast integer to floating point cast
	// These are the limits for exact representation of whole numbers in floating
	// point numbers
	template <typename T>
	struct FloatingIntegerBound {};
	template <>
	struct FloatingIntegerBound<float> {
	static const int64_t value = 1LL << 24;
	};
	template <>
	struct FloatingIntegerBound<double> {
	static const int64_t value = 1LL << 53;
	};
	template <typename InType, typename OutType, typename InT = typename InType::c_type,
	typename OutT = typename OutType::c_type,
	bool IsSigned = is_signed_integer_type<InType>::value>
	Status CheckIntegerFloatTruncateImpl(const ExecValue& input) {
	using InScalarType = typename TypeTraits<InType>::ScalarType;
	const int64_t limit = FloatingIntegerBound<OutT>::value;
	InScalarType bound_lower(IsSigned ? -limit : 0);
	InScalarType bound_upper(limit);
	return CheckIntegersInRange(input.array, bound_lower, bound_upper);
	}
	Status CheckForIntegerToFloatingTruncation(const ExecValue& input, Type::type out_type) {
	switch (input.type()->id()) {
	// Small integers are all exactly representable as whole numbers
	case Type::INT8:
	case Type::INT16:
	case Type::UINT8:
	case Type::UINT16:
	return Status::OK();
	case Type::INT32: {
	if (out_type == Type::DOUBLE) {
	return Status::OK();
	}
	return CheckIntegerFloatTruncateImpl<Int32Type, FloatType>(input);
	}
	case Type::UINT32: {
	if (out_type == Type::DOUBLE) {
	return Status::OK();
	}
	return CheckIntegerFloatTruncateImpl<UInt32Type, FloatType>(input);
	}
	case Type::INT64: {
	if (out_type == Type::FLOAT) {
	return CheckIntegerFloatTruncateImpl<Int64Type, FloatType>(input);
	} else {
	return CheckIntegerFloatTruncateImpl<Int64Type, DoubleType>(input);
	}
	}
	case Type::UINT64: {
	if (out_type == Type::FLOAT) {
	return CheckIntegerFloatTruncateImpl<UInt64Type, FloatType>(input);
	} else {
	return CheckIntegerFloatTruncateImpl<UInt64Type, DoubleType>(input);
	}
	}
	default:
	break;
	}
	DCHECK(false);
	return Status::OK();
	}
	Status CastIntegerToFloating(KernelContext* ctx, const ExecSpan& batch, ExecResult* out) {
	const auto& options = checked_cast<const CastState*>(ctx->state())->options;
	Type::type out_type = out->type()->id();
	if (!options.allow_float_truncate) {
	RETURN_NOT_OK(CheckForIntegerToFloatingTruncation(batch[0], out_type));
	}
	CastNumberToNumberUnsafe(batch[0].type()->id(), out_type, batch[0].array,
	out->array_span_mutable());
	return Status::OK();
	}

I am not fully sure this is something we should change. First, I think this is a lot simpler in implementation to just check for values within the range, compared to checking for certain integers that can still be represented as float outside of that range. But also for the user this seems easier to understand and gives more consistent behaviour? (just everything outside of that range will fail with the default safe=True, and not depending on the exact value)