DOC: numpy.result_type promotion "algorithm" is outdated

[MustafaYounes1]

Describe the issue:

According to the API Reference, the algorithm of numpy.result_type determines the output type following these steps:

1. Categories are determined by first checking which of boolean, integer (int/uint), or floating point (float/complex) the maximum kind of all the arrays and the scalars are.

2. If there are only scalars or the maximum category of the scalars is higher than the maximum category of the arrays, the data types are combined with promote_types to produce the return value.

3. Otherwise, min_scalar_type is called on each scalar, and the resulting data types are all combined with promote_types to produce the return value.

However, when it's provided with a Python scalar and an ndarray of the same category (e.g., int), it seems it completely ignores the scalar and outputs whatever datatype the ndarray has (and that contradicts the third point above; according to the doc, it should apply min_scalar_type on the scalar, and follow that with a call to promote_types).

However, when the provided scalar is a NumPy scalar, the behavior is different: what it seemingly does is only a call to promote_types

So, I'm wondering why the behavior is inconsistent on NumPy/Python scalars, and what exactly the role of min_scalar_type is in this context?

Please take a look at the provided code that shows this unexpected behaviour with a couple of examples.

Reproduce the code example:

import numpy as np

#############
# Example 1 #
#############

# Both the ndarray and the scalar are of the same category (int)

s = -129
a = np.empty(1, dtype='i1')

# Python Scalar & ndarray

# The following outputs 'int8' (completely ignoring the scalar)
print(np.result_type(s, a))

# according to the doc; it should have called:
print(np.promote_types(np.min_scalar_type(s), a.dtype)) # gives int16

# NumPy Scalar & ndarray

# gives 'int16' (seemingly only calls np.promote_types)
print(np.result_type(np.int16(s), a)) 
 
#############
# Example 2 #
#############

# The ndarray has a higher category (float vs int)

s = np.iinfo(np.int64).max
a = np.empty(1, np.float32)

# Python Scalar & ndarray

# The following outputs 'float32' (completely ignoring the scalar)
print(np.result_type(s, a))

# according to the doc; it should have called:
print(np.promote_types(np.min_scalar_type(s), a.dtype)) # gives float64

# NumPy Scalar & ndarray

# gives 'float64' (seemingly only calls np.promote_types)
print(np.result_type(np.int64(1), a))

# Pay attention to the following: (np.min_scalar_type is not used at all)
print(np.min_scalar_type(np.int64(1))) # gives uint8
print(np.promote_types(np.min_scalar_type(np.int64(1)), a.dtype)) # gives float32

Python and NumPy Versions:

Python: 3.11.5
NumPy: 2.3.1

[jakevdp]

However, when it's provided with a Python scalar and an ndarray of the same category (e.g., int), it seems it completely ignores the scalar and outputs whatever datatype the ndarray has

I believe what you describe is the expected behavior in NumPy 2.0. See NumPy 2.0 migration guide: Changes to NumPy data type promotion for details on this.

It may be that the docs you are looking at predate NumPy 2.0 and need to be updated.

[MustafaYounes1]

@jakevdp Thanks for the info!

I took a glance at the relevant NEP. It has a lot of information about old behavior vs new behavior, issues, reasoning, related work, etc. (An excellent reference for backwards compatibility)

However, I do agree with you and believe that it's crucial for the API reference to have a sort of brief explanation of the current state of NumPy 2.0's new promotion rules.

I presume this issue shouldn't be closed for now.

[seberg]

Agreed thanks for noticing. The documentation needs updating it is completely incorrect now. We have a new overview page for promotion (outside of the NEP), unless I am very mistaken. The best thing might be to just link to there.

If there is something to link, then this is a good small issue to just attack (otherwise it's OK, but formulating this stuff right tends to be tricky).

[Alverok]

Hi everyone, I tried to do my homework around this issue and would like to help!

What I have understood so far is that the result_type docstring needs an update - removing the old, incorrect algorithm and replacing it with a brief summary of the new NEP 50 rules, along with a link to the NumPy 2.0 Migration Guide as mentioned by @MustafaYounes1 and @jakevdp.

I tried to locate the result_type docstring in numpy/_core/multiarray.py, and for consistency, I thought why not apply the same update to the promote_types docstring - but I was unable to find its definition in the same file. I tried searching it in numpy\core\_multiarray_umath.py too, but couldn't, though.

Before I open a pull request, could you please confirm the best way to proceed? Should I submit a PR that only updates result_type, or could you point me to the correct location for the promote_types docstring so I can update both or apparently ignore the latter if it doesn't need to be updated?

Thanks for your help!

[MustafaYounes1]

@Alverok You can find the docstring of np.promote_types here

While I believe that every single contribution would be highly appreciated, I think that what @seberg rightly pointed out, formulating accurate and comprehensive documentation for the type promotion rules, is a nuanced task.

Given the complexity and the potential for edge cases, it may be beneficial to consult with the contributors who were directly involved in the NEP 50 implementation. They can provide the most authoritative guidance to ensure the documentation is both clear and technically precise.

PS: @seberg mentioned a page that thoroughly explains the new rules outside of the NEP, which could also be helpful (if it ever exists)

[mattip]

Closed by #30199

[mdhaber]

Notable behavior with Python builtin type classes acknowledges the difference between using result_type with Python scalar values (e.g. 1) vs Python scalar types (e.g. int) and explains "why" at a surface level:

The reason for this behavior is that NumPy converts int its default integer type, and uses that type for promotion

But why does it have to behave that way?

I suppose it doesn't deviate from the letter of the array API standard because the standard doesn't require that result_type accept Python scalar types. But if result_type is supposed to predict the result of an operation between values of the provided types/dtypes,

import numpy as np
np.result_type(np.int32(1), int(1)) # dtype('int32')
np.result_type(np.int32, int) # dtype('int64')
np.result_type(np.float32(1), float(1)) # dtype('float32')
np.result_type(np.float32, float) # dtype('float64')

seems incorrect.

I'd speculate that the NumPy behavior is just an accidental holdout from pre-NEP 50 promotion rules, since before that:, both:

np.result_type(xp.asarray(1), float) # dtype('float64')
np.result_type(np.asarray(1), 1.0) # dtype('float64')

so the behaviors should have both changed at that time.
As such, I think there's still time to label it a bug, not a feature.