Update list.{count, index, remove} to accept object type

[randolf-scholz]

Partially fixes#15271

[randolf-scholz]

Hm, not exactly sure what's going on with numpy, my best guess is that since _NestedSequence is defined with (https://github.com/numpy/numpy/blob/main/numpy/_typing/_nested_sequence.py)

defcount(self, value: Any, /) ->int:
"""Return the number of occurrences of `value`."""raiseNotImplementedErrordefindex(self, value: Any, /) ->int:
"""Return the first index of `value`."""raiseNotImplementedError

then after we make list use object rather than Any, this is still a subtype but possibly no longer a "proper_subtype". This is a mypy interal distinction and likely needs to be addressed on mypys side.

EDIT: I am starting to think that this may be related to mypy interal solver semantics. we removed some constrains, so now it may find solutions for the T of a list-constructor inside nested function calls, where previously there was no solution and it defaulted to Never which then propagate throught the call stack and may change inference in some cases.

[srittau]

The primer output is not promising. The fixed # type: ignores are mostly in test code. On the other hand there seems to be some churn, and even some true positives (in colour) that aren't flagged anymore.

[randolf-scholz]

The main issue seems to be [var-annotated] issue originating from numpy:

importnumpyasnpreveal_type(np.array([]))

# on main
tmp_a.py:3: note: Revealed type is "numpy.ndarray[tuple[Any, ...], numpy.dtype[Any]]"
# PR:
tmp_a.py:3: note: Revealed type is "numpy.ndarray[tuple[Any, ...], numpy.dtype[Never]]"

[randolf-scholz]

@srittau I wouldn't call the colour results true-positives. For instance https://github.com/colour-science/colour/blob/c9cfa1f333452cc05e42f09eebd2f3dd935a5981/colour/utilities/network.py#L1756-L1773 is exactly one of the cases of a valid list.remove call that is currently a false positive.

[randolf-scholz]

OK, I think I have a repro of the numpy issue that causes the [var-annotated] errors. It has to do with the NestedSequence Protocol. Here is a minimal reproduction:

fromtypingimportAny, Protocol, reveal_typeclassNestedSeq[_T_co](Protocol):
def__getitem__(self, index: int, /) ->"_T_co | NestedSeq[_T_co]":
raiseNotImplementedErrordefcount(self, value: Any, /) ->int:
raiseNotImplementedErrordefas_nested_sequence[T](seq: NestedSeq[T], /) ->NestedSeq[T]:
returnseqreveal_type(as_nested_sequence([]))
# main: NestedSeq[Any], PR: NestedSeq[Never]

And this happens due to how mypy solves type-vars. on main, count is [T] -> int, so mypy solves T=Any, whereas on PR, count is [object] -> int, so mypy has no constraints for T and picks T=Never.

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

[jorenham]

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Sure, we could do that.

[jorenham]

Imo, this is more of a mypy problem than a typeshed problem. But it could be worked around if for example numpy used default=Any for the type-variable of _NestedSequence. cc @jorenham

Sure, we could do that.

The workaround will be present in the upcoming 2.4.5 and 2.5.0 releases.

[jorenham]

numpy 2.4.5 was just released (https://github.com/numpy/numpy/releases/tag/v2.4.5)

[randolf-scholz]

@jorenham Ugh, terribly sorry but that wasn't sufficient. mypy still resolves _ScalarT to Never in this overload:

@overloaddefarray(
object: _ArrayLike[_ScalarT],
dtype: None=None,
*,
copy: bool|_CopyMode|None=True,
order: _OrderKACF="K",
subok: bool=False,
ndmin: int=0,
ndmax: int=0,
like: _SupportsArrayFunc|None=None,
) ->NDArray[_ScalarT]: ...

because it doesn't find any constraints on _ScalarT when doing np.array([]), so mypy emits var-annotated¹.

So actually it is this typevar that needs a default (either Any or np.generic I guess).

Again, sorry for not testing this more thoroughly locally, doing the default on _NestedSequence itself in hindisght obviously only helps when we use _NestedSequence directly, but not with def [T](_NestedSequence[T]) -> Array[T] type constructors. Then we need def [T=Any](_NestedSequence[T]) -> Array[T] instead.

Footnotes

1. Which, let's be honest, probably shouldn't even be an error enabled by default. pyrights / pyrefly / ty all use list[Unknown] when doing x = [] and don't emit any errors with standard settings. ↩

[jorenham]

Ah no worries. At least we know what to put in the mypy bug report now ;)

[randolf-scholz]

@srittau I inspected the mypy-primer in more detail. Looking at the issues in detail, I believe the mypy-primer changes are positive in almost all cases.

optuna

visualization_tests/test_rank.py#L699: This error should be expected. If mypy complains on x2=[], then one should expect it to also complain on x2=np.array([])

colour

• colour/utilities/network.py:1772 This is a false positive. Here, node is an instance of PortNode and _children is annotated as list[Self], i.e. list[PortGraph], and PortGraph subclasses PortNode. Since list.remove is equality based, this does the right thing at runtime, and shows that the current annotation is too strict.

• colour/notation/munsell/centore2014.py:2053 the flagged code here is correct. The underlying issue is an incorrect annoation in _munsell_maximum_chromas_from_renotation, is annotated as returning tuple[tuple[Pair, float]] , but actually returns tuple[tuple[Triple, float]] (with Triple=tuple[float, float, float], Pair=tuple[float, float]). mypy doesn't catch this incorrect annotation because chromas = {} is not annotated inside the function.

pandas

• the var-annotated errors are expected, as with the optuna case.
• pandas/plotting/_matplotlib/core.py:1207 Not enitrely sure what happens here, didn't debug this one.

xarray

• xarray/tests/test_concat.py:112 Not exactly sure what's happening here, if required I could look into it
• xarray/tests/test_backends.py:2090 I am not entirely sure what happens here, but it seems that even on main, np.ma.array(y, mask=[True, True, False, False, False]) with y: list[float] produces a var-annotated. However, with an inlined list constructor, on main mypy thinks that the first list expression in np.ma.array([-1, -1, 10, 10.1, 10.2], mask=[True, True, False, False, False]) is a list[Any], whereas with this PR it thinks it is a list[float], which is more accurate.

scipy-stubs

• tests/linalg/test_blas.pyi:27 With the new annotations, list[str] should satisfy the _SequenceNotStr[str] Protocol, which it previously didn't. This overload and result seems more precise then bevor.
• 3 unused-ignores, nice
• tests/special/test_ufuncs.pyi:75 This is already flagged with pyright: ignore and pyrefly: ignore, so I'd assume its a mypy false-negative.

[randolf-scholz]

The pyright failures seem unrelated

[jorenham]

• tests/special/test_ufuncs.pyi:75 This is already flagged with pyright: ignore and pyrefly: ignore, so I'd assume its a mypy false-negative.

Those are indeed tests for true positives. The trick is that in scipy-stubs all type-checkers are configured to complain in case of unused ignore comments. So it's indeed a good thing that mypy now also reports an error.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

optuna (https://github.com/optuna/optuna)
+ tests/visualization_tests/test_rank.py:699: error: Need type annotation for "x2" [var-annotated]
colour (https://github.com/colour-science/colour)
- colour/utilities/network.py:1772: error: Argument 1 to "remove" of "list" has incompatible type "PortNode"; expected "Self" [arg-type]- colour/notation/munsell/centore2014.py:2053: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]" [arg-type]- colour/notation/munsell/centore2014.py:2058: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]" [arg-type]- colour/notation/munsell/centore2014.py:2065: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]" [arg-type]- colour/notation/munsell/centore2014.py:2070: error: Argument 1 to "index" of "list" has incompatible type "tuple[Any, Any, Any]"; expected "tuple[float, float]" [arg-type]- colour/plotting/models.py:1457: error: Incompatible types in assignment (expression has type "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", variable has type "ndarray[tuple[Any, ...], dtype[Any]]") [assignment]+ colour/plotting/models.py:1457: error: Incompatible types in assignment (expression has type "Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | str | _NestedSequence[complex | bytes | str]", variable has type "ndarray[Any, Any]") [assignment]
pwndbg (https://github.com/pwndbg/pwndbg)
+ pwndbg/aglib/elf.py:448: error: Unused "type: ignore" comment [unused-ignore]
dedupe (https://github.com/dedupeio/dedupe)
+ dedupe/labeler.py:410: error: List item 1 has incompatible type "Iterable[Literal[0, 1]]"; expected "_SupportsArray[dtype[signedinteger[_64Bit]]] | _NestedSequence[_SupportsArray[dtype[signedinteger[_64Bit]]]]" [list-item]
pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/dtypes/dtypes.py:983: error: Need type annotation for "np_dtype" [var-annotated]+ pandas/core/dtypes/dtypes.py:984: error: Argument 1 has incompatible type "list[dtype[Any] | ExtensionDtype]"; expected "_SupportsArray[dtype[Never]] | _NestedSequence[_SupportsArray[dtype[Never]]]" [arg-type]+ pandas/core/arrays/sparse/array.py:1526: error: Need type annotation for "sparse_values" [var-annotated]+ pandas/core/internals/managers.py:2502: error: List comprehension has incompatible type List[ndarray[tuple[Any, ...], dtype[Any]] | ExtensionArray]; expected List[_SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]]] [misc]+ pandas/plotting/_matplotlib/misc.py:170: error: Need type annotation for "s" [var-annotated]+ pandas/plotting/_matplotlib/core.py:1207: error: List item 0 has incompatible type "Number | number[Any, int | float | complex]"; expected "_SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]]" [list-item]
xarray (https://github.com/pydata/xarray)
+ xarray/tests/test_concat.py: note: In function "create_typed_datasets":+ xarray/tests/test_concat.py:112: error: Argument 1 to "reshape" has incompatible type "list[Timedelta]"; expected "_SupportsArray[dtype[Never]] | _NestedSequence[_SupportsArray[dtype[Never]]]" [arg-type]+ xarray/tests/test_concat.py: note: At top level:+ xarray/tests/test_backends.py: note: In member "test_mask_and_scale" of class "NetCDF4Base":+ xarray/tests/test_backends.py:2090: error: Need type annotation for "expected" [var-annotated]+ xarray/tests/test_backends.py: note: At top level:
scipy-stubs (https://github.com/scipy/scipy-stubs)
+ tests/linalg/test_blas.pyi:27: error: Expression is of type "list[_FortranFunction]", not "list[_FortranFunction] | _FortranFunction" [assert-type]+ tests/linalg/test__decomp_svd.pyi:72: error: Unused "type: ignore" comment [unused-ignore]+ tests/linalg/test__decomp_svd.pyi:73: error: Unused "type: ignore" comment [unused-ignore]+ tests/linalg/test__decomp_svd.pyi:74: error: Unused "type: ignore" comment [unused-ignore]+ tests/special/test_ufuncs.pyi:75: error: Argument 1 to "__call__" of "_UFunc11f" has incompatible type "list[complex]"; expected "CanArrayND[floating[_16Bit] | integer[Any] | numpy.bool[builtins.bool], tuple[Any, ...]] | SequenceND[CanArrayND[floating[_16Bit] | integer[Any] | numpy.bool[builtins.bool], tuple[Any, ...]]] | SequenceND[JustFloat | int]" [arg-type]+ tests/special/test_ufuncs.pyi:75: note: "list" is missing following "CanArrayND" protocol member:+ tests/special/test_ufuncs.pyi:75: note: __array__

[Akuli]

I think list[int].index(12.34) should be a type checker error. If the thing can fail at runtime with some float values, we shouldn't say that float is supported.

More generally, I don't think it makes sense to allow an overlapping type in these methods (let alone object). If the runtime returns a fallback or is otherwise silent, like dict.get or set.discard, that's different.

[randolf-scholz]

@Akuli There are legitimate use cases, for instance calling .index with a union type (like T | None) or supertype of the generic type.

classRGB: ...
classRGBA(RGB): ...
classDemo:
colors: list[RGBA]
defcheck_value(self, value: RGB) ->None:
ifvalueinself.colors:
position=self.colors.index(value)
...

I understand why you want list[int].index(12.34) to be a typing error, this is a code smell and likely incorrect code. But again, if we set this in stone in the stubs you deny any legitimate use case. What you want should imho be some opt-in linter rule that type checkers like mypy or ty can implement if they want. But typeshed is the wrong place for this imo.

[Akuli]

In your example, ideally the type checker would narrow the type of value into RGBA because it is in a List[RGBA]. Then the .index() would succeed as is.

You could also make the same argument with any other operation that needs an RGBA object, and I don't see how list.index() is special here. For example:

classRGB:
red: floatgreen: floatblue: float
...
classRGBA(RGB):
alpha: float
...
colors: list[RGBA] = [...]
value=get_some_rgb_color()
ifvalueincolors:
print(f"opacity is {value.alpha}")

More generally, the correct fix is to help type checkers know the correct type, not to silence errors when they have the wrong type.

To be concrete, I would be very surprised if mistakes like list[int].index(12.34) (or list[int].index("hello")) were less frequently occurring, and hence less important, than the problems that this PR solves. And as I said in your other PR, mypy_primer is not a good way to measure this because most of the code it checks already passes the type check.

[randolf-scholz]

In your example, ideally the type checker would narrow the type of value into RGBA because it is in a List[RGBA]. Then the .index() would succeed as is.

This would not be ideal, but actually incorrect as list containment is equality based. Your own example will error on the line print(f"opacity is {value.alpha}") if RGBA overwrites __eq__ in such a way that an RGB can equal an RGBA:

fromdataclassesimportdataclass@dataclass(frozen=True)classRGB:
red: intgreen: intblue: int@dataclass(frozen=True)classRGBA(RGB):
alpha: int=255def__eq__(self, other):
ifisinstance(other, RGBA):
returnhash(self) ==hash(other)
ifisinstance(other, RGB):
returnRGB.__hash__(self) ==hash(other) andself.alpha==255returnNotImplementedblack=RGB(0, 0, 0)
colors= [RGBA(0, 0, 0)]
ifblackincolors:
print(black.alpha) # ERROR at runtime

another example with pytorch:

importtorch# Note: torch.Size subclasses tuple[int, ...]sizes: list[torch.Size] = [torch.Size([1, 2, 3])]
print(sizes[0].numel()) # 6 (product of values)value= (1, 2, 3)
ifvalueinsizes:
print(value.numel()) # ERROR

[randolf-scholz]

There is actually a regression in mypy type checking here, introduced in 1.20.0 python/mypy#21512

[Akuli]

I see. There might not be a way to express this so that overrided __eq__ methods and the error for list[int].index(float) both work. But even then, falling back to object is just not the way to go.

There are probably many other places too where overrided __eq__ methods behave weirdly.

[randolf-scholz]

Again I think what you want is a fine strict linter rule to have, but you seem very eager to force this linter rule on everyone else, even when it caues false positives in their code.