Respecting return type of __new__ does not extend to derived classes

[ikonst]

Following #7188, we can now have:

from typing import overload
from typing_extensions import Literal


class Cat:
 @overload
 def __new__(cls, name: Literal['Tabby']) -> 'Tabby': ...
 @overload
 def __new__(cls, name: Literal['Kitty']) -> 'Kitty': ...

 def __new__(cls, name: str) -> 'Cat': ...

class Kitty(Cat):
 pass

class Tabby(Cat):
 pass


reveal_type(Cat('Kitty'))
reveal_type(Cat('Tabby'))

result in:

Revealed type is 'test.Kitty'
Revealed type is 'test.Tabby'

(Meant as a trivial example.)

However, this doesn't extend to derived classes:

reveal_type(Tabby('Kitty'))
reveal_type(Tabby('Tabby'))

would result in:

Revealed type is 'test.Tabby'
Revealed type is 'test.Tabby'

I'm using mypy 0.761.

[ilevkivskyi]

This is actually intentional, because doing the opposite (as it was initially implemented) caused even more false positives.

I am not sure what to do here, cc @msullivan

[ikonst]

Curious, point to discussion / false positives?

[sbdchd]

I think I'm hitting this issue when trying to type an overloaded __new__ in a base class

My attempt overloading __new__ to match up with the untyped source:

from __future__ import annotations
from typing import Any, Dict, List, Type, TypeVar, overload
from typing_extensions import Literal

T = TypeVar("T", bound="MiniBaseSerializer")


class MiniBaseSerializer:
 @overload
 def __new__(cls, *, many: Literal[True]) -> MiniListSerializer:
 ...

 @overload
 def __new__(cls: Type[T], *, many: Literal[False] = ...) -> T:
 ...

 def __new__(cls: Type[T], many: bool = ...) -> Any:
 ...


class MiniListSerializer(MiniBaseSerializer):
 @property
 def validated_data(self) -> List[Dict[str, object]]:
 ...


class MiniSerializer(MiniBaseSerializer):
 @property
 def validated_data(self) -> Dict[str, object]:
 ...


class ExampleSerializer(MiniSerializer):
 # define some fields
 ...


example_many = ExampleSerializer(many=True)
reveal_type(example_many)
# pyright: Type of "example_many" is "MiniListSerializer"
# mypy: Revealed type is 'test_foo.ExampleSerializer'


example_single = ExampleSerializer()
reveal_type(example_single)
# pyright: Type of "example_many" is "MiniListSerializer"
# mypy: Revealed type is 'test_foo.ExampleSerializer*'

It works in pyright, but mypy forgets about the __new__ in the base class

I then tried overloading __init__ and adding a generic, which sort of works, but would require any users to provide a generic type param and it is also kind of lying about what is happening since the actual source code is returning a different type by overloading __new__

from __future__ import annotations
from typing import Dict, Generic, List, TypeVar, overload
from typing_extensions import Literal


T = TypeVar("T")


class MiniBaseSerializer(Generic[T]):
 @overload
 def __init__(self: MiniBaseSerializer[List[Dict[str, object]]], many: Literal[True]) -> None:
 ...

 @overload
 def __init__(self: MiniBaseSerializer[Dict[str, object]], many: Literal[False] = ...) -> None:
 ...

 def __init__(self, many: bool = False) -> None:
 ...

 @property
 def validated_data(self: MiniBaseSerializer[T]) -> T:
 ...


class MiniSerializer(Generic[T], MiniBaseSerializer[T]):
 ...


class ExampleSerializer(MiniSerializer[object]):
 ...


example_many = MiniBaseSerializer(many=True)
reveal_type(example_many)
# pyright - Type of "example_many" is "MiniBaseSerializer[List[Dict[str, object]]]"
# mypy - Revealed type is 'test_bar.MiniBaseSerializer[builtins.list[builtins.dict[builtins.str, builtins.object]]]'


example_single = MiniBaseSerializer()
reveal_type(example_single)
# pyright - Type of "example_single" is "MiniBaseSerializer[Dict[str, object]]"
# mypy - Revealed type is 'test_bar.MiniBaseSerializer[builtins.dict[builtins.str, builtins.object]]'

In search the issues, I also found #9482 which seemed related

Edit: Additionally, in the linked PR, there is:

mypy/test-data/unit/check-classes.test

Lines 6276 to 6288 in 30efa34

	from typing import TypeVar, Type
	# Check for __new__ using type vars
	TX = TypeVar('TX', bound='X')
	class X:
	def __new__(lol: Type[TX], x: int) -> TX:
	pass
	class Y(X): pass
	reveal_type(X(20)) # N: Revealed type is "__main__.X*"
	reveal_type(Y(20)) # N: Revealed type is "__main__.Y*"

which has the subclass maintaining the __new__ defined on the superclass

[Viicos]

As stated in typeddjango/djangorestframework-stubs#315, django-rest-framework is a good example that would benefit from the requested behavior.

I'm curious to hear about the false positives that this caused

[intgr]

Finally found the PR and issue that introduced this restriction:

• Only return a declared type from __new__ if it is a subtype of the class #7656
• Unexpected error when inheriting a class from Popen and overriding some method in it #7597

Since then, subprocess.Popen in typeshed no longer defines __new__ which solves the concrete example in #7597 AFAICT.

Also with PEP 673 Self type freshly merged in #11871, it should be easier to declare __new__ methods that correctly apply to subclasses as def __new__(...) -> Self, removing another reason for this restriction.

Returning non-subclasses from __new__() is a hack for sure, but as explained in #8330 (comment) and #14426 there are real-world libraries like Django REST Framework which are using this hack. And this restriction prevents accurately type hinting this library.

[ikonst]

Thanks for the thorough code archeology, @intgr!

I suppose we can try adjusting it once more and seeing what breaks in mypy-primer etc.

[Viicos]

I've tried reverting this specific check (Viicos@016abc3), but according to the mypy_primer output, this seems to break a lot of things (my fix is probably wrong)

[intgr]

I've tried reverting this specific check

Maybe you need to replace it with logic that retains the old behavior if __new__ return type was Any (e.g. untyped).

Where can I see the mypy_primer results?

[Viicos]

I ran mypy_primer locally, but I'll make the PR so that we can have the output