Use typing_extensions.Self in the stdlib

[AlexWaygood]

Only doing the stdlib for now, so that we don't have a repeat of #9690.

The first commit in this PR was prepared using the following script, which uses libcst:

Details

frompathlibimportPathimportlibcstclassModuleFixer(libcst.CSTTransformer):
defleave_ImportFrom(self, original_node: libcst.ImportFrom, updated_node: libcst.ImportFrom) ->libcst.ImportFrom:
ifnot (isinstance(original_node.module, libcst.Name) andoriginal_node.module.value=="_typeshed"):
returnupdated_nodeifisinstance(original_node.names, libcst.ImportStar):
returnupdated_nodenew_imports= [
itemforiteminoriginal_node.namesifnot (isinstance(item, libcst.ImportAlias) andisinstance(item.name, libcst.Name) anditem.name.value=="Self")
]
iflen(original_node.names) ==2andlen(new_imports) ==1:
new_imports[0] =new_imports[0].with_changes(comma=libcst.MaybeSentinel.DEFAULT)
returnupdated_node.with_changes(names=new_imports)
defleave_Parameters(self, original_node: libcst.Parameters, updated_node: libcst.Parameters) ->libcst.Parameters:
ifnotoriginal_node.params:
returnupdated_nodefirst_param=original_node.params[0]
ifnot (
isinstance(first_param, libcst.Param)
andisinstance(first_param.annotation, libcst.Annotation)
):
returnupdated_nodefirst_annotation=first_param.annotation.annotationifisinstance(first_annotation, libcst.Name):
iffirst_annotation.value!="Self":
returnupdated_nodeelifisinstance(first_annotation, libcst.Subscript):
ifnot (
isinstance(first_annotation.value, libcst.Name)
andfirst_annotation.value.value=="type"andlen(first_annotation.slice) ==1andisinstance(first_annotation.slice[0], libcst.SubscriptElement)
andisinstance(first_annotation.slice[0].slice, libcst.Index)
andisinstance(first_annotation.slice[0].slice.value, libcst.Name)
andfirst_annotation.slice[0].slice.value.value=="Self"
):
returnupdated_nodeelse:
returnupdated_nodenew_params=list(updated_node.params)
new_params[0] =first_param.with_changes(annotation=None)
returnupdated_node.with_changes(params=new_params)
defchange_self_annotations(path: Path) ->None:
source=path.read_text()
ifnotsource:
returnif"from _typeshed import"notinsource:
returnifnot (": Self"insourceor": type[Self]"insource):
returnsourcelines=source.splitlines()
try:
sourcelines.remove("from _typeshed import Self")
exceptValueError:
passfirst_non_comment_lineno=next(ifori, lineinenumerate(sourcelines) ifline.strip() andnotline.strip().startswith("#"))
sourcelines.insert(first_non_comment_lineno, "from typing_extensions import Self")
cst=libcst.parse_module("\n".join(sourcelines) +"\n")
new_source=cst.visit(ModuleFixer()).codepath.write_text(new_source)
defmain() ->None:
forpathinPath("stdlib").rglob("*.pyi"):
print(f"Fixing '{path}'...")
change_self_annotations(path)
if__name__=="__main__":
main()

Cf. #6300

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ bson/son.py: note: In member "__new__" of class "SON":+ bson/son.py:69: error: Value of type variable "Self" of "__new__" of "dict" cannot be "SON[_Key, _Value]" [type-var]
prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/pydantic.py:161: note: def [Self] __new__(cls: Type[Self], str, Tuple[type, ...], Dict[str, Any], /, **kwds: Any) -> Self+ src/prefect/utilities/pydantic.py:161: note: def [_typeshed.Self] __new__(cls: Type[_typeshed.Self], str, Tuple[type, ...], Dict[str, Any], /, **kwds: Any) -> _typeshed.Self- src/prefect/utilities/pydantic.py:163: note: def [Self] __new__(cls: Type[Self], str, Tuple[type, ...], Dict[str, Any], /, **kwds: Any) -> Self+ src/prefect/utilities/pydantic.py:163: note: def [_typeshed.Self] __new__(cls: Type[_typeshed.Self], str, Tuple[type, ...], Dict[str, Any], /, **kwds: Any) -> _typeshed.Self
steam.py (https://github.com/Gobot1234/steam.py)
- steam/chat.py:401: note: def [Self] __new__(cls: Type[Self], str, Tuple[type, ...], Dict[str, Any], /, **kwds: Any) -> Self+ steam/chat.py:401: note: def [_typeshed.Self] __new__(cls: Type[_typeshed.Self], str, Tuple[type, ...], Dict[str, Any], /, **kwds: Any) -> _typeshed.Self

[AlexWaygood]

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ bson/son.py: note: In member "__new__" of class "SON":+ bson/son.py:69: error: Value of type variable "Self" of "__new__" of "dict" cannot be "SON[_Key, _Value]" [type-var]

Not sure what's going on in this one (source here), but it looks like a minor mypy bug in its handling of typing_extensions.Self. The other two mypy_primer hits are just noise caused by existing error messages changing slightly.

[srittau]

The primer hit is probably due to dict.__new__ now being incompatible with the bson type annotation. I'm fine with that.

[AlexWaygood]

The primer hit is probably due to dict.__new__ now being incompatible with the bson type annotation. I'm fine with that.

It's weird that dict.__new__ has a different signature to dict.__init__ in our stubs. Maybe we should look into whether we actually need that.

[Avasam]

I noticed this change has introduced a new pyright issue at https://github.com/python/typeshed/blob/main/scripts/stubsabot.py#L37

[AlexWaygood]

My guess is that's probably a pyright issue in how it handles Self. What's the error message pyright produces?

[Avasam]

It thinks member is a int when previously it saw it as a ActionLevelSelf

vs before

I guess pyright doesn't take into account that the first classmethod parameter (cls) could be something else than it's own class.
mypy does it correctly:

Do you feel like opening a pyright issue or shall I?

[AlexWaygood]

Hmm, yeah that seems like it might be a pyright bug (though Eric will be the judge of that). If you file an issue over there, he'll probably either fix it or triage it as "wontfix" within 24 hours :D

We also don't need the ActionLevelSelfTypeVar in stubsabot anymore, though, we can just use typing_extensions.Self there.

[AlexWaygood]

We also don't need the ActionLevelSelfTypeVar in stubsabot anymore, though, we can just use typing_extensions.Self there.

• [Minor] Simplify some type hints in stubsabot #9785

[Avasam]

Alright, created an issue here: microsoft/pyright#4668

[ShaneHarvey]

Coming here because I'm attempting to fix or # type: ignore the SON error (in mongodb/mongo-python-driver#1194).

It's weird that dict.new has a different signature to dict.init in our stubs. Maybe we should look into whether we actually need that.

Has an issue been created for this yet? It's not clear to me whether this is a bug in pymongo or a bug in mypy.

[AlexWaygood]

Coming here because I'm attempting to fix or # type: ignore the SON error (in mongodb/mongo-python-driver#1194).

It's weird that dict.new has a different signature to dict.init in our stubs. Maybe we should look into whether we actually need that.

Has an issue been created for this yet? It's not clear to me whether this is a bug in pymongo or a bug in mypy.

It looks like a mypy bug to me. The dict.__new__/dict.__init__ discrepancy in our stubs is strange, and it's very possible we could get rid of it, but I don't think it's the cause of this error. I'd suggest opening an issue over at mypy.

[ShaneHarvey]

Thanks I opened: python/mypy#15135