Narrow type variable bounds in binder

[ilevkivskyi]

Fixes#5720
Fixes#8556
Fixes#9778
Fixes#10003
Fixes#10817
Fixes#11163
Fixes#11664
Fixes#12882
Fixes#13426
Fixes#13462
Fixes#14941
Fixes#15151
Fixes#19166

This handles a (surprisingly) common edge case. The charges in bind_self() and bind_self_fast() are tricky. I got a few "Redundant cast" errors there, which seemed good, but then I realized that attribute access etc. on a type variable go through slow PyObject paths, so I am actually forcing CallableType instead of F(bound=CallableType) there, since these are performance-critical functions.

[ilevkivskyi]

OK, the mypy_primer results are good, and although it looks like a niche use case, this PR fixes more than dozen issues (some with many upvotes).

[hauntsaninja]

Hooray!

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

PyGithub (https://github.com/PyGithub/PyGithub)
+ github/MainClass.py:988: error: Unused "type: ignore" comment [unused-ignore]
antidote (https://github.com/Finistere/antidote): 1.47x slower (65.5s -> 96.3s in single noisy sample)
prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Union[State[Any], Any]]"; expected "list[State[Any]]" [arg-type]+ src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Never]"; expected "list[State[Any]]" [arg-type]+ src/prefect/futures.py:222: note: def result(self, raise_on_failure: Literal[True] = ..., retry_result_failure: bool = ...) -> Any+ src/prefect/futures.py:222: note: def result(self, raise_on_failure: Literal[False] = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]+ src/prefect/futures.py:222: note: def result(self, raise_on_failure: bool = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]- src/prefect/task_engine.py:467: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result" [union-attr]+ src/prefect/task_engine.py:467: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result" [union-attr]- src/prefect/task_engine.py:1022: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result" [union-attr]+ src/prefect/task_engine.py:1022: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result" [union-attr]- src/prefect/flow_runs.py:292: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load" [union-attr]+ src/prefect/flow_runs.py:292: error: "type[T]" has no attribute "load" [attr-defined]- src/prefect/flow_runs.py:307: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save" [union-attr]+ src/prefect/flow_runs.py:307: error: "type[T]" has no attribute "save" [attr-defined]- src/prefect/flow_runs.py:319: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load" [union-attr]+ src/prefect/flow_runs.py:319: error: "type[T]" has no attribute "load" [attr-defined]- src/prefect/flow_runs.py:446: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load" [union-attr]+ src/prefect/flow_runs.py:446: error: "type[T]" has no attribute "load" [attr-defined]- src/prefect/flow_runs.py:456: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save" [union-attr]+ src/prefect/flow_runs.py:456: error: "type[T]" has no attribute "save" [attr-defined]
kopf (https://github.com/nolar/kopf)
+ kopf/_cogs/structs/dicts.py:243: error: Unused "type: ignore" comment [unused-ignore]+ kopf/_core/intents/handlers.py:36: error: Redundant cast to "execution.CauseT" [redundant-cast]
python-chess (https://github.com/niklasf/python-chess)
- chess/pgn.py:1315: error: Returning Any from function declared to return "HeadersT" [no-any-return]
kornia (https://github.com/kornia/kornia)
+ kornia/utils/helpers.py:387: error: Unused "type: ignore" comment [unused-ignore]
mypy (https://github.com/python/mypy)
+ mypy/expandtype.py:125: error: Redundant cast to "F" [redundant-cast]+ mypy/expandtype.py:125: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-redundant-cast for more info+ mypy/typeops.py:421: error: Redundant cast to "F" [redundant-cast]+ mypy/typeops.py:430: error: Redundant cast to "F" [redundant-cast]+ mypy/typeops.py:469: error: Incompatible types in assignment (expression has type "CallableType", variable has type "F") [assignment]+ mypy/typeops.py:484: error: Redundant cast to "F" [redundant-cast]+ mypy/checkmember.py:1489: error: Redundant cast to "F" [redundant-cast]+ mypy/checkmember.py:1492: error: Redundant cast to "F" [redundant-cast]+ mypy/checkmember.py:1502: error: Redundant cast to "F" [redundant-cast]
hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "type[Any]") [assignment]+ src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "_T") [assignment]+ src/hydra_zen/third_party/beartype.py:130: error: Redundant cast to "_T" [redundant-cast]+ src/hydra_zen/third_party/pydantic.py:111: error: Redundant cast to "_T" [redundant-cast]+ src/hydra_zen/third_party/pydantic.py:115: error: Redundant cast to "_T" [redundant-cast]- src/hydra_zen/third_party/pydantic.py:123: error: Argument 1 has incompatible type "_T | type[object]"; expected "_T" [arg-type]
pytest (https://github.com/pytest-dev/pytest)
- src/_pytest/fixtures.py:1132: error: Returning Any from function declared to return "FixtureValue" [no-any-return]
pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/__init__.py:67: error: Unused "type: ignore" comment [unused-ignore]+ tests/__init__.py:85: error: Unused "type: ignore" comment [unused-ignore]
beartype (https://github.com/beartype/beartype)
+ beartype/_decor/_type/decortype.py:133: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/_type/decortype.py:295: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/_nontype/decornontype.py:294: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/_nontype/decornontype.py:301: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/_nontype/decornontype.py:380: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/_nontype/decornontype.py:395: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/_nontype/decornontype.py:567: error: Unused "type: ignore" comment [unused-ignore]+ beartype/_decor/decorcore.py:133: error: Unused "type: ignore" comment [unused-ignore]
xarray (https://github.com/pydata/xarray)
+ xarray/structure/alignment.py: note: In function "_broadcast_helper":+ xarray/structure/alignment.py:1140: error: Redundant cast to "T_Alignable" [redundant-cast]+ xarray/structure/alignment.py:1142: error: Redundant cast to "T_Alignable" [redundant-cast]
steam.py (https://github.com/Gobot1234/steam.py)
- steam/ext/commands/commands.py:284: error: Argument 1 to "append" of "list" has incompatible type "Command[Any, Any, Any]"; expected "Self" [arg-type]- steam/ext/commands/cog.py:182: error: Argument 1 to "decorator" has incompatible type "Callable[[Any | None, VarArg(Any), KwArg(Any)], Coroutine[Any, Any, None]]"; expected "F" [arg-type]- steam/ext/commands/bot.py:415: error: Argument 1 to "decorator" has incompatible type "Callable[..., Coroutine[Any, Any, Any]]"; expected "F" [arg-type]

[netomi]

ftr: this commit fixes a bug we encountered with mypy 1.16.0 (and not in 1.15.0) in different places with a stacktrace like:

asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
sys.exit(console_entry())
File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

[ilevkivskyi]

@netomi Hm, this is interesting, could you please post a full traceback? (use --show-traceback)

[netomi]

that is already with this option:

tn@proteus:~/workspace/apache/infrastructure-asfyaml$ poetry run mypy --show-traceback asfyaml/feature/github/pull_requests.py asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
sys.exit(console_entry())
File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

the project can be found here: https://github.com/apache/infrastructure-asfyaml

it is now using 1.15.0, but updating to 1.16.0 shows the bug.

[ilevkivskyi]

A bit more of the traceback

 File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2062, in wrap_context
yield
File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2391, in finish_passes
self._patch_indirect_dependencies(self.type_checker().module_refs, all_types)
File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/build.py", line 2424, in _patch_indirect_dependencies
encountered = self.manager.indirection_detector.find_modules(types) | module_refs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/indirection.py", line 35, in find_modules
self._visit(typs)
File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/indirection.py", line 46, in _visit
typ.accept(self)
File "/home/ivan/venvs/mypy_dev_3_12/lib/python3.12/site-packages/mypy/types.py", line 276, in accept
raise RuntimeError("Not implemented", type(self))
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)

This looks like something where I would say this PR may be indeed a proper fix. So looks like no action needed (apart from releasing 1.17 LOL)

[netomi]

just wanted to mention it as there might be others that report issues about that, so there should be a changelog entry to track that this is solved in 1.17.0 imho.