Remove list.__add__ overloads.

[randolf-scholz]

See #14283.

The overloads on list.__add__ lead to divergence of mypy and pyright. Code sample in pyright playground, https://mypy-play.net/?mypy=latest&python=3.12&gist=abf6a8834020af17a16bd8cfb44b2f10

fromtypingimportAny, overloadclassListA[T]: # emulates builtins list@overloaddef__add__(self, other: "ListA[T]", /) ->"ListA[T]": returnListA()
@overloaddef__add__[S](self, other: "ListA[S]", /) ->"ListA[T | S]": returnListA()
classListB[T]: # without overloadsdef__add__[S](self, other: "ListB[S]", /) ->"ListB[T | S]": returnListB()
# mypy | pyright reveal_type( list[str]() +list[str]() ) # list[str] | list[str] ✅reveal_type( list[str]() +list[int]() ) # list[str | int] | list[str | int] ✅reveal_type( list[str]() +list[Any]() ) # list[Any] | list[str] ❌reveal_type( ListA[str]() +ListA[str]() ) # ListA[str] | ListA[str] ✅reveal_type( ListA[str]() +ListA[int]() ) # ListA[str | int] | ListA[str | int] ✅reveal_type( ListA[str]() +ListA[Any]() ) # ListA[Any] | ListA[str] ❌reveal_type( ListB[str]() +ListB[str]() ) # ListB[str] | ListB[str] ✅reveal_type( ListB[str]() +ListB[int]() ) # ListB[str | int] | ListB[str | int] ✅reveal_type( ListB[str]() +ListB[Any]() ) # ListB[str | Any] | ListB[str | Any] ✅

A comment from 3 years added in #8293 states that

# Overloading looks unnecessary, but is needed to work around complex mypy problems

I want to see the impact on mypy primer, and whether this comment is still valid given that mypy has had several major releases since.

[randolf-scholz]

There must be a strange bug in mypy. Take this example from the mypy primer above

If I change the offending line (Unsupported operand types for + ("list[FrameSummary]" and "list[FrameSummary]")) from

forsintraceback.format_list(tb+tb2):

to

dummy=tb+tb2forsintraceback.format_list(dummy):

Then the file passes without issue. (using python -m mypy.stubtest --custom-typeshed-dir=../typeshed/ tmp)

[jorenham]

maybe something like this could work?

Suggested change

	def__add__(self, value: list[_S], /) ->list[_S|_T]: ...
	def__add__(self: list[_S], value: list[_S], /) ->list[_S]: ...

[randolf-scholz]

With this, wouldn't we get list[str] + list[int] = list[object] when checking with mypy, rather then list[str | int]?

I'd prefer python/mypy#19304 be fixed; then I think we would see most mypy-primer issues here go away. If you have seen this before and know and easier way to reproduce, please comment it.

[jorenham]

With this, wouldn't we get list[str] + list[int] = list[object] when checking with mypy, rather then list[str | int]?

With this, wouldn't we get list[str] + list[int] = list[object] when checking with mypy, rather then list[str | int]?

I'd prefer python/mypy#19304 be fixed; then I think we would see most mypy-primer issues here go away. If you have seen this before and know and easier way to reproduce, please comment it.

That's what I indeed expect here, given mypy's join vs union behavior. But if that's how they decide to behave, then that's up to them, afaik. At least it won't be rejected this way (is the hope)

[randolf-scholz]

Well, my goal with this PR was to reduce type-checker divergence, not to cement it.

[jorenham]

fair enough

[randolf-scholz]

This is probably gonna sit here until python/mypy#19304 is fixed. A minimal repro of this is

classVec[T]:
defget(self) ->T: ... # just here to insure invariancedefset(self, index: int, value: T) ->None: ... # just here to insure invariancedef__add__[S](self, other: "Vec[S]", /) ->"Vec[T | S]": returnVec()
defpprint(arg: Vec[object]) ->None: ...
defdemo(ints: Vec[int], strs: Vec[str]):
pprint(ints+strs) # ❌️ mypy false positive

Last year I spend quite a bit of time trying to fix this: randolf-scholz/mypy#4

But gave up after a while because there were other blockers:

• RFC: Rework *args parsing mypy#20472
• Improved Literal? types mypy#20444

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

mypy (https://github.com/python/mypy)
+ mypyc/crash.py:29: error: Unsupported operand types for + ("list[FrameSummary]" and "list[FrameSummary]") [operator]+ mypyc/crash.py:29: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-operator for more info+ mypy/strconv.py:467: error: Unsupported operand types for + ("list[Any]" and "list[str | tuple[str, list[Any]]]") [operator]+ mypy/errors.py:1396: error: Unsupported operand types for + ("list[FrameSummary]" and "StackSummary") [operator]+ mypy/fastparse.py:1603: error: Unsupported operand types for + ("list[expr]" and "list[expr]") [operator]+ mypy/test/testcmdline.py:80: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]
freqtrade (https://github.com/freqtrade/freqtrade)
+ freqtrade/data/entryexitanalysis.py:285: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ freqtrade/data/entryexitanalysis.py:290: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ freqtrade/data/entryexitanalysis.py:296: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ freqtrade/data/entryexitanalysis.py:297: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]
spark (https://github.com/apache/spark)
+ python/pyspark/pandas/groupby.py:3797: error: Unsupported operand types for + ("list[Series[Any]]" and "list[Series[Any]]") [operator]
graphql-core (https://github.com/graphql-python/graphql-core)
+ src/graphql/type/validate.py:417: error: Unsupported operand types for + ("list[NamedTypeNode]" and "list[NamedTypeNode]") [operator]+ src/graphql/validation/rules/overlapping_fields_can_be_merged.py:86: error: Unsupported operand types for + ("list[FieldNode]" and "list[FieldNode]") [operator]
tornado (https://github.com/tornadoweb/tornado)
+ tornado/netutil.py:158: error: Incompatible types in assignment (expression has type "tuple[int | Any, ...]", variable has type "tuple[str, int] | tuple[str, int, int, int] | tuple[int, bytes]") [assignment]+ tornado/autoreload.py:234: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]
rotki (https://github.com/rotki/rotki)
+ rotkehlchen/chain/evm/decoding/uniswap/utils.py:318: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/pendle/decoder.py:496: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/beefy_finance/decoder.py:270: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/balancer/v3/decoder.py:238: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/odos/common.py:177: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/curve/decoder.py:438: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/balancer/v2/decoder.py:118: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/uniswap/v2/utils.py:393: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]+ rotkehlchen/chain/evm/decoding/uniswap/v2/utils.py:395: error: Unsupported operand types for + ("list[EvmEvent]" and "list[EvmEvent]") [operator]
spack (https://github.com/spack/spack)
+ lib/spack/spack/environment/environment.py:2702: error: Incompatible return value type (got "list[tuple[Any, Any] | tuple[Spec, None]]", expected "list[tuple[Spec, Spec]]") [return-value]
sympy (https://github.com/sympy/sympy)
+ sympy/polys/galoistools.py:1755: error: Need type annotation for "Q" [var-annotated]
materialize (https://github.com/MaterializeInc/materialize)
+ misc/python/materialize/cli/ci_upload_heap_profiles.py:42: error: Unsupported operand types for + ("list[str | Any]" and "list[str]") [operator]+ misc/python/materialize/cargo.py:279: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ misc/python/materialize/cargo.py:283: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ misc/python/materialize/mzbuild.py:1568: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ misc/python/materialize/mzbuild.py:1572: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]
pandas (https://github.com/pandas-dev/pandas)
+ pandas/io/stata.py:1403: error: List comprehension has incompatible type List[object]; expected List[str | dtype[Any]] [misc]+ pandas/tests/groupby/conftest.py:130: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ pandas/tests/frame/test_constructors.py:2357: error: Unsupported operand types for + ("list[ExtensionDtype | str | dtype[Any] | type[object] | type[str] | type[complex] | type[bool]]" and "list[ExtensionDtype | str | dtype[Any] | type[str] | type[complex] | type[bool] | type[object]]") [operator]+ pandas/tests/frame/test_constructors.py:2367: error: Unsupported operand types for + ("list[ExtensionDtype | str | dtype[Any] | type[object] | type[str] | type[complex] | type[bool]]" and "list[ExtensionDtype | str | dtype[Any] | type[str] | type[complex] | type[bool] | type[object]]") [operator]+ pandas/tests/arrays/test_datetimes.py:70: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ pandas/conftest.py:1661: error: Unsupported operand types for + ("list[ExtensionDtype | str | dtype[Any] | type[object] | type[str] | type[complex] | type[bool]]" and "list[ExtensionDtype | str | dtype[Any] | type[str] | type[complex] | type[bool] | type[object]]") [operator]
prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/callables.py:579: error: Incompatible types in assignment (expression has type "list[None]", variable has type "list[expr | None]") [assignment]- src/prefect/utilities/callables.py:579: note: "list" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance- src/prefect/utilities/callables.py:579: note: Consider using "Sequence" instead, which is covariant
pydantic (https://github.com/pydantic/pydantic)
- pydantic/aliases.py:29: error: Incompatible types in assignment (expression has type "list[str]", variable has type "list[int | str]") [assignment]- pydantic/aliases.py:29: note: "list" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance- pydantic/aliases.py:29: note: Consider using "Sequence" instead, which is covariant- pydantic/aliases.py:29: error: Argument 1 to "list" has incompatible type "tuple[str | int, ...]"; expected "Iterable[str]" [arg-type]+ pydantic/aliases.py:29: error: Argument 1 to "list" has incompatible type "tuple[str | int, ...]"; expected "Iterable[int]" [arg-type]
scipy (https://github.com/scipy/scipy)
+ scipy/fft/_pocketfft/tests/test_basic.py:473: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fft/_pocketfft/tests/test_basic.py:483: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fft/_pocketfft/tests/test_basic.py:502: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fft/_pocketfft/tests/test_basic.py:512: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fftpack/tests/test_basic.py:429: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fftpack/tests/test_basic.py:439: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fftpack/tests/test_basic.py:458: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/fftpack/tests/test_basic.py:468: error: Unsupported operand types for + ("list[int]" and "list[int]") [operator]+ scipy/optimize/tests/test_optimize.py:1295: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]
jax (https://github.com/google/jax)
+ jax/_src/internal_test_util/test_harnesses.py:422: error: Unsupported operand types for + ("list[str | type[Any] | dtype[Any] | SupportsDType]" and "list[str | type[Any] | dtype[Any] | SupportsDType]") [operator]
meson (https://github.com/mesonbuild/meson)
+ mesonbuild/scripts/gettext.py:54:70: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/utils/universal.py:2382:87: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/cmake/interpreter.py:1080:69: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/cmake/interpreter.py:1225:60: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/modules/gnome.py:1558:23: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/modules/gnome.py:1575:77: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/depfixer.py:515:66: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:62:40: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:73:40: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:83:35: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:120:35: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:128:35: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:134:59: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:140:64: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:147:64: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:156:47: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/scripts/coverage.py:172:40: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]+ mesonbuild/mtest.py:2263:47: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]
cloud-init (https://github.com/canonical/cloud-init)
+ cloudinit/config/cc_mounts.py:420: error: Incompatible return value type (got "list[list[str | Any | None]]", expected "list[list[str]]") [return-value]
paasta (https://github.com/yelp/paasta)
+ paasta_tools/config_utils.py:100: error: Unsupported operand types for + ("list[str]" and "list[str]") [operator]