Improved Literal? types

[randolf-scholz]

This PR aims to formalize the behavior of Literal? types (Instance with last_known_value) by treating them as-if they were AnyOf-types (see python/typing#566)

For example, Literal["x"]? is treated as-if it were AnyOf[str, Literal["x"]].

• FixesSpecify behavior for Literal? types. #19625 (specification)
• Fixesmeet_types gives unexpected results when meeting literal and Instance #19560
• FixesRegression: false positive avoid redundant cast with literal union #19055

New Tests

• JoinSuite.test_mixed_literal_types tests join between Literal? and other types.
• MeetSuite.test_mixed_literal_types tests meet between Literal? and other types
• TypeOpsSuite.test_simplified_union_with_mixed_str_literals2 tests simplified unions containing Literal? types
• SubtypingSuite.test_literal tests subtype checks with Literal? types
• RestrictionSuite: new test suite for testing the restrict_subtype_away method.
• testJoinLiteralInstanceAndEnum tests join between Literal? and StrEnum

[randolf-scholz]

Repro of the operator issue:

fromtypingimportLiteral, reveal_typeclassPort:
protocol: Literal["tcp", "udp", "icmp"]
port: int|Nonedefshow(*ports: int|Port) ->None:
# master: set[tuple[L['tcp', 'udp', 'icmp'], int | None]]# PR: set[tuple[L['tcp']?, int] | tuple[L['tcp', 'udp', 'icmp'], int | None]]reveal_type( { ( "tcp", port ) ifisinstance(port, int) else ( port.protocol, port.port )
forportinports
}
)

On master the set comprehension combines both branches into set[tuple[L['tcp', 'udp', 'icmp'], int | None]].
With this PR, Literal['tcp']? is a subtype but not a proper subtype of Literal['tcp'], so the tuples are not simplified away in make_simplified_union, which checks proper subtyping. For context, here are other type checker results:

checker	result
mypy master	set[tuple[L['tcp', 'udp', 'icmp'], int | None]]
mypy PR	set[tuple[L['tcp']?, int] | tuple[L['tcp', 'udp', 'icmp'], int | None]]
pyright	set[tuple[str, int] | tuple[str, int | None]]
ty	set[tuple[str, int | None] | unknown]
pyrefly	set[tuple[str, int] | tuple[str, int | None]]

RFC: would this be considered a regression, or is the behavior proposed by the PR OK?

[randolf-scholz]

Maybe it's possible to do something smarter here, so that e.g. list[Literal["x"]] and list[Literal["x"]?] would be combined as well

[davidhalter]

Just in case you're interested, I have implicit literals implemented as well in Zuban. However it works in a different way, I store the implicit information on the literal. This makes it a bit easier to deal with literals in my opinion. I understand that that might cause a lot of changes in Mypy and it also might cause other issues, but I feel like it's a bit better to store it that way, because it essentially is a literal.

[randolf-scholz]

Update: The operator issue could be patched by a logical change in typeshed: python/typeshed#15160

[randolf-scholz]

Converting to draft since recent changes give "Non-overlapping equality check" errors.

[randolf-scholz]

suspicious primer results:

# pandas/io/pytables.py:4837ifvalues.ndim==1andisinstance(values, np.ndarray):
values=values.reshape((1, values.shape[0]))

# pytest/testing/test_capture.py:971cr=CaptureResult("out", "err") # namedtuple
...
assertcr== ("out", "err")
assertcr!= ("out", "wrong")
...
assertcr._replace(err="replaced") == ("out", "replaced")

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pyinstrument (https://github.com/joerick/pyinstrument)
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "float" [arg-type]+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "float" [arg-type]- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "Literal['enabled', 'disabled', 'strict']" [arg-type]+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "Literal['enabled', 'disabled', 'strict']" [arg-type]- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "bool | None" [arg-type]+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "bool | None" [arg-type]
prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/server/services/task_run_recorder.py:194: error: "Collection[Any]" has no attribute "state" [attr-defined]+ src/prefect/server/services/task_run_recorder.py:194: error: "object" has no attribute "state" [attr-defined]- src/prefect/server/services/task_run_recorder.py:197: error: "Collection[Any]" has no attribute "id" [attr-defined]+ src/prefect/server/services/task_run_recorder.py:197: error: "object" has no attribute "id" [attr-defined]- src/prefect/server/services/task_run_recorder.py:203: error: "Collection[Any]" has no attribute "keys" [attr-defined]+ src/prefect/server/services/task_run_recorder.py:203: error: "object" has no attribute "keys" [attr-defined]- src/prefect/concurrency/_asyncio.py:112: error: R? has no attribute "json" [attr-defined]+ src/prefect/task_worker.py:387: error: Argument 1 to "submit" of "Executor" has incompatible type "Callable[[Callable[_P, _T], **_P], _T]"; expected "Callable[[Callable[[Task[P, R], UUID | None, TaskRun | None, dict[str, Any] | None, PrefectFuture[Any] | Any | Iterable[PrefectFuture[Any] | Any] | None, Literal['state', 'result'], dict[str, set[RunInput]] | None, dict[str, Any] | None], R | State[Any] | None], Task[[VarArg(Any), KwArg(Any)], Any], UUID, TaskRun, dict[Any, Any], list[Any], str, Any | None], Any | State[Any] | None]" [arg-type]
steam.py (https://github.com/Gobot1234/steam.py)
- steam/ext/csgo/state.py:180: error: Argument "slot" to "Sticker" has incompatible type "int | None"; expected "Literal[0, 1, 2, 3, 4, 5] | None" [arg-type]
operator (https://github.com/canonical/operator)
- ops/pebble.py:1171: error: TypedDict key must be a string literal; expected one of ("command", "service-context", "environment", "user-id", "user", ...) [literal-required]- ops/pebble.py:1189: error: TypedDict key must be a string literal; expected one of ("url", "headers") [literal-required]+ ops/model.py:811: error: Incompatible types in assignment (expression has type "str", variable has type "Literal['tcp', 'udp', 'icmp']") [assignment]
discord.py (https://github.com/Rapptz/discord.py)
- discord/app_commands/transformers.py:141: error: Incompatible types in assignment (expression has type "list[dict[str, Any]]", target has type "str | int") [assignment]+ discord/app_commands/transformers.py:141: error: Incompatible types in assignment (expression has type "list[dict[str, Any]]", target has type "bool | str | int") [assignment]- discord/app_commands/transformers.py:143: error: Incompatible types in assignment (expression has type "list[int]", target has type "str | int") [assignment]+ discord/app_commands/transformers.py:143: error: Incompatible types in assignment (expression has type "list[int]", target has type "bool | str | int") [assignment]- discord/app_commands/transformers.py:151: error: Incompatible types in assignment (expression has type "int | float", target has type "str | int") [assignment]+ discord/app_commands/transformers.py:151: error: Incompatible types in assignment (expression has type "int | float", target has type "bool | str | int") [assignment]- discord/app_commands/transformers.py:153: error: Incompatible types in assignment (expression has type "int | float", target has type "str | int") [assignment]+ discord/app_commands/transformers.py:153: error: Incompatible types in assignment (expression has type "int | float", target has type "bool | str | int") [assignment]
pytest (https://github.com/pytest-dev/pytest)
+ testing/test_assertrewrite.py:940: error: Unused "type: ignore" comment [unused-ignore]
dedupe (https://github.com/dedupeio/dedupe)
- dedupe/api.py:1547: error: Redundant cast to "Literal['match', 'distinct']" [redundant-cast]
materialize (https://github.com/MaterializeInc/materialize)
- misc/python/materialize/cli/scratch/__main__.py:95: error: Argument 1 to "join" of "str" has incompatible type "list[str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]]"; expected "Iterable[str]" [arg-type]+ misc/python/materialize/cli/scratch/__main__.py:95: error: Argument 1 to "join" of "str" has incompatible type "list[Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]]"; expected "Iterable[str]" [arg-type]- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument 1 to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str" [arg-type]+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument 1 to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str" [arg-type]- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "aliases" to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "Sequence[str]" [arg-type]+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "aliases" to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "Sequence[str]" [arg-type]- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "description" to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None" [arg-type]+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "description" to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None" [arg-type]- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "help" to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None" [arg-type]+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "help" to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None" [arg-type]- misc/python/materialize/cli/scratch/__main__.py:103: error: Argument 1 to "run" has incompatible type "ArgumentParser"; expected "Namespace" [arg-type]+ misc/python/materialize/cli/scratch/__main__.py:103: error: Argument 1 has incompatible type "ArgumentParser"; expected "Namespace" [arg-type]
mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/net/server_spec.py:85: error: Unused "type: ignore" comment [unused-ignore]
jax (https://github.com/google/jax)
- jax/_src/pallas/mosaic/lowering.py:450: error: Redundant cast to "Literal['parallel', 'core_parallel', 'subcore_parallel', 'arbitrary']" [redundant-cast]- jax/_src/pallas/mosaic/lowering.py:636: error: Argument 1 has incompatible type "Callable[[str | None], str]"; expected "Callable[[Literal['parallel', 'core_parallel', 'subcore_parallel', 'arbitrary'] | GridDimensionSemantics], str]" [arg-type]+ jax/_src/pallas/mosaic/lowering.py:636: error: Argument 1 has incompatible type "Callable[[str | None], str]"; expected "Callable[[GridDimensionSemantics | str], str]" [arg-type]- jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]] | int, ...]", variable has type "list[list[int]]") [assignment]- jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]] | int, ...]", variable has type "tuple[int, ...]") [assignment]+ jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[object, ...]", variable has type "list[list[int]]") [assignment]+ jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[object, ...]", variable has type "tuple[int, ...]") [assignment]- jax/_src/internal_test_util/test_harnesses.py:2290: error: Incompatible types in assignment (expression has type "tuple[ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]] | int, ...]", variable has type "list[list[int]]") [assignment]+ jax/_src/internal_test_util/test_harnesses.py:2290: error: Incompatible types in assignment (expression has type "tuple[int | ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]], ...]", variable has type "list[list[int]]") [assignment]
mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ bson/__init__.py:1332: error: Cannot infer value of type parameter "_DocumentType" of "_bson_to_dict" [misc]