Add __parameters__ and __getitem__ in TypeVar and ParamSpec

[serhiy-storchaka]

BPO	44796
Nosy	@gvanrossum, @serhiy-storchaka, @JelleZijlstra, @sobolevn, @Fidget-Spinner, @AlexWaygood
PRs	gh-88959: Add __parameters__ and __getitem__ in TypeVar and ParamSpec #27511 bpo-44796: Unify TypeVar and ParamSpec substitution #31143
Dependencies	bpo-44794: Merge tests for typing.Callable and collection.abc.Callable

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-07-31.13:31:29.771>
labels = ['type-feature', 'library', '3.11']
title = 'Add __parameters__ and __getitem__ in TypeVar and ParamSpec'
updated_at = <Date 2022-03-12.13:54:12.028>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2022-03-12.13:54:12.028>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-07-31.13:31:29.771>
creator = 'serhiy.storchaka'
dependencies = ['44794']
files = []
hgrepos = []
issue_num = 44796
keywords = ['patch']
message_count = 9.0
messages = ['398640', '411555', '412464', '412483', '412484', '412575', '412704', '414895', '414992']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'serhiy.storchaka', 'JelleZijlstra', 'sobolevn', 'kj', 'AlexWaygood']
pr_nums = ['27511', '31143']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44796'
versions = ['Python 3.11']

[serhiy-storchaka]

Adding __parameters__ and __getitem__ in TypeVar and ParamSpec allows to generalize and simplify the code (especially the C code) and allows to add more runtime checks. It may open ways for further simplification.

Unfortunately it is not compatible with bpo-44098, so the latter changes should be reverted.

[JelleZijlstra]

The linked PR makes T = TypeVar("T"); T[int] work. That's not currently meaningful syntax to the type checker, but we may want it in the future for higher-kinded types. I would rather not make this change without a clear static typing use case in mind.

[serhiy-storchaka]

It does not have a use case of T[int] in mind. It is an implementation detail which simplifies the code, makes runtime checks more strict and makes types.GenericAlias more similar to typing._GenericAlias.

For example, currently:

>>> A = List[T]
>>> B = list[T]
>>> A[None]
typing.List[NoneType]
>>> B[None]
list[None]
>>> A['X']
typing.List[ForwardRef('X')]
>>> B['X']
list['X']

With the proposed change:

>>> B[None]
list[NoneType]
>>> B['X']
list[ForwardRef('X')]

Meanless expressions like A[42], A[Final], A[Final[int]], A[ClassVar], A[ClassVar[int]], etc which are silently accepted will now be errors.

The code simplification (especially for the C code) is my primary goal, and the rest is a nice side effect. It is possible to add more strict runtime checks and conversions without making TypeVar and ParamSpec subscriptable, but the code will not be so simple.

[gvanrossum]

I don't think that changing list[None] to list[NoneType] in the output is an improvement. I do appreciate the reduction in C code though!

[JelleZijlstra]

I'd also be wary about making changes that introduce additional runtime checks. As we've seen with the PEP-612 and 646 implementations, those can impede future iteration on typing.

[serhiy-storchaka]

I have created an alternative PR 31143 which adds special method __typing_subst__ in TypeVar and ParamSpec instead of __parameters__ and __getitem__. It has almost the same effect except making TypeVar and ParamSpec and subscriptable. The Python code in typing.py and _collections_abc.py is simplified, but the C code is little changed.

But on other hand, the reasons for PR 27511:

1. We already implemented __getitem__ for a Concatenate alias, although it is never documented. It is not even necessary, it is a pure implementation detail which makes the code for nested variables substitution simpler. Adding __parameters__ and __getitem__ in TypeVar and ParamSpec would have the same effect on simplification of the code.

2. There is a use case for subscription of TypeVar. For example:

if version < 2:
 RetType = T
elif version < 3:
 RetType = Optional[T]
else:
 RetType = Optional[T] | Literal[Unknown]

def get_int(precision: int) -> RetType[int]: ...
def get_str(encoding: str) -> RetType[str]: ...
def get_any(converter: Callable[[str], T]) -> RetType[T]: ...

[gvanrossum]

Super subtle stuff. Tonight I do not have time to really dive into this, but I think Serhiy is on to something. KJ or Jelle, do you have the guts to dive in here?

[serhiy-storchaka]

New changeset b6a5d85 by Serhiy Storchaka in branch 'main':
bpo-44796: Unify TypeVar and ParamSpec substitution (GH-31143)
b6a5d85

[serhiy-storchaka]

PR 31143 simplified some code without changing the common API:

Objects/genericaliasobject.c | 46 ++++++++++++++--------------------------------
1 file changed, 14 insertions(+), 32 deletions(-)
Lib/_collections_abc.py | 63 +++++++++------------------------------------------------------
1 file changed, 9 insertions(+), 54 deletions(-)

PR 27511 will simplify it even more:

Objects/genericaliasobject.c | 60 ++++++++++++++++++------------------------------------------
1 file changed, 18 insertions(+), 42 deletions(-)

(Lib/typing.py will be simplified too, even if this does not reduce the number of lines).