Merge typed_ast back into CPython

[gvanrossum]

BPO	35766
Nosy	@gvanrossum, @gpshead, @vstinner, @ambv, @ilevkivskyi, @asottile, @ethanhs, @miss-islington
PRs	bpo-35766: Merge typed_ast back into CPython #11645 bpo-35766: Merge typed_ast back into CPython #11645 bpo-35766: Merge typed_ast back into CPython #11645 bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod #11714 bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod #11714 bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod #11714 bpo-35766 follow-up: Add an error check to new_type_comment() #11766 bpo-35766 follow-up: Add an error check to new_type_comment() #11766 bpo-35766 follow-up: Add an error check to new_type_comment() #11766 bpo-35766: Whats new in the ast and typing modules #13984 [3.8] bpo-35766: What's new in the ast and typing modules (GH-13984) #13987 bpo-35766: Change format for feature_version to (major, minor) #13992 [3.8] bpo-35766: Change format for feature_version to (major, minor) (GH-13992) #13993 bpo-35766: compile(): rename feature_version parameter #13994 [3.8] bpo-35766: compile(): rename feature_version parameter (GH-13994) #13995 [3.8] bpo-35766: compile(): rename feature_version parameter (GH-13994) #14001

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-02-12.00:11:24.565>
created_at = <Date 2019-01-18.00:33:01.067>
labels = ['3.8', 'type-feature', 'library']
title = 'Merge typed_ast back into CPython'
updated_at = <Date 2019-06-12.15:34:26.524>
user = 'https://github.com/gvanrossum'

bugs.python.org fields:

activity = <Date 2019-06-12.15:34:26.524>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-12.00:11:24.565>
closer = 'gvanrossum'
components = ['Library (Lib)']
creation = <Date 2019-01-18.00:33:01.067>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 35766
keywords = ['patch', 'patch', 'patch']
message_count = 28.0
messages = ['333919', '334068', '334071', '334073', '334371', '334500', '334624', '334714', '335234', '335276', '335278', '345245', '345248', '345249', '345256', '345264', '345265', '345267', '345270', '345273', '345285', '345286', '345292', '345293', '345294', '345364', '345366', '345372']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'gregory.p.smith', 'vstinner', 'lukasz.langa', 'levkivskyi', 'Anthony Sottile', 'ethan smith', 'miss-islington']
pr_nums = ['11645', '11645', '11645', '11714', '11714', '11714', '11766', '11766', '11766', '13984', '13987', '13992', '13993', '13994', '13995', '14001']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35766'
versions = ['Python 3.8']

[gvanrossum]

(This started at https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377. It's somewhat related to https://bugs.python.org/issue33337.)

I now have a thorough understanding of what typed_ast does, and I think it would be straightforward to port it upstream. We’d need to define two new tokens to represent # type: ignore and # type: <whatever>, and tokenizer code to recognize these. Then we need a new flag to be passed to the tokenizer (via the parser) that enables this behavior. We make a small number of changes to Grammar (inserting optional TYPE_COMMENT tokens and to Python.asdl (adding fields to a few node types to hold the optional type comment), and a fair number of changes to ast.c to extract the type comments. We have similar patches for 3.6 and 3.7, so it's a simple matter of porting those patches to 3.8.

By default, ast.parse() should not return type comments, since this would reject some perfectly good Python code (with sonething looking like a type comment in a place where the grammar doesn’t allow it). But passing an new flag will cause the tokenizer to process type comments and the returned tree will contain them.

I could produce a PR with this in a few days (having just gone over most of the process for porting typed_ast from 3.6 to 3.7).

There’s one more feature I’d like to lobby for – a feature_version flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4). This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether async and await are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful.

In the next few days I will produce a PR so people can see for themselves.

In https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17, Łukasz offered to merge my PR.

[asottile]

Seems also related to https://bugs.python.org/issue24119

with python2 / python3.5 (hopefully) rapidly falling off in usage I would assume the specialized treatment of # type: ... comments would become less and less necessary

Though I guess there's still # type: ignore, though if I recall correctly ignores are bubbled up to the module level and are applied on a line-by-line basis and wouldn't necessarily need specialized AST treatment (though a second parse over the token stream is a bit unfortunate)

[gvanrossum]

You’d be surprised how tenacious old versions are.

Anywa, I am working on this on my copious spare time — you can follow my progress at https://github.com/gvanrossum/cpython/tree/ast-type-comments?files=1 .

[asottile]

You’d be surprised how tenacious old versions are.

heh that's true, at my last job we finally got rid of python2.6 in 2016 :'(

anyway -- I don't mean to discourage this, definitely seems valuable to the maintenance of mypy!

[gvanrossum]

The PR is ready for reviews now.

[gvanrossum]

Also the tests now all pass; for now I am happy with the solution I found for the indentation error (see #11645 (comment)).

[ambv]

New changeset dcfcd14 by Łukasz Langa (Guido van Rossum) in branch 'master':
bpo-35766: Merge typed_ast back into CPython (GH-11645)
dcfcd14

[gvanrossum]

New changeset 3a32e3b by Guido van Rossum in branch 'master':
bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod (bpo-11714)
3a32e3b

[gvanrossum]

New changeset 4b250fc by Guido van Rossum in branch 'master':
bpo-35766 follow-up: Add an error check to new_type_comment() (bpo-11766)
4b250fc

[gvanrossum]

I have some follow-up wishes but I'll create a new issue.

[gvanrossum]

See https://bugs.python.org/issue35975