[WIP] Python 3.9 parser by emmatyping · Pull Request #137 · python/typed_ast

emmatyping · 2020-05-09T05:25:02Z

emmatyping · 2020-05-09T05:34:40Z

I'm not really sure how best to get this reviewed. I probably should have created a commit when I imported the files...

My process was basically:

Import files
Update symbols/asdl_c/peg_generator
generate AST c files and Parser/pegen/parse.c
rewrite Custom/typed_ast.c to use the pegen API.
make it build (some extracting of CPython things needed, see for example Custom/pegen_extra.c and Include/pegen_extra.c
Make it build with older Python.

~~I'll take a look at Windows this weekend.~~ Looks like Windows builds are fine! Except 3.5, which this won't support.

gvanrossum · 2020-05-09T16:01:14Z

Hm... Something failing in the tests because it's using the walrus. I guess that's the meta-parser (primarily). Serhiy actually has attempted a fix for that, maybe it's useful?

This should be released as typed_ast 2.0 I suppose.

I will look at it in detail after breakfast. :-)

gvanrossum · 2020-05-09T18:14:26Z

I'm not really sure how best to get this reviewed. I probably should have created a commit when I imported the files...

Yeah, that would have helped. Is it too late to redo it? I guess you could create a temporary branch with that and then rebase this branch over that.

So far I've managed to build and install it using pip install .. When I do this using Python 3.7, and then run mypy, I immediately get this failure:

Traceback (most recent call last):
File "/Users/guido/v37/bin/mypy", line 6, in <module>
from mypy.__main__ import console_entry
File "/Users/guido/v37/lib/python3.7/site-packages/mypy/__main__.py", line 4, in <module>
from mypy.main import main
File "mypy/main.py", line 13, in <module>
File "mypy/build.py", line 33, in <module>
File "mypy/semanal.py", line 95, in <module>
File "mypy/typeanal.py", line 30, in <module>
File "mypy/exprtotype.py", line 10, in <module>
File "mypy/fastparse.py", line 80, in <module>
AttributeError: module 'typed_ast.ast3' has no attribute 'Ellipsis'

It seems ast.Ellipsis was deprecated and somehow didn't make it into your port? But mypy expects it.

Also, testing with mypy and Python 3.8 is tricky because mypy checks sys.version and uses ast instead of typed_ast if it's >= (3, 8).

gvanrossum · 2020-05-09T18:25:26Z

Running the tests manually (on Mac) is a pain. I tried pytest but that fails with essentially

ImportError: cannot import name '_ast3' from 'typed_ast' (/Users/guido/typed_ast/typed_ast/__init__.py)

Forcing PYTHONPATH to the build/lib* subdirectory where the built files live doesn't help.

Then I tried python setup.py test which succeeded -- with 0 tests run.

But this copies the .so file into the source tree. (!) So I tried pytestagain.

Now it segfaults:

(v38) ~/typed_ast$ pytest -v
================================================== test session starts ==================================================
platform darwin -- Python 3.8.3rc1, pytest-5.3.2, py-1.8.0, pluggy-0.13.1 -- /Users/guido/v38/bin/python3.8
cachedir: .pytest_cache
benchmark: 3.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/guido/typed_ast
plugins: forked-1.0.2, xdist-1.31.0, benchmark-3.2.3, cov-2.8.1
collected 15 items ast3/tests/test_basics.py::test_basics FAILED [ 6%]
ast3/tests/test_basics.py::test_redundantdef PASSED [ 13%]
ast3/tests/test_basics.py::test_vardecl FAILED [ 20%]
ast3/tests/test_basics.py::test_forstmt FAILED [ 26%]
ast3/tests/test_basics.py::test_withstmt FAILED [ 33%]
ast3/tests/test_basics.py::test_longargs FAILED [ 40%]
ast3/tests/test_basics.py::test_ignores FAILED [ 46%]
ast3/tests/test_basics.py::test_asyncfunc FAILED [ 53%]
ast3/tests/test_basics.py::test_asyncvar FAILED [ 60%]
ast3/tests/test_basics.py::test_asynccomp Fatal Python error: Segmentation fault
Current thread 0x0000000107bcfdc0 (most recent call first):
File "/Users/guido/typed_ast/ast3/tests/test_basics.py", line 262 in test_asynccomp
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/python.py", line 1435 in runtest
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 131 in pytest_runtest_call
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 207 in <lambda>
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 234 in from_call
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 206 in call_runtest_hook
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 182 in call_and_report
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 96 in runtestprotocol
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/runner.py", line 81 in pytest_runtest_protocol
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 270 in pytest_runtestloop
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 246 in _main
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 196 in wrap_session
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
File "/Users/guido/v38/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
File "/Users/guido/v38/lib/python3.8/site-packages/_pytest/config/__init__.py", line 91 in main
File "/Users/guido/v38/bin/pytest", line 8 in <module>
Segmentation fault: 11
(v38) ~/typed_ast$

So at least we know there's something fishy with async comprehensions...

gvanrossum · 2020-05-09T18:27:14Z

Oh, it's simpler.

(v38) ~/typed_ast$ python
Python 3.8.3rc1 (tags/v3.8.3rc1:802eb676ba, Apr 30 2020, 11:47:35) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Cannot read termcap database;
using dumb terminal settings.
>>> from typed_ast import ast3; ast3.parse("pass")
Segmentation fault: 11
(v38) ~/typed_ast$

gvanrossum · 2020-05-09T18:34:40Z

I have a bit more from lldb, but then I'll stop (probably better to try and understand what you did before trying to debug the segfault).

The crash is at line 410 in pegen.c:

 if (t->col_offset == -1) {

It appears t is nonsense. This in turn appears because at line 388, t is initialized like this:

 Token *t = p->known_err_token != NULL ? p->known_err_token : p->tokens[p->fill - 1];

and the problem is that p->known_err_token is NULL, but p->fill is 0, so the expression p->tokens[p->fill - 1] produces nonsense.

All that points to deeper problems. Probably it's easier for you.

emmatyping · 2020-05-09T21:06:38Z

Thank you for looking into this! I will definitely be able to redo it. And I was going to stream some later today anyway, so I will also record it (maybe that will also help 🤷‍♂️)

gvanrossum · 2020-05-10T00:55:32Z

Whatever can be automated in the adaptation of the copied files to work here should be automated (as should the copying itself, perhaps). We will be doing this many times. I wouldn't expect that it's only the grammar that changes (then we could just regenerate parse.c and be done) -- I expect plenty of changes to the support code as well, sometimes systematic changes like python/cpython#19987.

emmatyping · 2020-05-10T10:55:13Z

Superseded by #138

Import Python 3.9 parser
ed9113d

the-knights-who-say-ni added the CLA signed label May 9, 2020

Add notes on the origin of the backports
9a20276

emmatyping mentioned this pull request May 10, 2020
[WIP] Automated 3.9 update #138
Closed

emmatyping closed this May 10, 2020

Navigation Menu

Uh oh!

Uh oh!

[WIP] Python 3.9 parser - #137

[WIP] Python 3.9 parser#137
emmatyping wants to merge 2 commits into
python:masterfrom
emmatyping:3.9

emmatyping commented May 9, 2020

Uh oh!

emmatyping commented May 9, 2020•
edited
Loading

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

emmatyping commented May 9, 2020

Uh oh!

gvanrossum commented May 10, 2020

Uh oh!

emmatyping commented May 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

emmatyping commented May 9, 2020

Uh oh!

emmatyping commented May 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

gvanrossum commented May 9, 2020

Uh oh!

emmatyping commented May 9, 2020

Uh oh!

gvanrossum commented May 10, 2020

Uh oh!

emmatyping commented May 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

emmatyping commented May 9, 2020•
edited
Loading