bpo-40939: Add the new grammar to the grammar specification documentation by pablogsal · Pull Request #19969 · python/cpython

pablogsal · 2020-05-06T21:00:16Z

Example of how it looks like when using the new lexer:

https://bugs.python.org/issue40939

gvanrossum

I worry about some of the rather verbose comments in the grammar file, and the various error rules. Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed). And maybe the ordering and formatting of some of the rules could be better.

PS. Your initial comments still shows variables (a=...) but the script correctly erases those.

pablogsal · 2020-05-06T21:35:20Z

I worry about some of the rather verbose comments in the grammar file, and the various error rules.

I can try to eliminate alternatives if they start with "invalid_" but I think it will not be super resilient :(

Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed).

Would you like to bring lookaheads back?

And maybe the ordering and formatting of some of the rules could be better.

Not much we can do here with a custom lexer unless we touch the original file.

PS. Your initial comments still shows variables (a=...) but the script correctly erases those.

Yeah, I removed the variables in the last amend after I realized that we wanted to get rid of them.

gvanrossum · 2020-05-06T22:43:47Z

I worry about some of the rather verbose comments in the grammar file, and the various error rules.
I can try to eliminate alternatives if they start with "invalid_" but I think it will not be super resilient :(

Plus, you'd have to eliminate both the alternatives and the rules. But these really do not deserve to be listed in the reference manual.

Also there are some cases where lookaheads are essential (notably the way formal parameters are now parsed).
Would you like to bring lookaheads back?

Only the ones that are significant -- and those in formal parameters are significant (without them, def foo(bar baz) would be legal)

And maybe the ordering and formatting of some of the rules could be better.
Not much we can do here with a custom lexer unless we touch the original file.

Yeah, we can do that later.

pablogsal · 2020-05-06T22:46:50Z

Only the ones that are significant -- and those in formal parameters are significant (without them, def foo(bar baz) would be legal)

Hmmm.... is there any obvious automatic way (using regular expressions) to detect the ones we want from the ones we don't want? Otherwise, we may need to maintain a copy of the grammar for displaying here and manually trim what we need.

gvanrossum · 2020-05-06T22:53:29Z

I dunno. Let's think about this some more before rushing into it. (We'd have to explain a big about PEG's ordering of alternatives too, to guide people using this as a basis for anything.)

lysnikolaou · 2020-05-06T23:05:51Z

Hmmm.... is there any obvious automatic way (using regular expressions) to detect the ones we want from the ones we don't want?

A heuristic that is currently 100% correct is that a lookahead is only used for optimisation and can be skipped if and only if it is a positive lookahead and it appears first in its alternative. Although this is currently correct, it seems like something that could very easily be invalidated in the future, so I don't know if it can be used in the script.

willingc

Overall, I like the approach. There are a few places where color makes a line choppy and hard to follow and may make it more difficult to read than just black text.

…tion

Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>

gvanrossum · 2020-07-26T22:00:06Z

I think I'm going to do something with this after all. Let's see if we can't make the highlighter also remove invalid_* rules and comments.

gvanrossum · 2020-07-26T22:41:20Z

Okay, @lysnikolaou we may need your review because now it's both Pablo's and my code.

pablogsal · 2020-07-27T10:45:08Z

Should we backport this to 3.9? (Minus the Grammar/Grammar delete. :-)

I think it makes sense to do the backport. What do you think @lysnikolaou ?

gvanrossum · 2020-07-27T14:25:56Z

Okay then @pablogsal what do you think of my version?

willingc

LGTM. Thanks @pablogsal and @gvanrossum.

pablogsal

Okay then @pablogsal what do you think of my version?

I think is great :) I left a minor question, but feel free to ignore

Note: I cannot approve my own PR, so go ahead and merge!

lysnikolaou · 2020-07-27T15:10:17Z

Should we backport this to 3.9? (Minus the Grammar/Grammar delete. :-)
I think it makes sense to do the backport. What do you think @lysnikolaou ?

I also think that backporting this to 3.9 makes sense.

gvanrossum · 2020-07-27T17:25:28Z

The new version I pushed has a longer intro in grammar.rst, both to explain (some of) what's omitted and to clarify the notation. It also restores most lookaheads, except for the positive ones that are purely optimizations (as a heuristic I use | followed by a positive lookahead). It makes the grammar a little uglier but reduces the possibility of looking like an ambiguity.

lysnikolaou

LGTM. Thanks!!

miss-islington · 2020-07-27T18:20:40Z

Thanks @pablogsal for the PR, and @gvanrossum for merging it 🌮🎉.. I'm working now to backport this PR to: 3.9.
🐍🍒⛏🤖

miss-islington · 2020-07-27T18:20:46Z

Sorry, @pablogsal and @gvanrossum, I could not cleanly backport this to 3.9 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 72cabb2aa636272e608285f5a6ba83b62be9be4e 3.9

…cumentation (GH-19969) (We censor the heck out of actions and some other stuff using a custom "highlighter".) Co-authored-by: Guido van Rossum <guido@python.org>. (cherry picked from commit 72cabb2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

bedevere-bot · 2020-07-27T18:31:36Z

GH-21641 is a backport of this pull request to the 3.9 branch.

…cumentation (GH-19969) (#21641) (We censor the heck out of actions and some other stuff using a custom "highlighter".) (cherry picked from commit 72cabb2) Co-authored-by: Pablo Galindo <Pablogsal@gmail.com>

…ation (pythonGH-19969) (We censor the heck out of actions and some other stuff using a custom "highlighter".) Co-authored-by: Guido van Rossum <guido@python.org>

pablogsal added the skip issue label May 6, 2020

pablogsal requested review from gvanrossum and lysnikolaou May 6, 2020 21:00

the-knights-who-say-ni added the CLA signed label May 6, 2020

bedevere-bot added the awaiting core review label May 6, 2020

pablogsal self-assigned this May 6, 2020

pablogsal requested a review from willingc May 6, 2020 21:01

pablogsal added skip news and removed skip issue labels May 6, 2020

pablogsal force-pushed the grammar branch from 806b72f to 360b195Compare May 6, 2020 21:05

gvanrossum reviewed May 6, 2020
View reviewed changes

Comment threadDoc/tools/extensions/peg_highlight.py Outdated

willingc reviewed May 7, 2020
View reviewed changes

lysnikolaou mentioned this pull request May 7, 2020
Refactor Grammar for publishing on Python Docs we-like-parsers/pegen_experiments#69
Closed

pablogsal mentioned this pull request May 11, 2020
Labeled Exceptions for pegen - Better error-reporting in CPython we-like-parsers/cpython#123
Closed

pablogsal mentioned this pull request May 26, 2020
Fix lookahead of soft keywords in the PEG parser #20436
Merged

pablogsal mentioned this pull request Jun 11, 2020
Document the new grammar we-like-parsers/cpython#135
Closed

gvanrossum changed the title ~~bpo-40334: Add the new grammar to the grammar specification documentation~~bpo-40939: Add the new grammar to the grammar specification documentationJul 26, 2020

pablogsaland others added 2 commits July 26, 2020 14:55

bpo-40334: Add the new grammar to the grammar specification documenta…
de4d919
…tion

Update Doc/tools/extensions/peg_highlight.py
22e6c71
Co-authored-by: Guido van Rossum <gvanrossum@gmail.com>

gvanrossum force-pushed the grammar branch from 2f674b7 to 22e6c71Compare July 26, 2020 21:57

gvanrossum added 2 commits July 26, 2020 15:10

Tweak text intro to the grammar
c83284e

Suppress all rules named invalid_* or incorrect_*
27f7f00

Now we can remove Grammar/Grammar
266a99a

willingc approved these changes Jul 27, 2020
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Jul 27, 2020

pablogsal commented Jul 27, 2020
View reviewed changes

Comment threadDoc/reference/grammar.rst

pablogsal commented Jul 27, 2020
View reviewed changes

Comment threadDoc/tools/extensions/peg_highlight.py

lysnikolaou reviewed Jul 27, 2020
View reviewed changes

Comment threadDoc/tools/extensions/peg_highlight.py

gvanrossum added 2 commits July 27, 2020 10:20

More conservative suppression of lookaheads
13aa70d

Lengthen intro for grammar.rst
7705e02

Fix typo and comma
bac8971

gvanrossum added the needs backport to 3.9 label Jul 27, 2020

lysnikolaou approved these changes Jul 27, 2020
View reviewed changes

gvanrossum merged commit 72cabb2 into python:masterJul 27, 2020

bedevere-bot removed the awaiting merge label Jul 27, 2020

miss-islington assigned gvanrossum and unassigned pablogsal Jul 27, 2020

bedevere-bot removed the needs backport to 3.9 label Jul 27, 2020

pablogsal deleted the grammar branch May 19, 2021 18:57

Uh oh!

Uh oh!

Conversation

pablogsal commented May 6, 2020• edited by gvanrossum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pablogsal commented May 6, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented May 6, 2020

Uh oh!

pablogsal commented May 6, 2020

Uh oh!

gvanrossum commented May 6, 2020 via email

Uh oh!

lysnikolaou commented May 6, 2020

Uh oh!

willingc left a comment

Choose a reason for hiding this comment

Uh oh!

gvanrossum commented Jul 26, 2020

Uh oh!

gvanrossum commented Jul 26, 2020

Uh oh!

pablogsal commented Jul 27, 2020

Uh oh!

gvanrossum commented Jul 27, 2020

Uh oh!

willingc left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou commented Jul 27, 2020

Uh oh!

gvanrossum commented Jul 27, 2020

Uh oh!

lysnikolaou left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Jul 27, 2020

Uh oh!

miss-islington commented Jul 27, 2020

Uh oh!

bedevere-bot commented Jul 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pablogsal commented May 6, 2020•
edited by gvanrossum
Loading

pablogsal commented May 6, 2020•
edited
Loading