gh-127833: Docs: Add a `grammar-snippet` directive & replace `productionlist` by encukou · Pull Request #127835 · python/cpython

encukou · 2024-12-11T16:59:18Z

As a first step to what's proposed in the issue, this overrides the ReST productionlist directive to:

use : instead of the ::= symbol
add syntax highlighting for strings (the implementation is preliminary; I intend to later add a proper class to the theme instead of borrowing a Pygments class.)

All links and link targets should be preserved. (Unfortunately, this reaches into some Sphinx internals; I don't see a better way to do exactly what Sphinx does.)

This also adds a new directive, grammar-snippet, which formats the snippet almost exactly like what's in the source, modulo syntax highlighting and keeping the backtick character to mark links to other rules.
This will allow formatting the snippets as in the grammar file (most importantly: to remove the the indentation that Sphinx adds to align the text).

I'd appreciate if a Sphinx expert could take a look to see if there's a better way to do something :) cc @AA-Turner

Issue: Align the grammar documentation with Python's actual grammar #127833

📚 Documentation preview 📚: https://cpython-previews--127835.org.readthedocs.build/

Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: William Ferreira <wqferr@gmail.com>

to conserve the productionlist usage in the docs.

Co-Authored-By: bswck <bartoszpiotrslawecki@gmail.com>

AA-Turner

A few initial notes. The targets problem is annoying, Sphinx doesn't make it easy...

…_token out

- Add a module docstring - Use the snake_case convention for node classes - Make :group: a required option - declare parallel_write_safe = True

AA-Turner

I'm happy with this as it stands, thank you @encukou for the discussion. As discussed, I've pushed a commit to add type annotations. I've included some other minor changes and left a self-review for a brief explanation -- feel free to revert any you disagree with.

A

miss-islington-app · 2025-02-05T15:12:26Z

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

…roductionlist` (pythonGH-127835) As a first step toward aligning the grammar documentation with Python's actual grammar, this overrides the ReST `productionlist` directive to: - use `:` instead of the `::=` symbol - add syntax highlighting for strings (using a Pygments highlighting class) All links and link targets should be preserved. (Unfortunately, this reaches into some Sphinx internals; I don't see a better way to do exactly what Sphinx does.) This also adds a new directive, `grammar-snippet`, which formats the snippet almost exactly like what's in the source, modulo syntax highlighting and keeping the backtick character to mark links to other rules. This will allow formatting the snippets as in the grammar file (file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html). The new directive is applied to two simple rules in toplevel_components.rst --------- (cherry picked from commit 58a4357) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: William Ferreira <wqferr@gmail.com> Co-authored-by: bswck <bartoszpiotrslawecki@gmail.com> Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>

bedevere-app · 2025-02-05T15:12:53Z

GH-129689 is a backport of this pull request to the 3.13 branch.

…roductionlist` (pythonGH-127835) As a first step toward aligning the grammar documentation with Python's actual grammar, this overrides the ReST `productionlist` directive to: - use `:` instead of the `::=` symbol - add syntax highlighting for strings (using a Pygments highlighting class) All links and link targets should be preserved. (Unfortunately, this reaches into some Sphinx internals; I don't see a better way to do exactly what Sphinx does.) This also adds a new directive, `grammar-snippet`, which formats the snippet almost exactly like what's in the source, modulo syntax highlighting and keeping the backtick character to mark links to other rules. This will allow formatting the snippets as in the grammar file (file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html). The new directive is applied to two simple rules in toplevel_components.rst --------- (cherry picked from commit 58a4357) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: William Ferreira <wqferr@gmail.com> Co-authored-by: bswck <bartoszpiotrslawecki@gmail.com> Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>

bedevere-app · 2025-02-05T15:12:58Z

GH-129690 is a backport of this pull request to the 3.12 branch.

…roductionlist` (pythonGH-127835) As a first step toward aligning the grammar documentation with Python's actual grammar, this overrides the ReST `productionlist` directive to: - use `:` instead of the `::=` symbol - add syntax highlighting for strings (using a Pygments highlighting class) All links and link targets should be preserved. (Unfortunately, this reaches into some Sphinx internals; I don't see a better way to do exactly what Sphinx does.) This also adds a new directive, `grammar-snippet`, which formats the snippet almost exactly like what's in the source, modulo syntax highlighting and keeping the backtick character to mark links to other rules. This will allow formatting the snippets as in the grammar file (file:///home/encukou/dev/cpython/Doc/build/html/reference/grammar.html). The new directive is applied to two simple rules in toplevel_components.rst --------- Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: William Ferreira <wqferr@gmail.com> Co-authored-by: bswck <bartoszpiotrslawecki@gmail.com> Co-authored-by: Adam Turner <9087854+aa-turner@users.noreply.github.com>

In gh-138418, `!` was added to links to rules that don't exist in the docs, in order to silence broken link warnings. However, productionlist doesn't parse the `!`, which ends up in the rendered documentation. (It's possible that gh-127835 broke the `!` support.) Replace the names with ones that appear in docs: - `star_named_expression` in the grammar corresponds to `flexible_expression` in the docs - `star_named_expressions` in the grammar corresponds to `flexible_expression_list` in the docs - `named_expression` in the grammar corresponds to `assignment_expression` in the docs Having two sets of names isn't great of course. Consolidating them is tracked in (subissues of) gh-127833.

…49836) Link to existing rules in compound_stmts.rst (GH-149811) In gh-138418, `!` was added to links to rules that don't exist in the docs, in order to silence broken link warnings. However, productionlist doesn't parse the `!`, which ends up in the rendered documentation. (It's possible that gh-127835 broke the `!` support.) Replace the names with ones that appear in docs: - `star_named_expression` in the grammar corresponds to `flexible_expression` in the docs - `star_named_expressions` in the grammar corresponds to `flexible_expression_list` in the docs - `named_expression` in the grammar corresponds to `assignment_expression` in the docs Having two sets of names isn't great of course. Consolidating them is tracked in (subissues of) gh-127833. (cherry picked from commit c375292) Co-authored-by: Petr Viktorin <encukou@gmail.com>

…49837) Link to existing rules in compound_stmts.rst (GH-149811) In gh-138418, `!` was added to links to rules that don't exist in the docs, in order to silence broken link warnings. However, productionlist doesn't parse the `!`, which ends up in the rendered documentation. (It's possible that gh-127835 broke the `!` support.) Replace the names with ones that appear in docs: - `star_named_expression` in the grammar corresponds to `flexible_expression` in the docs - `star_named_expressions` in the grammar corresponds to `flexible_expression_list` in the docs - `named_expression` in the grammar corresponds to `assignment_expression` in the docs Having two sets of names isn't great of course. Consolidating them is tracked in (subissues of) gh-127833. (cherry picked from commit c375292) Co-authored-by: Petr Viktorin <encukou@gmail.com>

…49838) Link to existing rules in compound_stmts.rst (GH-149811) In gh-138418, `!` was added to links to rules that don't exist in the docs, in order to silence broken link warnings. However, productionlist doesn't parse the `!`, which ends up in the rendered documentation. (It's possible that gh-127835 broke the `!` support.) Replace the names with ones that appear in docs: - `star_named_expression` in the grammar corresponds to `flexible_expression` in the docs - `star_named_expressions` in the grammar corresponds to `flexible_expression_list` in the docs - `named_expression` in the grammar corresponds to `assignment_expression` in the docs Having two sets of names isn't great of course. Consolidating them is tracked in (subissues of) gh-127833. (cherry picked from commit c375292) Co-authored-by: Petr Viktorin <encukou@gmail.com>

encukouand others added 6 commits November 20, 2024 17:55

First version of GrammarSnippetDirective, copied from an earlier branch
86a0ecc
Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: William Ferreira <wqferr@gmail.com>

Add a grammar-snippet directive
f6ffb21

Remove monkey patch for production list and Add CompatProductionList
5cfc701
to conserve the productionlist usage in the docs.

Fix ReST grammar
f0cbed8

Complete the CompatProductionList class
9139e25
Co-Authored-By: bswck <bartoszpiotrslawecki@gmail.com>

Adjust docstrings & move GrammarSnippetDirective class up
b10d78b

encukou requested a review from willingc as a code owner December 11, 2024 16:59

bedevere-appBot mentioned this pull request Dec 11, 2024
Align the grammar documentation with Python's actual grammar #127833
Open

bedevere-appBot added the awaiting core review label Dec 11, 2024

encukou added 2 commits December 11, 2024 18:04

Merge in the main branch
c18a4d3

Run Ruff
a403fb6

encukou added the skip news label Dec 11, 2024

Run Ruff again with different flags
074c189

encukou requested a review from AA-Turner December 11, 2024 17:11

AA-Turner reviewed Jan 15, 2025
View reviewed changes

Comment threadDoc/reference/toplevel_components.rst

AA-Turner reviewed Jan 15, 2025
View reviewed changes

Comment threadDoc/tools/extensions/grammar_snippet.py Outdated
Comment threadDoc/tools/extensions/grammar_snippet.py Outdated
Comment threadDoc/tools/extensions/grammar_snippet.py Outdated
Comment threadDoc/tools/extensions/grammar_snippet.py Outdated

encukou added 2 commits January 22, 2025 16:39

Merge in the main branch
b2fb819

Turn the common function into a class with method; break make_link_to…
ea9e2f0
…_token out

encukou requested a review from hugovk as a code owner January 22, 2025 15:47

encukouand others added 9 commits January 22, 2025 16:50

Remove blank line
b2205a8

Run ruff format
829e4f0

Add forgotten arguments
e57a37e

Merge in the main branch
fca04c7

Use Sphinx's token_xrefs function for formatting the tokens
2732143

Improve the highlight class injection
ef3c552

Appease Ruff
c0c8432

Appase Ruff again
e2359b7

Add type annotations, minor clean-ups
3fcb730
- Add a module docstring - Use the snake_case convention for node classes - Make :group: a required option - declare parallel_write_safe = True

AA-Turner reviewed Jan 29, 2025
View reviewed changes

Comment threadDoc/tools/extensions/grammar_snippet.py
Comment threadDoc/tools/extensions/grammar_snippet.py
Comment threadDoc/tools/extensions/grammar_snippet.py

AA-Turner approved these changes Jan 29, 2025
View reviewed changes

bedevere-appBot added awaiting merge and removed awaiting core review labels Jan 29, 2025

AA-Turner added docs Documentation in the Doc dir needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Jan 29, 2025

encukou merged commit 58a4357 into python:mainFeb 5, 2025

bedevere-appBot removed the awaiting merge label Feb 5, 2025

encukou deleted the doc-grammar-snippets branch February 5, 2025 15:12

bedevere-appBot removed the needs backport to 3.13 bugs and security fixes label Feb 5, 2025

bedevere-appBot removed the needs backport to 3.12 only security fixes label Feb 5, 2025

AA-Turner mentioned this pull request Feb 10, 2025
Remove productionlist hard-coding in translators sphinx-doc/sphinx#13326
Merged

encukou mentioned this pull request May 13, 2026
Link to existing rules in compound_stmts.rst #149811
Merged

This was referenced May 14, 2026
[3.15] Link to existing rules in compound_stmts.rst (GH-149811) #149836
Merged
[3.14] Link to existing rules in compound_stmts.rst (GH-149811) #149837
Merged
[3.13] Link to existing rules in compound_stmts.rst (GH-149811) #149838
Merged

Navigation Menu

Uh oh!

Uh oh!

gh-127833: Docs: Add a `grammar-snippet` directive & replace `productionlist` - #127835

gh-127833: Docs: Add a `grammar-snippet` directive & replace `productionlist`#127835
encukou merged 20 commits into
python:mainfrom
encukou:doc-grammar-snippets

encukou commented Dec 11, 2024•
edited
Loading

Uh oh!

Uh oh!

AA-Turner left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AA-Turner left a comment

Uh oh!

miss-islington-appBot commented Feb 5, 2025

Uh oh!

bedevere-appBot commented Feb 5, 2025

Uh oh!

bedevere-appBot commented Feb 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

encukou commented Dec 11, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

AA-Turner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AA-Turner left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington-appBot commented Feb 5, 2025

Uh oh!

bedevere-appBot commented Feb 5, 2025

Uh oh!

bedevere-appBot commented Feb 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

encukou commented Dec 11, 2024•
edited
Loading