gh-130587: Add hand-written docs for non-OP tokens by encukou · Pull Request #130588 · python/cpython

encukou · 2025-02-26T15:40:38Z

Add hand-written docs for non-OP tokens
Make the automation (generate_token.py) check that the hand-written docs are present, and only generate docs for the OP tokens
Switch to list-table for the OP tokens, to make their docs more compact
Add ENDMARKER to the grammar docs where it appears (toplevel components)
Add forgotten versionchanged entry for EXCLAMATION
Remove docs for NT_OFFSET

Issue: token module documentation is incomplete #130587

📚 Documentation preview 📚: https://cpython-previews--130588.org.readthedocs.build/

…e rest

…ARKER to that

Co-authored-by: Blaise Pabon <blaise@gmail.com>

encukou · 2025-02-26T16:44:41Z

The rendered docs are at https://cpython-previews--130588.org.readthedocs.build/en/130588/library/token.html

encukou · 2025-02-26T16:45:20Z

@lysnikolaou, does this look reasonable to you?

lysnikolaou

This looks like a good improvement to me! Thanks @encukou!

I've left some inline comments regarding some specifics in the docs.

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

encukou · 2025-02-27T11:00:52Z

Thank you for the reviews! I addressed some; I'll continue next week.

encukou · 2025-03-05T15:37:57Z

+Note that a token's value may depend on tokenizer options. For example, a
+``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or
+a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`.
+


I added this note to clarify the “fuzziness” of these categories.

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

miss-islington-app · 2025-03-19T15:46:34Z

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

miss-islington-app · 2025-03-19T15:46:37Z

Sorry, @encukou, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4bced29a74c4dbcf310a47e8202292aaa39b617b 3.13

bedevere-app · 2025-03-19T15:50:01Z

GH-131465 is a backport of this pull request to the 3.13 branch.

…0588) Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com> (cherry picked from commit 4bced29)

…onGH-130588) (cherry picked from commit 4bced29) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

…GH-131465) Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

encukouand others added 12 commits January 22, 2025 17:57

generate_tokens.py: Only generate docs for 'literal' tokens; check th…
7129f00
…e rest

Change docs for the "literal" tokens to a list-table
d400ae7

Merge in the main branch
352611b

Document most of the tokens; improve top-level grammar docs link ENDM…
4411c37
…ARKER to that

Write prose; reorganize the token list
1bf5511

Fixups
e1e498b

Merge in the main branch
56c5d7c

Correct directive name
8bbbb0f
Co-authored-by: Blaise Pabon <blaise@gmail.com>

Don't use the Gather syntax
eed407e
Co-authored-by: Blaise Pabon <blaise@gmail.com>

Revert some changes to the toplevel_components
0dd236f

Merge in the main branch
90cd1e2

Revert an order change
dca268e

encukou requested review from AA-Turner and willingc as code owners February 26, 2025 15:40

bedevere-appBot added the awaiting core review label Feb 26, 2025

bedevere-appBot mentioned this pull request Feb 26, 2025
token module documentation is incomplete #130587
Closed

encukou added docs Documentation in the Doc dir skip news labels Feb 26, 2025

AA-Turner reviewed Feb 26, 2025
View reviewed changes

StanFromIreland reviewed Feb 26, 2025
View reviewed changes

Comment threadDoc/library/token.rst Outdated
Comment threadDoc/library/token.rst Outdated
Comment threadDoc/library/token.rst

lysnikolaou reviewed Feb 27, 2025
View reviewed changes

Comment threadDoc/library/token.rst Outdated
Comment threadDoc/library/token.rst Outdated
Comment threadDoc/library/token.rst Outdated
Comment threadDoc/library/token.rst Outdated
Comment threadDoc/library/token.rst Outdated

encukou commented Feb 27, 2025
View reviewed changes

Comment threadDoc/library/token.rst Outdated

encukou commented Feb 27, 2025
View reviewed changes

Comment threadDoc/library/token.rst Outdated

Apply suggestions from code review
e02ced8
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>

Add a note and reword OP docs
02fff75

encukou commented Mar 5, 2025
View reviewed changes

Apply suggestions from code review
9b6eb2a
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

encukou merged commit 4bced29 into python:mainMar 19, 2025

bedevere-appBot removed the awaiting core review label Mar 19, 2025

github-project-automationBot moved this from Todo to Done in Docs PRs Mar 19, 2025

encukou deleted the token-docs branch March 19, 2025 15:42

encukou added the needs backport to 3.13 bugs and security fixes label Mar 19, 2025

miss-islington-appBot assigned encukou Mar 19, 2025

bedevere-appBot removed the needs backport to 3.13 bugs and security fixes label Mar 19, 2025

encukou mentioned this pull request Mar 24, 2025
Align the grammar documentation with Python's actual grammar #127833
Open

Uh oh!

Uh oh!

Conversation

encukou commented Feb 26, 2025• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

encukou commented Feb 26, 2025

Uh oh!

encukou commented Feb 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lysnikolaou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

encukou commented Feb 27, 2025

Uh oh!

encukouMar 5, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miss-islington-appBot commented Mar 19, 2025

Uh oh!

miss-islington-appBot commented Mar 19, 2025

Uh oh!

bedevere-appBot commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

encukou commented Feb 26, 2025•
edited by github-actions Bot
Loading

encukouMar 5, 2025•
edited
Loading