Skip to content

gh-135676: Simplify docs on lexing names - #140464

Merged
encukou merged 8 commits into
python:mainfrom
encukou:lex-analysis-names-simpler
Nov 26, 2025
Merged

gh-135676: Simplify docs on lexing names#140464
encukou merged 8 commits into
python:mainfrom
encukou:lex-analysis-names-simpler

Conversation

@encukou

@encukouencukou commented Oct 22, 2025•
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Member

This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.

It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:

  1. parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
  2. normalizes the name
  3. validates the name, using the id_start/id_continue sets (referred to in previous sections as “letter-like” and “number-like” characters, with a link to the details)

This also means we don't need xid_start/xid_continue to define the behaviour :)

📚 Documentation preview 📚: https://cpython-previews--140464.org.readthedocs.build/

encukouand others added 4 commits October 8, 2025 17:58
@encukou@StanFromIreland
@blaisep@miameows@KeithTheEE
Simplify Names section
4606120
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Micha Albert <info@micha.zone>
Co-authored-by: KeithTheEE <kmurrayis@gmail.com>
@encukou
Casing; 3 dots for character ranges
6163c24
@encukou
Clean-ups
de6d1af
@encukou
Mention Unicode's *ID_Start* and *ID_Continue*
152e7aa
@bedevere-appbedevere-appBot added docs Documentation in the Doc dir skip news labels Oct 22, 2025
@bedevere-appbedevere-appBot mentioned this pull request Oct 22, 2025
Open
@StanFromIrelandStanFromIreland linked an issue Oct 22, 2025 that may be closed by this pull request
Docs: note requirement to normalise unicode identifiers passed to globals() and locals() #86846
Closed
willingc
willingc approved these changes Oct 22, 2025
View reviewed changes

@willingcwillingc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Outstanding document @encukou. I had one small suggestion to be a bit more explicit on the normalization example with number.

Comment threadDoc/reference/lexical_analysis.rst Outdated
This means that, for example, some typographic variants of characters are
converted to their "basic" form, for example::

>>> nᵘₘᵇₑʳ = 3

@willingcwillingcOct 22, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to add an explicit comment that the normalized form of nᵘₘᵇₑʳis number.

@encukouencukouOct 29, 2025

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this look good?

@encukouencukou mentioned this pull request Nov 4, 2025
Merged
@encukou
encukou marked this pull request as draft November 5, 2025 10:45
@encukou

encukou commented Nov 5, 2025

Copy link
Copy Markdown
MemberAuthor

There was an insightful conversation in #140269. I'll update this PR to make things even clearer.

willingc
willingc approved these changes Nov 10, 2025
View reviewed changes

@willingcwillingc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @encukou

@encukou
encukou marked this pull request as ready for review November 19, 2025 16:08
willingc
willingc approved these changes Nov 19, 2025
View reviewed changes

@willingcwillingc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @encukou!

@encukou

encukou commented Nov 20, 2025

Copy link
Copy Markdown
MemberAuthor

Thank you for the review!

@malemburg, do you also want to take a look?

@encukou
encukou merged commit 2ff8608 into python:mainNov 26, 2025
36 checks passed
@encukou
encukou deleted the lex-analysis-names-simpler branch November 26, 2025 15:10
@github-project-automationgithub-project-automationBot moved this from Todo to Done in Docs PRsNov 26, 2025
@encukouencukou added the needs backport to 3.14 bugs and security fixes label Nov 26, 2025
@miss-islington-app

miss-islington-appBot commented Nov 26, 2025

Copy link
Copy Markdown

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

@miss-islington-app

miss-islington-appBot commented Nov 26, 2025

Copy link
Copy Markdown

Sorry, @encukou, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 2ff8608b4da33f667960e5099a1a442197acaea4 3.14

@bedevere-app

bedevere-appBot commented Nov 27, 2025

Copy link
Copy Markdown

GH-142015 is a backport of this pull request to the 3.14 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.14 bugs and security fixes label Nov 27, 2025
StanFromIreland added a commit to StanFromIreland/cpython that referenced this pull request Nov 27, 2025
@encukou@StanFromIreland
@blaisep@miameows@KeithTheEE
[3.14]pythongh-135676: Simplify docs on lexing names (pythonGH-140464)
69312e4
This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.
It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:
- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets
(cherry picked from commit 2ff8608)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Micha Albert <info@micha.zone>
Co-authored-by: KeithTheEE <kmurrayis@gmail.com>
encukou added a commit that referenced this pull request Dec 3, 2025
@StanFromIreland@encukou
@blaisep@miameows@KeithTheEE
[3.14]gh-135676: Simplify docs on lexing names (GH-140464) (GH-142015)
79245a4
This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.
It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:
- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets
(cherry picked from commit 2ff8608)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Micha Albert <info@micha.zone>
Co-authored-by: KeithTheEE <kmurrayis@gmail.com>
StanFromIreland added a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@encukou@StanFromIreland
@blaisep@miameows@KeithTheEE
pythongh-135676: Simplify docs on lexing names (pythonGH-140464)
9a421ea
This simplifies the Lexical Analysis section on Names (but keeps it technically correct) by putting all the info about non-ASCII characters in a separate (and very technical) section.
It uses a mental model where the parser doesn't handle Unicode complexity “immediately”, but:
- parses any non-ASCII character (outside strings/comments) as part of a name, since these can't (yet) be e.g. operators
- normalizes the name
- validates the name, using the xid_start/xid_continue sets
Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Micha Albert <info@micha.zone>
Co-authored-by: KeithTheEE <kmurrayis@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@willingcwillingcwillingc approved these changes
@AA-TurnerAA-TurnerAwaiting requested review from AA-TurnerAA-Turner is a code owner

Assignees

@encukouencukou

Labels

docsDocumentation in the Doc dirskip news

Projects

Docs PRs
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Docs: note requirement to normalise unicode identifiers passed to globals() and locals()

2 participants

@encukou@willingc