Document that CPython accepts "invalid" identifiers

[vstinner]

BPO	35105
Nosy	@rhettinger, @terryjreedy, @nedbat, @stevendaprano, @cjerdonek, @serhiy-storchaka, @pablogsal, @Windsooon, @tirkarthi, @orlnub123, @akulakov
PRs	bpo-35105: Document that CPython accepts "invalid" identifiers #11263

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-10-29.12:24:51.107>
labels = ['3.8', 'docs']
title = 'Document that CPython accepts "invalid" identifiers'
updated_at = <Date 2021-08-13.22:18:37.761>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-08-13.22:18:37.761>
actor = 'andrei.avk'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2018-10-29.12:24:51.107>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35105
keywords = ['patch']
message_count = 20.0
messages = ['328816', '329034', '329038', '329039', '329040', '329161', '329169', '329208', '329209', '329215', '329289', '329496', '331404', '331405', '332418', '332420', '333033', '372885', '399549', '399566']
nosy_count = 13.0
nosy_names = ['rhettinger', 'terry.reedy', 'roysmith', 'nedbat', 'steven.daprano', 'chris.jerdonek', 'docs@python', 'serhiy.storchaka', 'pablogsal', 'Windson Yang', 'xtreak', 'orlnub123', 'andrei.avk']
pr_nums = ['11263']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue35105'
versions = ['Python 3.8']

[vstinner]

The Python 3 language has a strict definition for an identifier:
https://docs.python.org/dev/reference/lexical_analysis.html#identifiers

... but in practice, it's possible to create "invalid" identifiers using setattr().

Example with PyPy:

$ pypy
Python 2.7.13 (0e7ea4fe15e82d5124e805e2e4a37cae1a402d4b, Apr 12 2018, 14:50:12)
>>>> class A: pass
>>>> 
>>>> a=A()
>>>> setattr(a, "1", 2)
>>>> getattr(a, "1")
2

>>>> vars(a)
{'1': 2}
>>>> a.__dict__
{'1': 2}

>>>> a.1
 File "<stdin>", line 1
 a.1
 ^
SyntaxError: invalid syntax

The exact definition of "identifiers" is a common question. Recent examples:

• bpo-25205
• [Python-Dev] Arbitrary non-identifier string keys when using **kwargs
  https://mail.python.org/pipermail/python-dev/2018-October/155435.html

It would be nice to document the answer. Maybe in the Langage Specification, maybe in the setattr() documentation, maybe in a FAQ, maybe everywhere?

[orlnub123]

I'd argue that it's an implementation detail. Documenting it might be nice as some projects such as pytest do use it but I don't think it would make sense in setattr() or getattr() since all they do (at least in this case) is assign/retrieve from the __dict__. One thing to note is that __slots__ doesn't accept them.

[vstinner]

I'd argue that it's an implementation detail. Documenting it might be nice as some projects such as pytest do use it but I don't think it would make sense in setattr() or getattr() since all they do (at least in this case) is assign/retrieve from the __dict__. One thing to note is that __slots__ doesn't accept them.

Maybe it's obvious to you, but the question is asked again and again, at least once per year. So it seems like we need to document it somewhere.

[Windsooon]

I agreed we should document it, it' not obvious to me at least.

[orlnub123]

The customizing attribute access section of the data model might be a suitable place.

[terryjreedy]

It is an implementation detail that some people need to know, and that is very unlikely to change. In the pydev thread, Guido said
"
My feeling is that limiting it to strings is fine, but checking those
strings for resembling identifiers is pointless and wasteful."

We occasionally document such things in a 'CPython implementation detail' note. I don't know the proper markup for these. At present, I think the note should be in setattr and **kwargs docs.

[vstinner]

I don't know the proper markup for these.

It's ".. impl-detail::". See for example:
https://docs.python.org/dev/library/codecs.html#standard-encodings

[nedbat]

This seems like a confusion of two things: identifiers are lexical elements of the language. Attributes are not limited to identifiers.

We could add to the docs for setattr: "The attribute name does not have to be a valid identifier." I don't know what the language guarantees about what strings are valid as attribute names.

[cjerdonek]

In the pydev thread, Guido said "My feeling is that limiting it to strings is fine, but checking those strings for resembling identifiers is pointless and wasteful."

But in a later message, after additional discussion, he acknowledged there could be reasons to change and said, "we needn't rush this."

So if the docs do describe the current implementation, I think it should warn people that this behavior might not be subject to the same backwards compatibility guarantees as other documented behavior.

[terryjreedy]

Documenting something as an 'implementation detail' denies that it is a language feature and does not offer stability guarantees.

[orlnub123]

I take back my previous suggestion, I agree that documenting it in setattr() (and **kwargs) is the way to go. It's obvious that you can assign anything to the __dict__, since it represents a dict, but setattr() is more ambiguous.
'Anything' was the key word for me here. For example you can assign ints to __dict__ and it won't complain but try to do the same with setattr()/getattr() and it results in an error.

[Windsooon]

I try to create a PR for it. Should we add 'CPython implementation detail' at the document? Because this happens at cpython as well as pypy. BTW, where should we add the document? I have two choices.

• https://docs.python.org/3/reference/datamodel.html#object.\_\_setattr__
• https://docs.python.org/3/library/functions.html#setattr