Prohibit invisible control characters in string literals and comments

[stevendaprano]

BPO	45810
Nosy	@stevendaprano, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-11-15.23:11:23.999>
labels = ['type-security', 'interpreter-core', '3.11']
title = 'Prohibit invisible control characters in string literals and comments'
updated_at = <Date 2021-11-15.23:11:23.999>
user = 'https://github.com/stevendaprano'

bugs.python.org fields:

activity = <Date 2021-11-15.23:11:23.999>
actor = 'steven.daprano'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2021-11-15.23:11:23.999>
creator = 'steven.daprano'
dependencies = []
files = []
hgrepos = []
issue_num = 45810
keywords = []
message_count = 1.0
messages = ['406370']
nosy_count = 2.0
nosy_names = ['steven.daprano', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue45810'
versions = ['Python 3.11']

[stevendaprano]

Currently invisible control characters aside from whitespace (tabs, newlines, formfeeds, carriage returns) are prohibited outside of comments and string literals. As discussed in this thread:

https://mail.python.org/archives/list/python-dev@python.org/message/DN24FK3A2DSO4HBGEDGJXERSAUYK6VK6/

we should ban C0 and C1 control characters (aside from \t\n\f\r) in string literals and comments too.

To be clear, the ban is on actual invisible control characters, not escape sequences.

[zitterbewegung]

I am working on this.

[zitterbewegung]

Since this is an old issue and that removing these invisible control characters will break backwards compatibility; at first we will raise deprecation warnings.

[StanFromIreland]

As discussed in PEP 672 and this Discourse discussion, handling such characters is the responsibility of the renderer, not the language.

[terryjreedy]

To expand Stan's comment: The link in the opening post, meant to justify this issue, is one message in a pydev thread by Petr Viktorin originally entitled "pre-PEP: Unicode Security Considerations for Python" and retitled as "Preventing Unicode-related gotchas".
https://mail.python.org/archives/list/python-dev@python.org/thread/6DBJJRQHA2SP5Q27MOMDSTCOXMW7ITNR/#DN24FK3A2DSO4HBGEDGJXERSAUYK6VK6
The original message referenced a unicode-based attack based on the difference between actual parsing and expected parsing based on appearance to viewers. It continued with the view Stan gave above, also expressed in UTS #55, which is referenced in the Discourse discussion. The subsequent Informational PEP was aimed at authors of checkers (lints), viewers, and Unicode users in general. It proposed no change in the language. (And such would now require a Discourse discussion and, I believe, an SC-approved PEP.)

If the CPython PR test linter, part of CI, does not already fail on control chars (I don't know), a new failure condition could be proposed.