untokenize of specially crafted escaped characters does not round trip properly

[asottile]

Bug report

Bug description:

this small program does not roundtrip through tokenize / untokenize -- it appears to mishandle the escaped quote as a \N{NAMED ESCAPE}

bar = 1
print(f"{bar} \"{{SNOWMAN}} {{foo}}")

this is what it produces after a round of untokenization:

$ python3 t.py t3.py 
bar = 1
print(f"{bar} \"{ SNOWMAN}} {{foo}}")

annoyingly, tokenize_rt suffers from a different related bug which is why I was investigating this to begin with. an aside, the handling of curly braces in 3.12+ tokenization is a huge pain!

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[asottile]

cc @pablogsal ecf16ee

[pablogsal]

Sorry is a bit late here and I had a long day, but what I am missing here:

>>> import tokenize
>>> import io
>>> code = r'f"{bar} \"{{SNOWMAN}} {{foo}}"'
>>> tokens = list(tokenize.generate_tokens(io.StringIO(code).readline))
>>> tokenize.untokenize(tokens)
'f"{bar} \\"{{SNOWMAN}} {{foo}}"'
>>> code
'f"{bar} \\"{{SNOWMAN}} {{foo}}"'

Same with a file:

$ cat lel.py
bar = 1
print(f"{bar} \"{{SNOWMAN}} {{foo}}")

$ ./python.exe
Python 3.14.0a1+ (heads/main-dirty:03f9264ecef, Oct 22 2024, 09:15:42) [Clang 16.0.0 (clang-1600.0.26.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tokenize
>>> data = open('lel.py')
>>> tokens = list(tokenize.generate_tokens(data.readline))
>>> print(tokenize.untokenize(tokens))
bar = 1
print(f"{bar} \"{{SNOWMAN}} {{foo}}")

Sorry if I am missing anything obvious. Can you provide a small reproducer I can run?

[pablogsal]

Ah this was fixed in db23b8b in main and was back ported to 3.12 but not yet released:

#125021

[pablogsal]

Closing as duplicate of #125008

[asottile]

hah yep -- just checked my vod and I spent all the time waiting on a build only to rerun my system python 🤦