gh-105820: Fix tok_mode expression buffer in file & readline tokenizer

[lysnikolaou]

• Issue: patchcheck.py raises SystemError on main #105820
• Issue: F-string debug mode does not print correctly when in last line of file #105831

[pablogsal]

@lysnikolaou Hummm, can you quickly cover why moving this chunk solves the problem? I think I am missing some key insight here

[lysnikolaou]

Sure. There's two different cases to consider here:

1. When we're calling the tokenizer from the tokenize module, it uses the readline tokenizer, which you added in 9216e69. In that branch, update_fstring_expr wasn't being called at all from tok_underflow_readline, which resulted in the last_expr_size not being updated. That's why we were passing a negative count to PyUnicode_DecodeUTF8.

cpythonon  mainviaCv14.0.3-clangvia 🐍 pyenv3.11.3 ❯ cattmp/t.pyprint(f'''{3=}''')%cpythonon  mainviaCv14.0.3-clangvia 🐍 pyenv3.11.3 ❯ ./python.exe-mtokenizetmp/t.pyunexpectederror: NegativesizepassedtoPyUnicode_NewTraceback (mostrecentcalllast):
File"/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py", line198, in_run_module_as_mainreturn_run_code(code, main_globals, None,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File"/Users/lysnikolaou/repos/python/cpython/Lib/runpy.py", line88, in_run_codeexec(code, run_globals)
File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line547, in<module>main()
File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line489, inmaintokens=list(tokenize(f.readline))
^^^^^^^^^^^^^^^^^^^^^^^^^^File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line447, intokenizeyieldfrom_generate_tokens_from_c_tokenizer(rl_gen.__next__, encoding, extra_tokens=True)
File"/Users/lysnikolaou/repos/python/cpython/Lib/tokenize.py", line537, in_generate_tokens_from_c_tokenizerforinfoinit:
SystemError: NegativesizepassedtoPyUnicode_New

2. There was a second bug with an f-string being tokenized from a file, only when the closing brace of the f-string expression part is in the last line of the file and there's no newline at the end. In that branch we use tok->cur to calculate the size of the buffer before adding an implicit newline and then tok->startafter adding the newline. That resulted in the end buffer being one character too short.

cpythonon  mainviaCv14.0.3-clangvia 🐍 pyenv3.11.3took15s ❯ cattmp/t.pyprint(f'''{3=}''')%cpythonon  mainviaCv14.0.3-clangvia 🐍 pyenv3.11.3 ❯ ./python.exetmp/t.py33

[lysnikolaou]

Do you think I should split this PR in two and open a second issue for the second case above?

[pablogsal]

Do you think I should split this PR in two and open a second issue for the second case above?

Nah, I think is fine, but we may want to add two NEWS entries

[pablogsal]

LGTM

Thanks for the explanation @lysnikolaou. Fantastic work as always! 💪

[pablogsal]

@lysnikolaou If you want add the second NEWS entry and then land it.

[miss-islington]

Thanks @lysnikolaou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[bedevere-bot]

GH-105832 is a backport of this pull request to the 3.12 branch.