gh-104016: Fixed off by 1 error in f string tokenizer

[jx124]

• Issue: Out-of-bounds write in AST parser #104016

[ghost]

All commit authors signed the Contributor License Agreement.

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[Fidget-Spinner]

DIsclaimer: This is my friend who I'm helping guide to contribute to CPython. I will thus exclude myself from approving this to prevent any conflict of interest which may arise.

[pablogsal]

I was actually preparing a fix for this problem. I think the correct approach here is to separate the two limits. I will update your PR with my approach and we can land it afterwards as we need this fix to build upon for other fixes. In other circumstances we could iterate together but beta freeze is too close 😅

[bedevere-bot]

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

[pablogsal]

Great job @jx124 🚀

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot wasm32-wasi 3.x has failed when building commit 5078eed.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1046/builds/1882) and take a look at the build logs.
4. Check if the failure is related to this commit (5078eed) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1046/builds/1882

Failed tests:

• test_fstring

Failed subtests:

• test_fstring_nested_too_deeply - test.test_fstring.TestCase.test_fstring_nested_too_deeply

Summary of the results of the build (if available):

== Tests result: FAILURE ==

325 tests OK.

10 slowest tests:

• test_math: 1 min 39 sec
• test_tokenize: 1 min 20 sec
• test_lib2to3: 53.0 sec
• test_capi: 32.9 sec
• test_unparse: 30.4 sec
• test_hashlib: 24.5 sec
• test_unicodedata: 16.5 sec
• test_fstring: 15.3 sec
• test_pickle: 13.4 sec
• test_tarfile: 13.0 sec

1 test failed:
test_fstring

135 tests skipped:
test.test_asyncio.test_base_events
test.test_asyncio.test_buffered_proto
test.test_asyncio.test_context test.test_asyncio.test_events
test.test_asyncio.test_futures test.test_asyncio.test_futures2
test.test_asyncio.test_locks test.test_asyncio.test_pep492
test.test_asyncio.test_proactor_events
test.test_asyncio.test_protocols test.test_asyncio.test_queues
test.test_asyncio.test_runners
test.test_asyncio.test_selector_events
test.test_asyncio.test_sendfile test.test_asyncio.test_server
test.test_asyncio.test_sock_lowlevel test.test_asyncio.test_ssl
test.test_asyncio.test_sslproto test.test_asyncio.test_streams
test.test_asyncio.test_subprocess
test.test_asyncio.test_taskgroups test.test_asyncio.test_tasks
test.test_asyncio.test_threads test.test_asyncio.test_timeouts
test.test_asyncio.test_transports
test.test_asyncio.test_unix_events test.test_asyncio.test_waitfor
test.test_asyncio.test_windows_events
test.test_asyncio.test_windows_utils test__xxinterpchannels
test__xxsubinterpreters test_asyncgen test_bz2 test_clinic
test_cmd_line test_concurrent_futures test_contextlib_async
test_ctypes test_curses test_dbm_gnu test_dbm_ndbm test_devpoll
test_doctest test_docxmlrpc test_dtrace test_embed test_epoll
test_faulthandler test_fcntl test_file_eintr test_fork1
test_ftplib test_gdb test_grp test_gzip test_httplib
test_httpservers test_idle test_imaplib test_interpreters
test_ioctl test_kqueue test_launcher test_lzma test_mailbox
test_mmap test_msilib test_multiprocessing_fork
test_multiprocessing_forkserver test_multiprocessing_main_handling
test_multiprocessing_spawn test_nis test_openpty test_ossaudiodev
test_pdb test_peg_generator test_perf_profiler test_pipes
test_poll test_poplib test_pty test_pwd test_queue test_readline
test_regrtest test_repl test_resource test_select test_selectors
test_smtplib test_smtpnet test_socket test_socketserver test_spwd
test_sqlite3 test_ssl test_stable_abi_ctypes test_startfile
test_subprocess test_sys_settrace test_syslog test_tcl
test_telnetlib test_thread test_threadedtempfile test_threading
test_threading_local test_tix test_tkinter test_tools test_ttk
test_ttk_textonly test_turtle test_urllib test_urllib2
test_urllib2_localnet test_urllib2net test_urllib_response
test_urllibnet test_venv test_wait3 test_wait4 test_webbrowser
test_winconsoleio test_winreg test_winsound test_wmi test_wsgiref
test_xmlrpc test_xmlrpc_net test_xxlimited test_zipfile64
test_zipimport_support test_zlib test_zoneinfo
0:06:34 load avg: 3.31
0:06:34 load avg: 3.31 Re-running failed tests is not supported with --python host runner option.

Total duration: 6 min 34 sec

Click to see traceback logs

Traceback (most recent call last):
File "/Lib/test/test_fstring.py", line 34, in assertAllRaiseeval(str)
MemoryError

[brettcannon]

@pablogsal ☝️ blowing out memory on WASI.

[pablogsal]

@pablogsal point_up blowing out memory on WASI.

On it

[arhadthedev]

For the record: the WASM test got fixed gh-104071.