Interpreter exits on Windows due to ValueError raised in linecache.py

[devdanzin]

Bug report

Bug description:

It's possible to make the interpreter exit on Windows in 3.13.0b4 and main due to ValueError: stat: path too long for Windows being raised by os.stat() in updatecache in linecache.py when trying to print a traceback:

>>> exec(compile("print(2**100000)", "s"*99999, "exec"))
Exception ignored in the internal traceback machinery:
Traceback (most recent call last):
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 139, in _print_exception_bltin
 return print_exception(exc, limit=BUILTIN_EXCEPTION_LIMIT, file=file, colorize=colorize)
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 129, in print_exception
 te = TracebackException(type(value), value, tb, limit=limit, compact=True)
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 1133, in __init__
 context = TracebackException(
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 1043, in __init__
 self.stack = StackSummary._extract_from_extended_frame_gen(
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 492, in _extract_from_extended_frame_gen
 f.line
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 369, in line
 self._set_lines()
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 350, in _set_lines
 lines.append(linecache.getline(self.filename, lineno).rstrip())
 File "~\PycharmProjects\cpython\Lib\linecache.py", line 25, in getline
 lines = getlines(filename, module_globals)
 File "~\PycharmProjects\cpython\Lib\linecache.py", line 41, in getlines
 return updatecache(filename, module_globals)
 File "~\PycharmProjects\cpython\Lib\linecache.py", line 100, in updatecache
 stat = os.stat(fullname)
ValueError: stat: path too long for Windows
Traceback (most recent call last):
 File "~\PycharmProjects\cpython\Lib\code.py", line 91, in runcode
 exec(code, self.locals)
 File "<python-input-2>", line 1, in <module>
 File "sss[...]sssssss", line 1, in <module>
ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "~\PycharmProjects\cpython\Lib\runpy.py", line 198, in _run_module_as_main
 return _run_code(code, main_globals, None,
 File "~\PycharmProjects\cpython\Lib\runpy.py", line 88, in _run_code
 exec(code, run_globals)
 File "~\PycharmProjects\cpython\Lib\_pyrepl\__main__.py", line 6, in <module>
 __pyrepl_interactive_console()
 File "~\PycharmProjects\cpython\Lib\_pyrepl\main.py", line 59, in interactive_console
 run_multiline_interactive_console(console)
 File "~\PycharmProjects\cpython\Lib\_pyrepl\simple_interact.py", line 156, in run_multiline_interactive_console
 more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single") # type: ignore[call-arg]
 File "~\PycharmProjects\cpython\Lib\code.py", line 303, in push
 more = self.runsource(source, filename, symbol=_symbol)
 File "~\PycharmProjects\cpython\Lib\_pyrepl\console.py", line 200, in runsource
 self.runcode(code)
 File "~\PycharmProjects\cpython\Lib\code.py", line 95, in runcode
 self.showtraceback()
 File "~\PycharmProjects\cpython\Lib\_pyrepl\console.py", line 168, in showtraceback
 super().showtraceback(colorize=self.can_colorize)
 File "~\PycharmProjects\cpython\Lib\code.py", line 147, in showtraceback
 lines = traceback.format_exception(ei[0], ei[1], last_tb.tb_next, colorize=colorize)
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 154, in format_exception
 te = TracebackException(type(value), value, tb, limit=limit, compact=True)
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 1043, in __init__
 self.stack = StackSummary._extract_from_extended_frame_gen(
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 492, in _extract_from_extended_frame_gen
 f.line
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 369, in line
 self._set_lines()
 File "~\PycharmProjects\cpython\Lib\traceback.py", line 350, in _set_lines
 lines.append(linecache.getline(self.filename, lineno).rstrip())
 File "~\PycharmProjects\cpython\Lib\linecache.py", line 25, in getline
 lines = getlines(filename, module_globals)
 File "~\PycharmProjects\cpython\Lib\linecache.py", line 41, in getlines
 return updatecache(filename, module_globals)
 File "~\PycharmProjects\cpython\Lib\linecache.py", line 100, in updatecache
 stat = os.stat(fullname)
ValueError: stat: path too long for Windows
[Thread 30100.0x6108 exited with code 1]
[Thread 30100.0x5da4 exited with code 1]
[Thread 30100.0x90a0 exited with code 1]
[Inferior 1 (process 30100) exited with code 01]

CPython versions tested on:

3.13, CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-122170: Fix interpreter exiting due to ValueError from os.stat for too long filename on Windows #122175
• gh-122170: Handle ValueError raised by os.stat in linecache #122176
• [3.13] gh-122170: Handle ValueError raised by os.stat() in linecache (GH-122176) #122348
• [3.12] gh-122170: Handle ValueError raised by os.stat() in linecache (GH-122176) #122349

[picnixz]

I'm surprised that os.stat raises a ValueError in this case actually and not an OSError. I'll make a PR for handling this case in linecache.

[devdanzin]

I'll make a PR for handling this case in linecache.

I have one ready.

[picnixz]

Oh I was just going to push it now xD Go for yours then, I'll review it.

[devdanzin]

Yours is much more complete and polished, is it OK if I close mine?

[picnixz]

Err... it's up to you.

[devdanzin]

I'll close it, I'd rather have code of higher quality in the standard library than code I wrote :)

[eryksun]

I'm surprised that os.stat raises a ValueError in this case actually and not an OSError. I'll make a PR for handling this case in linecache.

For some reason, code was added to the path_t converter to raise ValueError if the path length exceeds 32767 characters. The length limit is actually 32766 characters, because the terminating null must be included in the buffer size. Properly, the expression UNICODE_STRING_MAX_CHARS - 1 should be used instead of 32766. But really, a path_t using function should just let the system API fail and raise an OSError, either for the system error code ERROR_FILENAME_EXCED_RANGE or ERROR_PATH_NOT_FOUND, depending on the system API function that's called.

cpython/Modules/posixmodule.c

Lines 1331 to 1336 in 2c1b1e7

	#ifdef MS_WINDOWS
	if (!path->nonstrict && length > 32767) {
	FORMAT_EXCEPTION(PyExc_ValueError, "%s too long for Windows");
	goto error_exit;
	}
	#endif

cpython/Modules/posixmodule.c

Lines 1398 to 1403 in 2c1b1e7

	#ifdef MS_WINDOWS
	if (!path->nonstrict && length > 32767) {
	FORMAT_EXCEPTION(PyExc_ValueError, "%s too long for Windows");
	goto error_exit;
	}
	#endif

[ZeroIntensity]

I get a different error when using the reproducer on Linux:

ValueError: Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/runpy.py", line 198, in _run_module_as_main
 return _run_code(code, main_globals, None,
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/runpy.py", line 88, in _run_code
 exec(code, run_globals)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/__main__.py", line 6, in <module>
 __pyrepl_interactive_console()
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/main.py", line 57, in interactive_console
 run_multiline_interactive_console(console)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/simple_interact.py", line 156, in run_multiline_interactive_console
 more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single") # type: ignore[call-arg]
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/code.py", line 303, in push
 more = self.runsource(source, filename, symbol=_symbol)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/console.py", line 200, in runsource
 self.runcode(code)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/code.py", line 95, in runcode
 self.showtraceback()
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/_pyrepl/console.py", line 168, in showtraceback
 super().showtraceback(colorize=self.can_colorize)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/code.py", line 147, in showtraceback
 lines = traceback.format_exception(ei[0], ei[1], last_tb.tb_next, colorize=colorize)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 155, in format_exception
 return list(te.format(chain=chain, colorize=colorize))
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 1384, in format
 yield from _ctx.emit(exc.stack.format(colorize=colorize))
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 747, in format
 formatted_frame = self.format_frame_summary(frame_summary, colorize=colorize)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 583, in format_frame_summary
 show_carets = self._should_show_carets(start_offset, end_offset, all_lines, anchors)
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 701, in _should_show_carets
 statement = tree.body[0]
IndexError: list index out of range

CC @picnixz, this might be a separate issue. This still occurs on both PRs.

[devdanzin]

I get a different error when using the reproducer on Linux:

[...]
 File "/home/zero/Desktop/Projects/Python/cpython/Lib/traceback.py", line 701, in _should_show_carets
 statement = tree.body[0]
IndexError: list index out of range

That's #122145.

[picnixz]

Which is fixed by #122161

[picnixz]

@eryksun I see the ValueError was implemented in #118355, which is recent enough. However, I think that os.stat raising an OSError would be more correct because on Linux, this is what happens: OSError: [Errno 36] File name too long: ' but on Windows, IIRC, it should be a "FileNotFound" (but I have no way to check this now). Do you think I should open another issue for os.stat ? (and maybe more functions?)