Assertion failure in Modules/_io/textio.c `_io_TextIOWrapper_tell_impl: Assertion 'skip_back <= PyBytes_GET_SIZE(next_input)' failed`

[YuanchengJiang]

Bug report

Bug description:

import io
import os
from test.support import os_helper

DATA_TEMPLATE = ['line1=1']
DATA_CR = r'\n\r\t'.join(DATA_TEMPLATE) + '\r'

open_func = io.open

READMODE = 'r'
WRITEMODE = 'wb'

data = DATA_CR
if 'b' in WRITEMODE:
 data = data.encode('ascii')

with open_func(os_helper.TESTFN, WRITEMODE) as fp:
 fp.write(data)

with open_func(os_helper.TESTFN, READMODE) as fp:
 _ = fp.readline()
 pos = fp.tell()

python: ../Modules/_io/textio.c:2848: _io_TextIOWrapper_tell_impl: Assertion `skip_back <= PyBytes_GET_SIZE(next_input)' failed.
Aborted (core dumped)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-141314: Fix TextIOWrapper.tell() assertion failure with standalone carriage return #141331
• [3.13] gh-141314: Fix TextIOWrapper.tell() assertion failure with standalone carriage return (GH-141331) #141452
• [3.14] gh-141314: Fix TextIOWrapper.tell() assertion failure with standalone carriage return (GH-141331) #141453

[StanFromIreland]

Smaller repro:

import io
data = b'line1=1\r'
with open('s', 'wb') as fp:
 fp.write(data)
with open('s', 'r') as fp:
 fp.readline()
 fp.tell()

[mohsinm-dev]

@StanFromIreland , Thanks, I reproduced this issue, and I think the assertion failure occurs because the tell() method's optimization algorithm assumes there's buffered input data to search through, but when a file ends with a standalone carriage return (\r), an empty snapshot buffer gets created. The algorithm calculates that it needs to skip some bytes based on the character-to-byte ratio, then tries to navigate within the buffer using a binary search approach, but fails when skip_back=1 exceeds the buffer size of 0. I think the cleanest fix is to detect when next_input is empty and skip the optimization phase entirely, letting the algorithm fall back to the precise byte-by-byte decoding method, which always works correctly. I can open the PR if this seems correct.

[ZeroIntensity]

cc @cmaloney

[cmaloney]

Happy for others to work on, planning to look at myself no earlier than next Tuesday. I've mainly been focusing on BufferdIO which sits below TextIO so not sure the likely code changes needed here.

[yihong0618]

more info:

gdb backtrace

(gdb) set args test.py 
(gdb) run
Starting program: /home/hyi/cpython/python test.py 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

python: ./Modules/_io/textio.c:2848: _io_TextIOWrapper_tell_impl: Assertion `skip_back <= PyBytes_GET_SIZE(next_input)' failed.

Program received signal SIGABRT, Aborted.
__pthread_kill_implementation (threadid=281474842398752, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
44	 return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
(gdb) 
(gdb) bt
#0 __pthread_kill_implementation (threadid=281474842398752, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1 0x0000fffff7df1804 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2 0x0000fffff7da9b3c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3 0x0000fffff7d96074 in __GI_abort () at abort.c:79
#4 0x0000fffff7da3494 in __assert_fail_base (fmt=0xfffff7ec1420 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
 assertion=assertion@entry=0x9f4200 "skip_back <= PyBytes_GET_SIZE(next_input)", 
 file=file@entry=0x9f3760 "./Modules/_io/textio.c", line=line@entry=2848, 
 function=function@entry=0x9f49e0 <__PRETTY_FUNCTION__.1> "_io_TextIOWrapper_tell_impl") at assert.c:123
#5 0x0000fffff7da34ec in __assert_fail (assertion=0x9f4200 "skip_back <= PyBytes_GET_SIZE(next_input)", 
 file=0x9f3760 "./Modules/_io/textio.c", line=2848, function=0x9f49e0 <__PRETTY_FUNCTION__.1> "_io_TextIOWrapper_tell_impl")
 at assert.c:132
#6 0x0000000000841c40 in _io_TextIOWrapper_tell_impl (self=0xfffff79ff890) at ./Modules/_io/textio.c:2848
#7 0x000000000083b09c in _io_TextIOWrapper_tell (self=<_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>, 
 _unused_ignored=0x0) at ./Modules/_io/clinic/textio.c.h:966
#8 0x0000000000508578 in method_vectorcall_NOARGS (func=<method_descriptor at remote 0xfffff77db5f0>, args=0xffffffffa350, 
 nargsf=9223372036854775809, kwnames=0x0) at Objects/descrobject.c:448
#9 0x00000000004f1d60 in _PyObject_VectorcallTstate (tstate=0xbca0e8 <_PyRuntime+358320>, 
 callable=<method_descriptor at remote 0xfffff77db5f0>, args=0xffffffffa350, nargsf=9223372036854775809, kwnames=0x0)
 at ./Include/internal/pycore_call.h:169
#10 0x00000000004f2ce8 in PyObject_Vectorcall (callable=<method_descriptor at remote 0xfffff77db5f0>, args=0xffffffffa350, 
 nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:327
#11 0x00000000006bcd30 in _PyEval_EvalFrameDefault (tstate=0xbca0e8 <_PyRuntime+358320>, frame=0xfffff7cf5020, throwflag=0)
 at Python/generated_cases.c.h:1620
#12 0x00000000006b39d0 in _PyEval_EvalFrame (tstate=0xbca0e8 <_PyRuntime+358320>, frame=0xfffff7cf5020, throwflag=0)
 at ./Include/internal/pycore_ceval.h:121
#13 0x00000000006e9138 in _PyEval_Vector (tstate=0xbca0e8 <_PyRuntime+358320>, func=0xfffff782ab10, 
 locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, args=0x0, argcount=0, kwnames=0x0) at Python/ceval.c:2005
#14 0x00000000006b5fd4 in PyEval_EvalCode (co=<code at remote 0xc63ec0>, 
--Type <RET> for more, q to quit, c to continue without paging--
 globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, 
 locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}) at Python/ceval.c:888
#15 0x00000000007970c4 in run_eval_code_obj (tstate=0xbca0e8 <_PyRuntime+358320>, co=0xc63ec0, 
 globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, 
 locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}) at Python/pythonrun.c:1365
#16 0x0000000000797430 in run_mod (mod=0xd71880, filename='/home/hyi/cpython/test.py', 
 globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, 
 locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, flags=0xffffffffe3b8, arena=0xfffff78a5780, interactive_src=0x0, 
 generate_new_source=0) at Python/pythonrun.c:1459
#17 0x0000000000796e80 in pyrun_file (fp=0xc3f2f0, filename='/home/hyi/cpython/test.py', start=257, 
 globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': --Type <RET> for more, q to quit, c to continue without paging--
<_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, 
 locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <SourceFileLoader(name='__main__', path='/home/hyi/cpython/test.py') at remote 0xfffff79ad460>, '__spec__': None, '__builtins__': <module at remote 0xfffff79c4320>, '__file__': '/home/hyi/cpython/test.py', '__cached__': None, 'io': <module at remote 0xfffff78869f0>, 'data': b'line1=1\r', 'fp': <_io.TextIOWrapper(mode='r') at remote 0xfffff79ff890>}, closeit=1, flags=0xffffffffe3b8) at Python/pythonrun.c:1293
#18 0x0000000000795428 in _PyRun_SimpleFileObject (fp=0xc3f2f0, filename='/home/hyi/cpython/test.py', closeit=1, 
 flags=0xffffffffe3b8) at Python/pythonrun.c:521
#19 0x000000000079460c in _PyRun_AnyFileObject (fp=0xc3f2f0, filename='/home/hyi/cpython/test.py', closeit=1, 
 flags=0xffffffffe3b8) at Python/pythonrun.c:81
#20 0x00000000007cf47c in pymain_run_file_obj (program_name='/home/hyi/cpython/python', filename='/home/hyi/cpython/test.py', 
 skip_source_first_line=0) at Modules/main.c:410
#21 0x00000000007cf52c in pymain_run_file (config=0xb951b8 <_PyRuntime+141440>) at Modules/main.c:429
#22 0x00000000007cfd70 in pymain_run_python (exitcode=0xffffffffe53c) at Modules/main.c:691
#23 0x00000000007cfea8 in Py_RunMain () at Modules/main.c:772
#24 0x00000000007cff44 in pymain_main (args=0xffffffffe5c0) at Modules/main.c:802
#25 0x00000000007cffb8 in Py_BytesMain (argc=2, argv=0xffffffffe778) at Modules/main.c:826
#26 0x0000000000420c60 in main (argc=2, argv=0xffffffffe778) at ./Programs/python.c:15