Skip to content

gh-141311: Avoid assertion in BytesIO readinto - #141333

Merged
serhiy-storchaka merged 9 commits into
python:mainfrom
cmaloney:bytesio_seek_far
Nov 12, 2025
Merged

gh-141311: Avoid assertion in BytesIO readinto#141333
serhiy-storchaka merged 9 commits into
python:mainfrom
cmaloney:bytesio_seek_far

Conversation

@cmaloney

@cmaloneycmaloney commented Nov 10, 2025•
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Contributor

Account for when self->pos is equal to PY_SSIZE_T_MAX. The length of read was correctly set to zero but the asserts assumed self->pos couldn't reach PY_SSIZE_T_MAX. Return early to avoid edge cases.

@cmaloney
pythongh-141311: Avoid assertion in BytesIO readinto
db81df6
Account for when self->pos is equal to PY_SSIZE_T_MAX. The length of
read was correctly set to zero but the asserts assumed
self->pos couldn't reach PY_SSIZE_T_MAX. Return early to avoid edge
cases.
@bedevere-appbedevere-appBot mentioned this pull request Nov 10, 2025
Closed
serhiy-storchaka
serhiy-storchaka reviewed Nov 10, 2025
View reviewed changes
Comment threadModules/_io/bytesio.c Outdated
Comment threadLib/test/test_io/test_memoryio.py Outdated
Comment threadMisc/NEWS.d/next/Library/2025-11-09-18-55-13.gh-issue-141311.qZ3swc.rst Outdated
@cmaloney
Simplify by using sys.maxsize
2c84efe
@cmaloney
Reference io.BytesIO.readinto directly
d119904
@cmaloney
Resolve read_bytes_lock_held and audit other cases
d78cc84
For `read_bytes_lock_held` the `size` parameter is always set to 0 by calling
code currently if self->pos > self->string_size. I added a range check for
self->pos as an extra safety as codepaths chagne but can remove if it's too much.
`write_bytes_lock_held`: `endpos` is a `size_t` so can hold
`2 * PY_SSIZE_T_MAX`. Value is bounds checked in `resize_buffer_lock_held`.
A number of cases use `scan_eol_lock_held` to move forward. That has code which
checks `self->pos >= self->string_size` and returns `0`. Callsites all seem to
handle that correctly.
@cmaloney

cmaloney commented Nov 11, 2025

Copy link
Copy Markdown
ContributorAuthor

I audited the codepaths which did self->pos + buffer.

For read_bytes_lock_held the size parameter is always set to 0 by calling code currently if self->pos > self->string_size. I added a range check for self->pos as an extra safety as codepaths change but can remove if it's too much.

write_bytes_lock_held: endpos is a size_t so can hold 2 * PY_SSIZE_T_MAX. Value is bounds checked in resize_buffer_lock_held.

A number of cases use scan_eol_lock_held to move forward. That has code which checks self->pos >= self->string_size and returns 0. Callers seem to handle that correctly (early exit or call read_bytes_lock_held)..

serhiy-storchaka
serhiy-storchaka reviewed Nov 11, 2025
View reviewed changes
Comment threadModules/_io/bytesio.c Outdated
Comment threadModules/_io/bytesio.c Outdated
Comment threadModules/_io/bytesio.c Outdated
serhiy-storchaka
serhiy-storchaka reviewed Nov 11, 2025
View reviewed changes
Comment threadModules/_io/bytesio.c Outdated
serhiy-storchaka
serhiy-storchaka approved these changes Nov 11, 2025
View reviewed changes

@serhiy-storchakaserhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. 👍

@serhiy-storchaka
serhiy-storchaka merged commit 7d54374 into python:mainNov 12, 2025
50 checks passed
@serhiy-storchakaserhiy-storchaka added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Nov 12, 2025
@miss-islington-app

miss-islington-appBot commented Nov 12, 2025

Copy link
Copy Markdown

Thanks @cmaloney for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

@miss-islington-app

miss-islington-appBot commented Nov 12, 2025

Copy link
Copy Markdown

Thanks @cmaloney for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

@miss-islington-app

miss-islington-appBot commented Nov 12, 2025

Copy link
Copy Markdown

Sorry, @cmaloney and @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 7d54374f9c7d91e0ef90c4ad84baf10073cf1d8a 3.13

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Nov 12, 2025
@cmaloney@miss-islington
pythongh-141311: Avoid assertion in BytesIO.readinto() (pythonGH-141333)
7151cfb
Fix error in assertion which causes failure if pos is equal to PY_SSIZE_T_MAX.
Fix undefined behavior in read() and readinto() if pos is larger that the size
of the underlying buffer.
(cherry picked from commit 7d54374)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
@bedevere-app

bedevere-appBot commented Nov 12, 2025

Copy link
Copy Markdown

GH-141457 is a backport of this pull request to the 3.14 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.14 bugs and security fixes label Nov 12, 2025
@serhiy-storchaka

serhiy-storchaka commented Nov 12, 2025

Copy link
Copy Markdown
Member

@cmaloney, could you please create a backport to 3.13 manually?

serhiy-storchaka pushed a commit that referenced this pull request Nov 12, 2025
@miss-islington@cmaloney
[3.14]gh-141311: Avoid assertion in BytesIO.readinto() (GH-141333) (G…
75b5157
…H-141457)
Fix error in assertion which causes failure if pos is equal to PY_SSIZE_T_MAX.
Fix undefined behavior in read() and readinto() if pos is larger that the size
of the underlying buffer.
(cherry picked from commit 7d54374)
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
@cmaloney
cmaloney deleted the bytesio_seek_far branch November 12, 2025 19:57
@bedevere-app

bedevere-appBot commented Nov 12, 2025

Copy link
Copy Markdown

GH-141478 is a backport of this pull request to the 3.13 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.13 bugs and security fixes label Nov 12, 2025
serhiy-storchaka pushed a commit that referenced this pull request Nov 12, 2025
@cmaloney
[3.13]gh-141311: Avoid assertion in BytesIO.readinto() (GH-141333) (G…
7c62bd5
…H-141478)
Fix error in assertion which causes failure if pos is equal to PY_SSIZE_T_MAX.
Fix undefined behavior in read() and readinto() if pos is larger that the size
of the underlying buffer.
(cherry picked from commit 7d54374)
StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
@cmaloney@StanFromIreland
pythongh-141311: Avoid assertion in BytesIO.readinto() (pythonGH-141333)
7592d29
Fix error in assertion which causes failure if pos is equal to PY_SSIZE_T_MAX.
Fix undefined behavior in read() and readinto() if pos is larger that the size
of the underlying buffer.
@serhiy-storchakaserhiy-storchaka removed their assignment Jul 15, 2026
@matthiasgoergensmatthiasgoergens mentioned this pull request Aug 5, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchakaserhiy-storchakaserhiy-storchaka approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cmaloney@serhiy-storchaka