Skip to content

gh-135557: Use atomic writes on heapq operations - #135601

Merged
kumaraditya303 merged 13 commits into
python:mainfrom
xuantengh:heap-list-lock-free-read
Jun 21, 2025
Merged

gh-135557: Use atomic writes on heapq operations#135601
kumaraditya303 merged 13 commits into
python:mainfrom
xuantengh:heap-list-lock-free-read

Conversation

@xuantengh

@xuantenghxuantengh commented Jun 17, 2025•
edited by bedevere-app Bot
Loading

Copy link
Copy Markdown
Contributor

Use atomic writes on heapq operation.

@bedevere-appbedevere-appBot mentioned this pull request Jun 17, 2025
Closed
@xuantenghxuantengh changed the title gh-135557: Fix lock-free list access on heapqgh-135557: Use atomic writes on heapq operationsJun 17, 2025
@xuantengh

xuantengh commented Jun 17, 2025

Copy link
Copy Markdown
ContributorAuthor

I think this could be "skip news"?

@rhettinger
rhettinger removed their request for review June 17, 2025 12:51
@ZeroIntensity

ZeroIntensity commented Jun 17, 2025

Copy link
Copy Markdown
Member

I think this could be "skip news"?

This is user-facing, so this does need a news. Something like "Fix race with lists and heapq on the free threaded build" would be fine. It would also be good to get a test. (Sorry for pushing back on both your PRs!)

@ZeroIntensityZeroIntensity added the needs backport to 3.14 bugs and security fixes label Jun 17, 2025
ZeroIntensity
ZeroIntensity reviewed Jun 17, 2025
View reviewed changes
Comment threadModules/_heapqmodule.c Outdated
@xuantengh
xuantenghforce-pushed the heap-list-lock-free-read branch from f7fbb68 to 08770ecCompareJune 17, 2025 16:16
ZeroIntensity
ZeroIntensity reviewed Jun 17, 2025
View reviewed changes
Comment threadMisc/NEWS.d/next/Library/2025-06-17-23-13-56.gh-issue-135557.Bfcy4v.rst Outdated
Comment threadLib/test/test_heapq.py Outdated
Comment threadLib/test/test_heapq.py Outdated
Comment threadLib/test/test_heapq.py Outdated
Comment threadLib/test/test_heapq.py Outdated
kumaraditya303
kumaraditya303 reviewed Jun 17, 2025
View reviewed changes
Comment threadModules/_heapqmodule.c Outdated
@python-cla-bot

python-cla-botBot commented Jun 18, 2025•
edited
Loading

Copy link
Copy Markdown

All commit authors signed the Contributor License Agreement.

CLA signed

@xuantengh
xuantenghforce-pushed the heap-list-lock-free-read branch from 0184b21 to 463963dCompareJune 18, 2025 06:38
kumaraditya303
kumaraditya303 reviewed Jun 18, 2025
View reviewed changes
Comment threadLib/test/test_free_threading/test_heapq.py Outdated
kumaraditya303
kumaraditya303 reviewed Jun 18, 2025
View reviewed changes
Comment threadModules/_heapqmodule.c Outdated
@xuantengh
xuantenghforce-pushed the heap-list-lock-free-read branch from 463963d to c49ff7dCompareJune 19, 2025 00:48
kumaraditya303
kumaraditya303 reviewed Jun 19, 2025
View reviewed changes
Comment threadModules/_heapqmodule.c
@xuantengh
xuantenghforce-pushed the heap-list-lock-free-read branch from c49ff7d to 27d6cb6CompareJune 19, 2025 10:31
ZeroIntensity
ZeroIntensity reviewed Jun 19, 2025
View reviewed changes
Comment threadLib/test/test_free_threading/test_heapq.py Outdated
Comment threadLib/test/test_free_threading/test_heapq.py Outdated
@xuantengh
xuantenghforce-pushed the heap-list-lock-free-read branch from 2cc859d to 8cf4794CompareJune 19, 2025 13:17
@xuantengh
xuantenghforce-pushed the heap-list-lock-free-read branch from 8cf4794 to 32d767fCompareJune 19, 2025 13:42
kumaraditya303
kumaraditya303 reviewed Jun 19, 2025
View reviewed changes
Comment threadModules/_heapqmodule.c Outdated
@ZeroIntensity

ZeroIntensity commented Jun 20, 2025

Copy link
Copy Markdown
Member

!buildbot nogil

@bedevere-bot

bedevere-bot commented Jun 20, 2025

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @ZeroIntensity for commit a10560e 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F135601%2Fmerge

The command will test the builders whose names match following regular expression: nogil

The builders matched are:

  • ARM64 MacOS M1 NoGIL PR
  • AMD64 Windows Server 2022 NoGIL PR
  • aarch64 Fedora Rawhide NoGIL refleaks PR
  • AMD64 Fedora Rawhide NoGIL PR
  • aarch64 Fedora Rawhide NoGIL PR
  • ARM64 MacOS M1 Refleaks NoGIL PR
  • AMD64 CentOS9 NoGIL Refleaks PR
  • AMD64 Fedora Rawhide NoGIL refleaks PR
  • s390x Fedora Rawhide NoGIL refleaks PR
  • PPC64LE Fedora Rawhide NoGIL PR
  • PPC64LE Fedora Rawhide NoGIL refleaks PR
  • x86-64 MacOS Intel NoGIL PR
  • AMD64 CentOS9 NoGIL PR
  • x86-64 MacOS Intel ASAN NoGIL PR
  • AMD64 Windows PGO NoGIL PR
  • s390x Fedora Rawhide NoGIL PR

ZeroIntensity
ZeroIntensity approved these changes Jun 20, 2025
View reviewed changes

@ZeroIntensityZeroIntensity left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as well, assuming buildbots pass.

@kumaraditya303
kumaraditya303 merged commit 13cac83 into python:mainJun 21, 2025
@miss-islington-app

miss-islington-appBot commented Jun 21, 2025

Copy link
Copy Markdown

Thanks @xuantengh for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 21, 2025
@xuantengh@miss-islington
pythongh-135557: use atomic stores inheapqoperations in free-thre…
753cec2
…ading (pythonGH-135601)
(cherry picked from commit 13cac83)
Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com>
@bedevere-app

bedevere-appBot commented Jun 21, 2025

Copy link
Copy Markdown

GH-135787 is a backport of this pull request to the 3.14 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.14 bugs and security fixes label Jun 21, 2025
YvesDup
YvesDup reviewed Jun 21, 2025
View reviewed changes
Comment threadLib/test/test_free_threading/test_heapq.py
except IndexError:
pass

self.run_concurrently(worker, (), n_threads * 2)

@YvesDupYvesDupJun 21, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test is successful here because the worker function has no arguments.
But shouldn't the second argument args be a tuple?

@kumaraditya303kumaraditya303Jun 21, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it an empty tuple here?

kumaraditya303 pushed a commit that referenced this pull request Jun 21, 2025
@miss-islington@xuantengh
[3.14]gh-135557: use atomic stores inheapqoperations in free-thr…
e388b29
…eading (GH-135601) (#135787)
gh-135557: use atomic stores in `heapq` operations in free-threading (GH-135601)
(cherry picked from commit 13cac83)
Co-authored-by: Xuanteng Huang <44627253+xuantengh@users.noreply.github.com>
@YvesDup

YvesDup commented Jun 23, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

isn't it an empty tuple here?

Oh yes it is.
Reading an empty tuple for arguments when creating new threads destabilized me (and then silly). This is because when creating a new Thread with no arguments, the args argument is always omitted (the default is an empty tuple).
Sorry for the noise.

@xuantengh
xuantengh deleted the heap-list-lock-free-read branch June 23, 2025 16:26
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@xuantengh@Pranjal095
pythongh-135557: use atomic stores inheapqoperations in free-thre…
442de3b
…ading (python#135601)
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@xuantengh@taegyunkim
pythongh-135557: use atomic stores inheapqoperations in free-thre…
bb95a55
…ading (python#135601)
Agent-Hellboy pushed a commit to Agent-Hellboy/cpython that referenced this pull request Aug 19, 2025
@xuantengh@Agent-Hellboy
pythongh-135557: use atomic stores inheapqoperations in free-thre…
a25c15e
…ading (python#135601)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mpagempagempage left review comments
@kumaraditya303kumaraditya303kumaraditya303 approved these changes
@ZeroIntensityZeroIntensityZeroIntensity approved these changes
+1 more reviewer
@YvesDupYvesDupYvesDup left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

topic-free-threading

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@xuantengh@ZeroIntensity@bedevere-bot@YvesDup@mpage@kumaraditya303