Skip to content

gh-132917: Use /proc/self/status for mem usage info. - #133544

Merged
nascheme merged 3 commits into
python:mainfrom
nascheme:gh-132917-use-proc-status
May 8, 2025
Merged

gh-132917: Use /proc/self/status for mem usage info.#133544
nascheme merged 3 commits into
python:mainfrom
nascheme:gh-132917-use-proc-status

Conversation

@nascheme

@naschemenascheme commented May 6, 2025

Copy link
Copy Markdown
Member

Using smaps_rollup is slower and we can get the similar info from /proc/self/status. We don't need the extra accuracy that smaps_rollup is giving.

Profiling is showing that reading the smaps_rollup file is taking on the order of 30 ms. Reading status is much faster. Some background detail on this difference:

https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/10966#note_410194443

Using smaps_rollup is quite a lot slower and we can get the similar info
from /proc/self/status.
@bedevere-bot

Copy link
Copy Markdown

🤖 New build scheduled with the buildbot fleet by @nascheme for commit 5b3621d 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133544%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

@bedevere-botbedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label May 6, 2025
@naschemenascheme added performance Performance or resource usage topic-free-threading labels May 7, 2025
@nascheme

nascheme commented May 7, 2025

Copy link
Copy Markdown
MemberAuthor

Some more precise run time comparison. With the script from GH-132917 running in 10 parallel threads, the /proc/self/smaps_rollup version takes 3000 us/per call and the /proc/self/status version uses 15 us/per call.

I also tested the MacOS version and it takes 4.5 us/per call. The full GC pass is taking roughly 70 ms.

@nascheme

Copy link
Copy Markdown
MemberAuthor

Comparing the info from /proc/self/smaps_rollup vs /proc/self/status. This is on a Linux 6.1.0 kernel with 16 GB of RAM and 16 GB of swap. I ran a Python program that just allocates a bunch of memory in a loop while printing the proc info.

Rss:Swap:VmRSS:VmSwap:
2,361,87602,361,6840
4,714,09204,713,9240
7,066,30807,066,1640
9,418,58809,418,4040
11,770,800011,770,6440
14,122,420014,122,2880
14,846,5521,625,61214,845,6041,625,612
15,439,2523,384,87215,478,7083,345,056

@Yhg1sYhg1s left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a more efficient approach and the change is pretty simple... But unless @hugovk wants to reopen the release branch I think it should just go into beta 2. (Also I still have the same concerns I mentioned before, but they're not pressing enough to worry about for b1.)

@hugovk

Copy link
Copy Markdown
Member

Let's keep this for b2, thanks!

@nascheme
nascheme enabled auto-merge (squash) May 7, 2025 16:53
@nascheme
nascheme merged commit 751db4e into python:mainMay 8, 2025
@naschemenascheme added the needs backport to 3.14 bugs and security fixes label May 8, 2025
@miss-islington-app

Copy link
Copy Markdown

Thanks @nascheme for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 8, 2025
…133544)
On Linux, use /proc/self/status for mem usage info. Using smaps_rollup is quite a lot slower and
we can get the similar info from /proc/self/status.
(cherry picked from commit 751db4e)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
@bedevere-app

Copy link
Copy Markdown

GH-133718 is a backport of this pull request to the 3.14 branch.

@bedevere-appbedevere-appBot removed the needs backport to 3.14 bugs and security fixes label May 8, 2025
nascheme added a commit that referenced this pull request May 8, 2025
… (gh-133718)
On Linux, use /proc/self/status for mem usage info. Using smaps_rollup is quite a lot slower and
we can get the similar info from /proc/self/status.
(cherry picked from commit 751db4e)
Co-authored-by: Neil Schemenauer <nas-github@arctrix.com>
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
…3544)
On Linux, use /proc/self/status for mem usage info. Using smaps_rollup is quite a lot slower and
we can get the similar info from /proc/self/status.
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
…3544)
On Linux, use /proc/self/status for mem usage info. Using smaps_rollup is quite a lot slower and
we can get the similar info from /proc/self/status.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performancePerformance or resource usageskip newstopic-free-threading

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@nascheme@bedevere-bot@hugovk@Yhg1s