Skip to content

GH-119169: Speed up os.walk(topdown=False) - #119186

Closed
barneygale wants to merge 12 commits into
python:mainfrom
barneygale:os-walk-speedup
Closed

GH-119169: Speed up os.walk(topdown=False)#119186
barneygale wants to merge 12 commits into
python:mainfrom
barneygale:os-walk-speedup

Conversation

@barneygale

@barneygalebarneygale commented May 19, 2024

Copy link
Copy Markdown
Contributor

Add entries to the stack while iterating over os.scandir() results, rather than afterwards.

Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`. In top-down mode, call
`os.path.join()` at most once per iteration.
@vstinner

Copy link
Copy Markdown
Member

cc @serhiy-storchaka

@barneygale
barneygale marked this pull request as draft May 23, 2024 08:46
@barneygale

Copy link
Copy Markdown
ContributorAuthor

#119473 fixes the test failure.

@barneygalebarneygale changed the title GH-119169: Speed up os.walk()GH-119169: Speed up os.[f]walk()May 30, 2024
barneygale added a commit to barneygale/cpython that referenced this pull request Jul 6, 2024
When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
@barneygalebarneygale changed the title GH-119169: Speed up os.[f]walk()GH-119169: Speed up os.walk()Jul 6, 2024
barneygale added a commit to barneygale/cpython that referenced this pull request Jul 6, 2024
Handle errors from `os.scandir()` and `ScandirIterator` similarly, which
lets us loop over directory entries with `for`.
@barneygalebarneygale changed the title GH-119169: Speed up os.walk()GH-119169: Speed up os.walk(topdown=False)Jul 6, 2024
barneygale added a commit to barneygale/cpython that referenced this pull request Jul 8, 2024
@barneygale

Copy link
Copy Markdown
ContributorAuthor

Closing because reversing sibling traversal order isn't kosher - see discussion on #119473.

encukou added a commit to barneygale/cpython that referenced this pull request Sep 2, 2025
encukou pushed a commit that referenced this pull request Sep 3, 2025
GH-119186: Slightly speed up `os.walk(topdown=True)`
When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
lkollar pushed a commit to lkollar/cpython that referenced this pull request Sep 9, 2025
…121431)
pythonGH-119186: Slightly speed up `os.walk(topdown=True)`
When `os.walk()` traverses into subdirectories in top-down mode, call
`os.path.join()` once to add a trailing slash, and use string concatenation
thereafter to generate child paths.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performancePerformance or resource usage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@barneygale@vstinner