gh-89727: Fix pathlib.Path.walk RecursionError on deep trees by zmievsa · Pull Request #100282 · python/cpython

zmievsa · 2022-12-15T22:46:49Z

Use a stack to implement pathlib.Path.walk iteratively instead of recursively to avoid hitting recursion limits on deeply nested trees.

Issue: Change shutil.rmtree and os.walk to support very deep hierarchies #89727

netlify · 2022-12-15T22:46:53Z

✅ Deploy Preview for python-cpython-preview canceled.

Name	Link
🔨 Latest commit	`2bfc47d`
🔍 Latest deploy log	https://app.netlify.com/sites/python-cpython-preview/deploys/639c50140be628000884140d

zmievsa · 2022-12-15T22:53:08Z

TODO:

Add a news blip
Add a test that checks for deeply nested directories
Figure out a better name for is_result

Update:
All done!

…on-depth

…f github.com:ovsyanka83/cpython into pythongh-89727/fix-pathlib.Path.walk-recursion-depth

brettcannon · 2022-12-16T19:23:49Z

/cc @barneygale

…on-depth

carljm

LGTM. I assume because of the initial underscore we don't have to worry about people who may have subclassed Path and overridden the _walk method (now to no effect.)

zmievsa · 2022-12-20T22:47:56Z

@carljm remember that Path.walk has only been created in 3.12 which means that 99% of libraries/projects do not and cannot depend on it yet.

zmievsa · 2022-12-24T10:38:09Z

@barneygale any estimates of when you will have time to take a look at it? No pressure or rush though.

barneygale · 2022-12-24T15:10:14Z

I'll look at this within the next few days!

Co-authored-by: Barney Gale <barney.gale@gmail.com>

barneygale

Looks good to me!

zmievsa · 2023-01-11T21:57:42Z

@AlexWaygood what would be the next steps for this PR? I am ready to continue improving it

…on-depth

AlexWaygood · 2023-01-11T21:59:58Z

Sorry @Ovsyanka83, I'll try to take a proper look soon!

Co-authored-by: Brett Cannon <brett@python.org>

barneygale · 2023-02-01T15:37:37Z

@AlexWaygood gentle nudge to review this when you have the time! Thanks :)

CuriousLearner

LGTM.

Thanks for your contribution.

barneygale · 2023-02-28T16:33:45Z

@AlexWaygood https://youtu.be/CdqoNKCCt7A

zmievsa · 2023-03-22T18:13:46Z

@barneygale thanks!

Wow. Have you received merge rights? Congrats! Did they make you a core developer or is that a special rule just for you and pathlib?

AlexWaygood · 2023-03-22T18:31:57Z

We made him a core dev! https://discuss.python.org/t/vote-to-promote-barney-gale/24801

barneygale · 2023-03-22T20:04:51Z

@Ovsyanka83 thanks for your patience and for working on this, it's a neat patch :). I'm hoping to build an iterative version of glob() atop!

…ythonGH-100282) Use a stack to implement `pathlib.Path.walk()` iteratively instead of recursively to avoid hitting recursion limits on deeply nested trees. Co-authored-by: Barney Gale <barney.gale@gmail.com> Co-authored-by: Brett Cannon <brett@python.org>

Use `Path.walk()` to implement the recursive wildcard `**`. This method uses an iterative (rather than recursive) walk - see pythonGH-100282.

Use `Path.walk()` to implement the recursive wildcard `**`. This method uses an iterative (rather than recursive) walk - see GH-100282.

Replace recursion with iteration
037a717

bedevere-bot mentioned this pull request Dec 15, 2022
Change shutil.rmtree and os.walk to support very deep hierarchies #89727
Open

bedevere-bot added the awaiting review label Dec 15, 2022

Unify walk and _walk methods
f8968b6

zmievsaand others added 5 commits December 16, 2022 03:10

Remove empty typehint
ecabf8c

Fix all tests
6ec3c3e

Add test for recursion limit
fb5a4c7

📜🤖 Added by blurb_it.
55beb6e

Merge branch 'main' intopythongh-89727/fix-pathlib.Path.walk-recursi…
7f36140
…on-depth

zmievsa marked this pull request as ready for review December 16, 2022 10:32

zmievsa requested a review from brettcannon as a code owner December 16, 2022 10:32

zmievsa added 2 commits December 16, 2022 15:01

Fix recursion limit issue in different envs
508bd4d

Merge branch 'pythongh-89727/fix-pathlib.Path.walk-recursion-depth' o…
2bfc47d
…f github.com:ovsyanka83/cpython into pythongh-89727/fix-pathlib.Path.walk-recursion-depth

pitrou requested a review from serhiy-storchaka December 16, 2022 14:27

brettcannon added the topic-pathlib label Dec 16, 2022

Make everything consistent with os.walk
20476ff

kalvdans reviewed Dec 20, 2022
View reviewed changes

Comment threadLib/pathlib.py Outdated

Remove unused import
4f2de62

carljm reviewed Dec 20, 2022
View reviewed changes

Comment threadLib/test/test_pathlib.py Outdated

zmievsa added 2 commits December 21, 2022 01:47

Merge branch 'main' intopythongh-89727/fix-pathlib.Path.walk-recursi…
6313b6a
…on-depth

Replace the usage of infinite_recursion with set_recursion_limit
17497f0

carljm approved these changes Dec 20, 2022
View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Dec 20, 2022

barneygale reviewed Jan 2, 2023
View reviewed changes

Comment threadLib/pathlib.py Outdated

Update Lib/pathlib.py
b7d36de
Co-authored-by: Barney Gale <barney.gale@gmail.com>

barneygale approved these changes Jan 2, 2023
View reviewed changes

Code review fixes
d428a89

Merge branch 'main' intopythongh-89727/fix-pathlib.Path.walk-recursi…
4ce5291
…on-depth

brettcannon approved these changes Jan 21, 2023
View reviewed changes

Comment threadLib/pathlib.py Outdated
Comment threadLib/test/test_pathlib.py Outdated

bedevere-bot added awaiting merge and removed awaiting core review labels Jan 21, 2023

Apply suggestions from code review
49d4237
Co-authored-by: Brett Cannon <brett@python.org>

CuriousLearner approved these changes Feb 18, 2023
View reviewed changes

barneygale reviewed Mar 11, 2023
View reviewed changes

Comment threadLib/pathlib.py Outdated

barneygale mentioned this pull request Mar 12, 2023
Path.rglob performance issues in deeply nested directories compared to glob.glob(recursive=True) #102613
Closed

ionite34 mentioned this pull request Mar 12, 2023
BOT-3EN: snekbox hits a recusion exceded exception in pathlib when parsing lots of nested directories and 500s python-discord/snekbox#172
Closed

Rename stack to paths
9bc2500

barneygale approved these changes Mar 19, 2023
View reviewed changes

barneygale merged commit 713df2c into python:mainMar 22, 2023

bedevere-bot removed the awaiting merge label Mar 22, 2023

zmievsa deleted the gh-89727/fix-pathlib.Path.walk-recursion-depth branch March 22, 2023 18:20

barneygale mentioned this pull request May 11, 2023
GH-102613: Fix recursion error from pathlib.Path.glob()#104373
Merged

barneygale added a commit that referenced this pull request May 15, 2023
GH-102613: Fix recursion error frompathlib.Path.glob()(GH-104373)
cb88ae6
Use `Path.walk()` to implement the recursive wildcard `**`. This method uses an iterative (rather than recursive) walk - see GH-100282.

Uh oh!

Uh oh!

Conversation

zmievsa commented Dec 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlifyBot commented Dec 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for python-cpython-preview canceled.

Uh oh!

zmievsa commented Dec 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brettcannon commented Dec 16, 2022

Uh oh!

Uh oh!

Uh oh!

carljm left a comment

Choose a reason for hiding this comment

Uh oh!

zmievsa commented Dec 20, 2022

Uh oh!

zmievsa commented Dec 24, 2022

Uh oh!

barneygale commented Dec 24, 2022

Uh oh!

Uh oh!

barneygale left a comment

Choose a reason for hiding this comment

Uh oh!

zmievsa commented Jan 11, 2023

Uh oh!

AlexWaygood commented Jan 11, 2023

Uh oh!

Uh oh!

Uh oh!

barneygale commented Feb 1, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CuriousLearner left a comment

Choose a reason for hiding this comment

Uh oh!

barneygale commented Feb 28, 2023

Uh oh!

Uh oh!

zmievsa commented Mar 22, 2023

Uh oh!

AlexWaygood commented Mar 22, 2023

Uh oh!

barneygale commented Mar 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

zmievsa commented Dec 15, 2022•
edited
Loading

netlifyBot commented Dec 15, 2022•
edited
Loading

zmievsa commented Dec 15, 2022•
edited
Loading

barneygale commented Feb 1, 2023•
edited
Loading