GH-101362: Optimise PurePath(PurePath(...)) by barneygale · Pull Request #101667 · python/cpython

barneygale · 2023-02-07T22:25:15Z

The previous _parse_args() method pulled the _parts out of any supplied PurePath objects; these were subsequently joined in _from_parts() using os.path.join(). This is actually a slower form of joining than calling fspath() on the path object, because it doesn't take advantage of the fact that the contents of _parts is normalized!

This reduces the time taken to run PurePath("foo", "bar") by ~20%, and the time taken to run PurePath(p, "cheese"), where p = PurePath("/foo", "bar", "baz"), by ~40%.

Issue: Optimize pathlib path construction #101362

Automerge-Triggered-By: GH:AlexWaygood

The previous `_parse_args()` method pulled the `_parts` out of any supplied `PurePath` objects; these were subsequently joined in `_from_parts()` using `os.path.join()`. This is actually a slower form of joining than calling `fspath()` on the path object, because it doesn't take advantage of the fact that the contents of `_parts` is normalized! This reduces the time taken to run `PurePath("foo", "bar") by ~20%, and the time taken to run `PurePath(p, "cheese")`, where `p = PurePath("/foo", "bar", "baz")`, by ~40%.

barneygale · 2023-02-25T01:07:06Z

@zooba per our discussion in #101362, I've tweaked the docs in this PR. I'm trying to convey that we treat PurePath objects like any other kind of os.PathLike object, which implies we call os.fspath(). Let me know if you'd like us to be more explicit? But I don't want to consume too much docs space on this, as it's a bit off the trodden path, and it stops making sense when paths have drives IMO.

zooba · 2023-02-27T16:22:47Z

I guess my only question now is whether join handles differing types the same way as parse_args used to? Specifically, each element is converted to a string before being joined, rather than failing with TypeError.

If it works the same, then I have no concerns.

barneygale · 2023-02-28T11:56:43Z

os.path.join() requires that you don't mix str and bytes arguments (after fspath()) - it raises TypeError if you do. But it supports subclasses of str and bytes and won't convert them to true strings/bytes.

pathlib requires that every argument is a str (after fspath()). It also supports subclasses of str, and will convert them to true strings.

I think this patch maintains all previous behaviour. You still get a TypeError if any argument you supply, after fspath(), isn't a subclass of str.

barneygale · 2023-02-28T12:01:01Z

Actually, there don't seem to be good tests for this. I'll write some!

zooba · 2023-02-28T16:23:54Z

LGTM!

AlexWaygood

Looks good! A few minor suggestions

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

…lk6mt.rst Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood

Thank you!

barneygale · 2023-03-05T23:38:12Z

Thanks so much, Alex! \o/

miss-islington · 2023-03-05T23:50:20Z

Status check is done, and it's a success ✅.

The previous `_parse_args()` method pulled the `_parts` out of any supplied `PurePath` objects; these were subsequently joined in `_from_parts()` using `os.path.join()`. This is actually a slower form of joining than calling `fspath()` on the path object, because it doesn't take advantage of the fact that the contents of `_parts` is normalized! This reduces the time taken to run `PurePath("foo", "bar")` by ~20%, and the time taken to run `PurePath(p, "cheese")`, where `p = PurePath("/foo", "bar", "baz")`, by ~40%. Automerge-Triggered-By: GH:AlexWaygood

* main: (21 commits) pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives in sub interpreters module (python#102472) pythongh-95672: Fix versionadded indentation of get_pagesize in test.rst (pythongh-102455) pythongh-102416: Do not memoize incorrectly loop rules in the parser (python#102467) pythonGH-101362: Optimise PurePath(PurePath(...)) (pythonGH-101667) pythonGH-101362: Check pathlib.Path flavour compatibility at import time (pythonGH-101664) pythonGH-101362: Call join() only when >1 argument supplied to pathlib.PurePath() (python#101665) pythongh-102444: Fix minor bugs in `test_typing` highlighted by pyflakes (python#102445) pythonGH-102341: Improve the test function for pow (python#102342) Fix unused classes in a typing test (pythonGH-102437) pythongh-101979: argparse: fix a bug where parentheses in metavar argument of add_argument() were dropped (python#102318) pythongh-102356: Add thrashcan macros to filter object dealloc (python#102426) Move around example in to_bytes() to avoid confusion (python#101595) pythonGH-97546: fix flaky asyncio `test_wait_for_race_condition` test (python#102421) pythongh-96821: Add config option `--with-strict-overflow` (python#96823) pythongh-101992: update pstlib module documentation (python#102133) pythongh-63301: Set exit code when tabnanny CLI exits on error (python#7699) pythongh-101863: Fix wrong comments in EUC-KR codec (pythongh-102417) pythongh-102302 Micro-optimize `inspect.Parameter.__hash__` (python#102303) pythongh-102179: Fix `os.dup2` error reporting for negative fds (python#102180) pythongh-101892: Fix `SystemError` when a callable iterator call exhausts the iterator (python#101896) ...

* main: (37 commits) pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives in sub interpreters module (python#102472) pythongh-95672: Fix versionadded indentation of get_pagesize in test.rst (pythongh-102455) pythongh-102416: Do not memoize incorrectly loop rules in the parser (python#102467) pythonGH-101362: Optimise PurePath(PurePath(...)) (pythonGH-101667) pythonGH-101362: Check pathlib.Path flavour compatibility at import time (pythonGH-101664) pythonGH-101362: Call join() only when >1 argument supplied to pathlib.PurePath() (python#101665) pythongh-102444: Fix minor bugs in `test_typing` highlighted by pyflakes (python#102445) pythonGH-102341: Improve the test function for pow (python#102342) Fix unused classes in a typing test (pythonGH-102437) pythongh-101979: argparse: fix a bug where parentheses in metavar argument of add_argument() were dropped (python#102318) pythongh-102356: Add thrashcan macros to filter object dealloc (python#102426) Move around example in to_bytes() to avoid confusion (python#101595) pythonGH-97546: fix flaky asyncio `test_wait_for_race_condition` test (python#102421) pythongh-96821: Add config option `--with-strict-overflow` (python#96823) pythongh-101992: update pstlib module documentation (python#102133) pythongh-63301: Set exit code when tabnanny CLI exits on error (python#7699) pythongh-101863: Fix wrong comments in EUC-KR codec (pythongh-102417) pythongh-102302 Micro-optimize `inspect.Parameter.__hash__` (python#102303) pythongh-102179: Fix `os.dup2` error reporting for negative fds (python#102180) pythongh-101892: Fix `SystemError` when a callable iterator call exhausts the iterator (python#101896) ...

bedevere-bot added the awaiting review label Feb 7, 2023

bedevere-bot mentioned this pull request Feb 7, 2023
Optimize pathlib path construction #101362
Closed

barneygale added the topic-pathlib label Feb 7, 2023

barneygale mentioned this pull request Feb 21, 2023
Inconsistent behavior of pathlib.WindowsPath with drive paths #80486
Closed

Tweak PurePath constructor docs to hint that we call os.fspath()
6db6d55

Add tests for bytes paths.
1b9e2d3

Merge branch 'main' intopythongh-101362-use-native-pathlib-joining
bee1ab5

AlexWaygood reviewed Mar 5, 2023
View reviewed changes

Comment threadDoc/library/pathlib.rst Outdated
Comment threadLib/pathlib.py Outdated
Comment threadLib/test/test_pathlib.py Outdated
Comment threadMisc/NEWS.d/next/Library/2023-02-07-22-20-32.gh-issue-101362.Jlk6mt.rst Outdated

Apply suggestions from code review
37be083
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

AlexWaygood reviewed Mar 5, 2023
View reviewed changes

Comment threadMisc/NEWS.d/next/Library/2023-02-07-22-20-32.gh-issue-101362.Jlk6mt.rst Outdated

barneygaleand others added 2 commits March 5, 2023 23:05

Update Misc/NEWS.d/next/Library/2023-02-07-22-20-32.gh-issue-101362.J…
782a940
…lk6mt.rst Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Use assertRaisesRegex() to check TypeError message in test.
0ecae3d

AlexWaygood reviewed Mar 5, 2023
View reviewed changes

Comment threadLib/pathlib.py Outdated

Improve TypeError message
fbfeedb

AlexWaygood approved these changes Mar 5, 2023
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Mar 5, 2023

AlexWaygood added the 🤖 automerge label Mar 5, 2023

AlexWaygood added the performance Performance or resource usage label Mar 5, 2023

miss-islington merged commit 6716254 into python:mainMar 5, 2023

bedevere-bot removed the awaiting merge label Mar 5, 2023

barneygale mentioned this pull request Apr 5, 2023
Support for sharing state between pathlib subclasses #100479
Closed

barneygale mentioned this pull request Apr 20, 2023
PurePosixPath no longer correctly parses PureWindowsPath #103631
Closed

Navigation Menu

Uh oh!

Uh oh!

GH-101362: Optimise PurePath(PurePath(...)) - #101667

GH-101362: Optimise PurePath(PurePath(...))#101667
miss-islington merged 8 commits into
python:mainfrom
barneygale:gh-101362-use-native-pathlib-joining

barneygale commented Feb 7, 2023•
edited by miss-islington
Loading

Uh oh!

barneygale commented Feb 25, 2023•
edited
Loading

Uh oh!

zooba commented Feb 27, 2023

Uh oh!

barneygale commented Feb 28, 2023

Uh oh!

barneygale commented Feb 28, 2023

Uh oh!

zooba commented Feb 28, 2023

Uh oh!

AlexWaygood left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment

Uh oh!

barneygale commented Mar 5, 2023

Uh oh!

miss-islington commented Mar 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

barneygale commented Feb 7, 2023• edited by miss-islington Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

barneygale commented Feb 25, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zooba commented Feb 27, 2023

Uh oh!

barneygale commented Feb 28, 2023

Uh oh!

barneygale commented Feb 28, 2023

Uh oh!

zooba commented Feb 28, 2023

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

barneygale commented Mar 5, 2023

Uh oh!

miss-islington commented Mar 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

barneygale commented Feb 7, 2023•
edited by miss-islington
Loading

barneygale commented Feb 25, 2023•
edited
Loading