Skip to content

gh-117587: Add C implementation of os.path.abspath - #117855

Closed
nineteendo wants to merge 87 commits into
python:mainfrom
nineteendo:speedup-posixpath.abspath
Closed

gh-117587: Add C implementation of os.path.abspath#117855
nineteendo wants to merge 87 commits into
python:mainfrom
nineteendo:speedup-posixpath.abspath

Conversation

@nineteendo

@nineteendonineteendo commented Apr 13, 2024

Copy link
Copy Markdown
Contributor

Benchmark

posixpath.py by @eryksun:

absolute, with "/a" repeated
len speedup
100 1.521x
500 1.157x
1000 1.128x
2000 1.068x
4000 1.019x
relative 'a', with cwd length
len speedup
93+2 1.861x
548+2 1.729x
1003+2 1.881x
2043+2 1.556x
4058+2 1.548x

ntpath.py

script
::speedup-posixpath.abspath.bat@echooffecho10 chars &&call main\python -m timeit -s "import os""os.path.abspath('a/' * 5)"&&call speedup-posixpath.abspath\python -m timeit -s "import os""os.path.abspath('a/' * 5)"echo100 chars &&call main\python -m timeit -s "import os""os.path.abspath('a/' * 50)"&&call speedup-posixpath.abspath\python -m timeit -s "import os""os.path.abspath('a/' * 50)"echo1000 chars &&call main\python -m timeit -s "import os""os.path.abspath('a/' * 500)"&&call speedup-posixpath.abspath\python -m timeit -s "import os""os.path.abspath('a/' * 500)"
10 chars
500000 loops, best of 5: 635 nsec per loop # before
500000 loops, best of 5: 517 nsec per loop # after
# -> 1.23x faster
100 chars
200000 loops, best of 5: 1.53 usec per loop # before
200000 loops, best of 5: 1.23 usec per loop # after
# -> 1.24x faster
1000 chars
50000 loops, best of 5: 9.87 usec per loop # before
50000 loops, best of 5: 7.73 usec per loop # after
# -> 1.28x faster

@nineteendo
nineteendo marked this pull request as ready for review April 13, 2024 20:03
@erlend-aaslanderlend-aasland changed the title gh-117587: Speedup posixpath.abspathgh-117587: Add C implementation of posixpath.abspathApr 13, 2024
Comment threadModules/posixmodule.c Outdated
Comment threadModules/posixmodule.c Outdated
Comment threadModules/posixmodule.c Outdated
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Comment threadModules/posixmodule.c Outdated
Comment threadPython/fileutils.c Outdated
Comment threadPython/fileutils.c
nineteendoand others added 2 commits April 13, 2024 23:43
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Pieter Eendebak <pieter.eendebak@gmail.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Comment threadModules/posixmodule.c Outdated
Comment threadLib/posixpath.py Outdated
@nineteendo

Copy link
Copy Markdown
ContributorAuthor

The same approach doesn't work on Window, because ntpath.join() is much slower than nt._getfullpathname():

PS C:\Users\wanne\cpython> python -m timeit -s "import nt""nt._getfullpathname('.')"; python -m timeit -s "import os""os.path.join(r'C:\Users\wanne\cpython', '')"1000000 loops, best of 5: 286 nsec per loop # _getfullpathname200000 loops, best of 5: 1.41 usec per loop # join

@eryksun

This comment was marked as resolved.

@nineteendo

Copy link
Copy Markdown
ContributorAuthor

Preferably, abspath() should not be naively implemented by simply joining a relative path with the working directory. It doesn't work for drive-relative paths. That's a long-standing bug in the generic implementation ntpath._abspath_fallback().

I didn't even get that far in the testing process. I already knew it wasn't going to work.
I did notice this while trying to eliminate normpath() from the current implementation.

@bedevere-app

Copy link
Copy Markdown

Thanks for making the requested changes!

@erlend-aasland: please review the changes made to this pull request.

Comment threadLib/test/test_ntpath.py Outdated
Comment threadModules/posixmodule.c Outdated
@bedevere-app

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@nineteendo

Copy link
Copy Markdown
ContributorAuthor

erlend-aasland, I have made the requested changes; please review again.

@bedevere-app

Copy link
Copy Markdown

Thanks for making the requested changes!

@erlend-aasland: please review the changes made to this pull request.

Comment threadModules/posixmodule.c
Comment threadPython/fileutils.c Outdated

@picnixzpicnixz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nitpicks

Comment threadPython/fileutils.c Outdated
Comment threadPython/fileutils.c Outdated
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Comment threadModules/posixmodule.c Outdated
Comment threadModules/posixmodule.c
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

@nineteendo@eryksun@erlend-aasland@hugovk@eendebakpt@barneygale@picnixz