Use uv venv and uv pip

[maurycy]

This is a conversation starter now: should we gradually start moving towards uv that handles custom builds, venvs and pinned runs more efficiently?

The code already works, confirmed with:

python dev.py run

Tests are 2x faster:

• last main run took 13m57s
• 3a402551d306ba7b4e80c1b308c426f5f2a08e30, with enable-cache in the workflow, took 7m14s
• 4b5f47a1292e94802af93873709b688cc0c43de1, without caching in the action, took 8m20s

I didn't want to do too many changes at once, but in my opinion the best direction would be iterating over each benchmark with uv. pyperformance is just a benchmark suite after all, and the whole venv, custom build etc. management makes it unnecessarily complex.

Notably, uv run has --project, --isolated (now it's isolated if the requirements cannot be met), --with-requirements and --python (supporting custom builds), eg:

8:41:50.613123804PM CEST maurycy@gunnbjorn ~/pyperformance (uv) % uv run --isolated --project=benchmarks/bm_sympy --python=/home/maurycy/cpython/python benchmarks/bm_sympy/run_benchmark.py
Installed 4 packages in 12ms
.....................
sympy_expand: Mean +- std dev: 281 ms +- 1 ms
.....................
sympy_integrate: Mean +- std dev: 12.8 ms +- 0.1 ms
.....................
sympy_sum: Mean +- std dev: 92.8 ms +- 0.8 ms
.....................
sympy_str: Mean +- std dev: 161 ms +- 1 ms

This is similar to what happens now, just merging temporary outputs.

From what I measured venv management and dependencies only take ~5% of the run time, so no dramatic speed up should be expected, but the simplification might be worth it. Important to keep in mind that many features are never ever used, if a large chunk of benchmarks stays broken for so long and some obvious issues lurked for a while.

Another elephant in the room is compile that could also be outsourced.

I'm more than open to any suggestions!

[corona10]

Since pyperformance users include not only CPython but also alternative implementations, I don’t think relying on uv would be a good approach for us to take. cc @hugovk

[maurycy]

Since pyperformance users include not only CPython but also alternative implementations, I don’t think relying on uv would be a good approach for us to take. cc @hugovk

uv supports CPython, PyPy, GraalPy and Pyodidie

--python accepts binaries, eg custom PyPy3 build:

10:36:09.653252000AM CET maurycy@gimel /Users/maurycy/src/pyperformance % uv run --isolated --project=benchmarks/bm_sympy --python=`which pypy3` benchmarks/bm_sympy/run_benchmark.py
Built psutil==7.1.2
Installed 4 packages in 68ms
.......
sympy_expand: Mean +- std dev: 73.9 ms +- 6.0 ms
.......
sympy_integrate: Mean +- std dev: 22.5 ms +- 3.8 ms
.......
sympy_sum: Mean +- std dev: 46.7 ms +- 6.6 ms
.......
sympy_str: Mean +- std dev: 51.1 ms +- 11.6 ms

If I understand correctly, the only requirement is that -I, -B and -C needs to be supported by an interpreter. I might be totaly wrong here, though.uv seems to check sys.implementation.name and match it

ref astral-sh/uv#2096

[corona10]

No there are a lot of implementations over you listed. e.g RustPython Jython gpython. Let s not make a big huddle to them.

[corona10]

Unless you have a strong motivation to maintain this project based on uv, let's keep it as is based on standard(pip).
I love uv but this should consider other factors.

[Fidget-Spinner]

No there are a lot of implementations over you listed. e.g RustPython Jython gpython. Let s not make a big huddle to them.

That doesn't matter. Not to diss the other implementations, but they most likely can't even run pip. If you can't even run pip, there's no point supporting these.

1. RustPython can't run pip Tracking pip support in RustPython RustPython/RustPython#5332
2. Jython only supports Python 2.7, which clearly can't use pyperformance.
3. GPython says gpython does not include many python modules as many of the core modules are written in C not python. So that's also not gonna work.

There's a reason to not switch to uv by default though --- that would be a breaking change of pyperformance likely requiring a major version bump.

@maurycy I think this is really great work and I would be willing to accept this PR if the default stayed pip, but you allow an option/command line flag to switch to uv. Then I think @corona10's concerns should be solved.

[Fidget-Spinner]

FWIW, starting a pip subprocess is extremely slow on other implementations like GraalPy. Even on CPython it is slow. Running a benchmarking run on GraalPy might be faster if we switch to uv, so I'm very much looking forward to this change.

[corona10]

@maurycy I think this is really great work and I would be willing to accept this PR if the default stayed pip, but you allow an option/command line flag to switch to uv. Then I think @corona10's concerns should be solved.

It will be fine with switching to uv as an optional feature as Ken Jin commented.

RustPython can't run pip RustPython/RustPython#5332

For clarity, they can run old pip but fail with the latest pip; anyway, they have a chance to run pyperformance using old pip.
RustPython/RustPython#5332 (comment)
so switching to uv by default will be a big hurdle for new challengers :)