perf(ci): run the python test suite in parallel

[Pfannkuchensack]

Problem

python tests took 27-32 minutes, essentially all of it the pytest step — install is cached at 17-31s. Measured on run 34303000072: run pytest was 1246s on linux, 1388s on macOS, 1586s on windows. 7267 tests with no single hotspot; the 30 slowest sum to ~230s of 1083s. The job timeout had already been raised to 30 minutes (invoke-ai#9391) rather than the runtime addressed.

Result

Six jobs green in ~10 minutes; the pytest step is 392-572s against 1246-1586s. Verified across two consecutive full runs (34373639774, both attempts).

Parallel execution

• -n logical, not -n auto: xdist's auto asks psutil for physical cores, half the vCPUs a standard runner exposes. psutil is a hard runtime dependency, so that branch always wins.
• OMP_NUM_THREADS=1 / MKL_NUM_THREADS=1: torch otherwise sizes its thread pools to the whole machine, so four workers on a four-vCPU runner ask for sixteen compute threads. Without this the Windows CPU build took a worker down inside the fp8 tests, twice in two runs; with it, twelve consecutive jobs are green.
• --dist loadfile lives in addopts, so it also applies when -n is passed by hand. It is inert without -n.
• --max-worker-restart=0: a dead worker fails the run immediately and names the test it was running, instead of sixteen silent restarts.
• Each worker now reports its peak RSS in the terminal summary. That is how the failure below would be caught next time — as a number that moves, rather than a runner that vanishes.
• uv.lock and this workflow join the change-detection globs; the lockfile now pins a package the test command depends on.
• The dead --cov-report flags leave addopts: without --cov they never enabled coverage, so fail_under = 85 was inert too. The coverage guide now names the flags it promises output from.

What parallel execution exposed

The suite could not fit in a runner

The first parallel run did not fail a test — the Linux runner died at 90% with The runner has received a shutdown signal, twice at the same point, later as exit 143. Per-file peak RSS found the cause: seven files built real checkpoint-sized tensors to assert key names, dtypes and rank. test_flux2_fp8mixed_keys.py materialized 2.3 billion elements as float32 before casting to fp8 and peaked at ~15GB — more than the runner's 14.7GB available. Serial CI had been surviving this by a hair.

file	peak before	after
test_flux2_fp8mixed_keys.py	14.90 GB	0.86 GB
test_wan_checkpoint_config.py	5.05 GB	1.21 GB
test_minimax_h3_lora_probe_independence.py	2.41 GB	0.86 GB
test_flux2_scaled_fp8_keys.py	1.96 GB	0.90 GB
test_z_image_scaled_fp8_keys.py	1.67 GB	0.86 GB
test_flux1_scaled_fp8_keys.py	1.64 GB	1.00 GB
test_anima_scaled_fp8_keys.py	1.11 GB	0.86 GB

0.86 GB is the bare import baseline. Extents are capped through a shared token_extents helper where only names, dtypes, values and rank are asserted. Where the code under test decides by shape — the fused qkv in test_flux2_scaled_fp8_keys.py is split into thirds, so 4 rows would not divide — one element is expanded to the exact captured layout instead, and materialized only where safetensors needs contiguous data. test_pid_chunked_equivalence.py keeps its real sizes: the chunk boundary is the subject of the test. krea2/test_attention.py is untouched because its footprint is the diffusers import, not its tensors.

Order and timing dependencies

• test_client_state_multiuser.py inherited its JWT secret from whichever file ran earlier: alone it was 6 passed / 9 errors, including the cross-user client-state isolation checks. Its admin_token fixture now requests setup_jwt_secret like every sibling. A per-file sweep of all 437 test files in fresh processes found no other instance.
• Two tests sampled a background worker's instantaneous state. The cache-worker test now identifies its own thread by set difference — a worker leaking from an earlier test made the count unreliable in both directions. The recovery test drops is_running is True, which asserts only that the work had not finished yet; the assertions establishing that recovery ran are unchanged.

Benchmarks, and what stayed protected

CI does not run -m slow; that lane is for what needs a development machine, and AGENTS.md and tests/AGENTS.md now say so plainly. A guard that only needed a steadier measurement does not belong there, so two were made deterministic rather than marked:

• Loop-scheduler linearity measures CPU time rather than wall clock and stays in the default run. CPU time is not inflated when a worker loses its core, and at 163ms/935ms both samples clear the ~15ms clock granularity that made this measurement unusable when the loops were cheaper. It holds across two full runs at 24 workers.
• bcrypt's work factor is read from the hash instead of timed. Every other assertion in that file — $2 prefix, length 60, round trip — holds at cost factor 4, so a passlib drift or an explicit low bcrypt__rounds was invisible.

Genuinely machine-dependent measurements moved behind the marker: the ten benchmarks in test_performance.py, test_timing_attack_resistance_same_length (200 bcrypt verifications, 34s), the absolute scheduler budget, and test_consumer_time_does_not_count_as_decoder_inactivity. The three concurrency-correctness tests from test_performance.py keep running as TestConcurrentAuthOperations, without timing assertions.

slow is the lane for tests that need a development machine — real accelerator hardware, or a quiet one for timing — and is run there with -m slow. test_consumer_time_does_not_count_as_decoder_inactivity belongs in it for the second reason: its first frame is decoded under the same deadline the consumer then sleeps past, so it fails whenever FFmpeg startup is starved. Widening the window from 2s to 5s did not stop it failing at 16 and 24 workers. An injectable clock in the decoder would make the accounting observable without a real timer; that is out of scope here.

Checks

• CI 34373639774: six jobs green, twice.
• Locally: 7239 passed, 169 skipped, 9 xfailed at four workers in 278s (1083s serial); clean across two consecutive runs at 24 workers.
• -m slow on each touched file passes. ruff check and ruff format --check on all changed files.
• Nine failures appear in every local run, on this branch and on main: a local HF mirror host rejected by the SSRF guard. Environment-specific.

Follow-ups

• An injectable clock in the video decoder, to get the consumer-time invariant back into CI.
• test_image_index_service.py is ~125s, almost all of it idle in the production retry backoff; injecting that schedule would cut the serial run too.
• keys_to_mock_state_dict in state_dicts/utils.py still allocates full extents with torch.empty; it commits no pages today, but the first caller that writes to those tensors turns 4-5.6GB virtual into resident.