Improves test suite performance by 8.5-25x [experiment!]

[NickSdot]

While working on #22799 I was bothered by the test suite being so slow. CI needs ~45 minutes from push to finish; running tests locally also takes quite a while. I was wondering why that is and whether it must be like that. Hence, I decided to run an experiment to find out if we could make the developer experience notably better.

Note

This experiment was fully executed by Codex. I was throwing some money on OpenAI to let GPT 5.6 Sol pursue the /goal to make the test suite "massively" faster. This PR is the result of Codex trying exactly that -- for 14 hours straight. Consequently, I will not pretend that I understand the implications of every proposal -- and I also did not review everything in detail yet. Though, I did spend plenty of time to confirm the claims and reworking things.

The result are interesting enough to open this clearly flagged as LLM-executed experiment, hoping that it can serve as a discussion starter to find out whether anything can be picked up. Would love to get feedback from people with more experience than me, to pin down which of these are legit ideas and which are dead-ends right away.

Extracted PRs

All canges are now extracted to logical PRs as outlined below.

PR	title	status
#22957	Run test subprocesses without shell	merged
#22939	Run tests in parallel by default	merged
#22942	Adds CLI test fork server for improved test suite performance	open
#22944	Cache failed PDO and SNMP service probes	merged
#22945	Add bounded test directory concurrency	open
#22946	Select Windows test workers automatically	merged
#22947	Parallelize new_oom subprocesses	open
#22948	Replace fixed test delays with deterministic synchronization	open
#22949	Make socket tests robust in restricted environments	open
#22950	Reduce expensive test fixtures	open
#22995	Improves CI performance	open

Surfaced fixes extracted to separate PRs

PR	title	status
#22934	Run scandir overflow test on Windows only	merged
#22911	Fixes leaked System V shared-memory segments in tests	merged
#22912	Prevents tests from leaking System V IPC objects	merged
#22952	Fix redirected-test progress accounting in parallel runs	merged

Results

Values in seconds; measured on macOS. The full suite improvements are already awesome on their own. But targeted runs also show up to 25x faster results (-j10 vs. -j10), many at least 2x faster, and very few slightly slower.

flag	master	here	gain
(none)	2,223.224	69.833	-2,153.39 (31.84x)	auto-detected concurrency by default
-j1	2,224.902	443.214	-1,781.69 (5.02x)	optionally opting out of concurrency
-j10	594.445	69.768	-524.68 (8.52x)

Misc

• At first, I found it weird that it would propose adding test-only code to the SAPI. But then I learned that the already
  existing, internal --repeat flag does exactly that.
• I am pretty sure that there are reasons for Windows in CI only running on -j2, and parallel not being the default. I, however, could not find out what these reasons are; hence, I kept changes Codex proposed.
• not sure whether we fancy adding new phpt sections like MAX_CONCURRENCY, but it shows gains.

Reminders / Context

Can be ignored, just here for myself.

• Run test subprocesses without shell #22957 (comment)
• run-tests.php should be split up #22961
• Test batching #22962
• Run tests in parallel by default #22939 (comment)
• Fix redirected-test progress accounting in parallel runs #22952 (review)
• Experimental run-tests.php parallelisation #2822, Experimental run-tests.php parallelisation (2019 rebase) #3838
• ext/standard: Refactor and add {s|f}scanf tests #20676
• Applied fixers to improve test robustness #23018 (review comments; follow up PR)
• Make property resolution direction-aware; amortize asymmetric visibility set checks into the runtime cache #22709 (comment) (notable)

[NickSdot]

Didn't expect this to be smooth in the very first CI run; will look into these.

[arnaud-lb]

Interesting work!

Related: #20980

Please submit each commit as a separate PR to make reviewing easier (maybe not all at once to avoid spamming the repos. Start with the most impacting ones). I will review at least some of them.

[NickSdot]

Interesting work!

Related: #20980

Please submit each commit as a separate PR to make reviewing easier (maybe not all at once to avoid spamming the repos. Start with the most impacting ones). I will review at least some of them.

Thanks, Arnaud! Would you, for now, be fine with reviewing the individual commit diffs here? Like, so that we can pre-decide a few things here first, and based on that I am happily splitting up into individual PRs. The commits table in the PR desc brings you straight to the same diff you would have in a dedicated PR.

#20980 is indeed (admittedly naively) solved by this change. By decreasing batch size, much longer running batches get automatically more rare. Though, this is something that surely could be further improved in the future.

[NickSdot]

Please submit each commit as a separate PR to make reviewing easier (maybe not all at once to avoid spamming the repos. Start with the most impacting ones). I will review at least some of them.

@arnaud-lb I stand corrected here. Didn't expect you commenting directly on the commits but here in the changes; my force pushes to keep the related changes all in one commit break the relation to the comments in that case. I already cherry picked #22939 and #22934. Will will follow up with more.

[NickSdot]

@arnaud-lb

I will keep answering here for meta things not related to only one specific PR.

1. The split into 10 logical PRs, plus one more fix, is done -- they are all linked in the description here. Didn't mark them all as ready for review yet and would appreciate if you could take a high level look at how things were split.
2. I feel like I missed to answer one of your comments somewhere; if that's the case please hint me at it.
3. Please let me know if any of the PRs need entries in NEWS, UPGRADING.INTERNALS or the like.

[NickSdot]

@arnaud-lb not sure if I should cc you in flakes like #23217 or #23225 that are likely surfaced by #22939; so I am asking one time here to find out if you want to be kept in the loop or not.

[arnaud-lb]

@NickSdot yes feel free to ping me on these :)