[6.x] Pest time-balanced sharding

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.

[jasonvarga]

Adds Pest as the CI test runner so we can shard the PHP suite across parallel jobs, using its time-balanced sharding to keep them even.

The suite is ~10,000 tests and the slowest job (Windows) had grown to nearly 13 minutes, which sets the floor for how long a PR sits waiting.

Tests themselves (after the overhead of setting up the jobs) goes from ~11m to ~3m.

A pull request workflow run goes from ~14m to ~4m.

Pest 4 is used because we still support Laravel 12. Luckily, the time-balanced sharding feature actually landed on Pest 4 - not Pest 5 like their marketing implies.

Pest is only the runner

Intentionally not rewriting any tests. Pest is just being used as the runner and for its sharding feature.

You can use vendor/bin/phpunit or vendor/bin/pest locally.

Ubuntu is split 4 ways, Windows 6

Windows runs the same tests in roughly twice the time, so splitting both evenly leaves Windows as the critical path with Ubuntu jobs sitting idle. Six shards brings Windows' test execution down to roughly the same as Ubuntu's, for two extra jobs.

Test fixes

Three pre-existing bugs surfaced while getting this green. They're follow-ups to #15143 — same class of problem, just not caught at the time — and are included here because sharding can't work without them:

• Invalid data provider key.ProvidesExternalUrls::externalUrlProvider() keyed its data sets by URL, and '' is one of the internal URLs — an empty-string key, which PHPUnit rejects. This broke --list-tests outright, which is how shard assignment is discovered.
• CP\ColorTest leaked state. It wrote theme colour preferences to resources/preferences.yaml and never cleaned up, so Preferences\EndpointsTest inherited a stray theme.colors block. Now cleaned up in tearDown(), matching DefaultPreferencesTest and PrecedenceTest.
• View tag state leaked between tests. The increment and rotate tags keep counters in static properties that are only cleared when the ClearState listener fires on a response. A test that renders Antlers without a full request leaves them populated. StateTest picked up a counter of 31 from IncrementTest once sharding put them in the same job. TestCase::setUp() now resets view state.

Notes

• Sharding requires tests to be independent, which is what [6.x] Stop tests depending on other tests having run first #15143, [6.x] Fix CP nav breaking when a form has no title #15144 and [6.x] Fix template scanning on Windows junctions #15145 were about. Those are merged; this sits on top.
• --parallel is deliberately not used. The suite writes into the testbench skeleton, and concurrent processes corrupt it.