Skip to content

test: make random component execution safer - #10169

Merged
paulbalandan merged 6 commits into
codeigniter4:developfrom
memleakd:refactor/random-tests-commands
May 11, 2026
Merged

test: make random component execution safer#10169
paulbalandan merged 6 commits into
codeigniter4:developfrom
memleakd:refactor/random-tests-commands

Conversation

@memleakd

@memleakdmemleakd commented May 6, 2026

Copy link
Copy Markdown
Contributor

Description

Related to #9968.

This one took a few rounds: every time the Commands suite looked settled, another bit of shared test state quietly raised its hand. The result is a more random-order-safe Commands component, plus a few small shared-state hardening fixes that the parallel random component runs exposed along the way.

This enables Commands in the random test config and updates the affected tests so they own their setup/cleanup instead of depending on earlier tests running first.

Main changes:

  • reset scaffold test CLI state and make generated command/scaffold cleanup reliable between tests
  • isolate migration command tests from generated App migrations and leftover tables
  • make database command tests seed and select their own expected state
  • close/reset shared database connections before deleting SQLite test databases
  • restore locale/config state in translation command tests
  • reset mocked cache services after cache command tests
  • make the Unsuffixable support command provide its own template data
  • scope CreateNewChangelogTest cleanup in AutoReview so it no longer runs repository-wide git restore / git clean while random component jobs are running in parallel

This PR also touches a few areas outside tests/system/Commands because the failures were not purely local to one test class:

  • system/Commands/Database/ShowTableInfo.php now keeps db:table --show display IDs sequential even when table arrays have sparse keys
  • system/HotReloader/DirectoryHasher.php now hashes files in a stable order, which avoids filesystem traversal order differences
  • system/Cache/FactoriesCache/FileVarExportHandler.php now guards cache writes against concurrent cleanup/recreation races
  • tests/system/HotReloader/DirectoryHasherTest.php now uses isolated writable fixtures instead of shared app/system directories
  • system/Log/Handlers/FileHandler.php now treats the final chmod after creating a log file as best-effort, so concurrent log cleanup cannot turn logging into a test/app error

Verified locally:

  • repeated random-order Commands runs, including failed CI seeds
  • HotReloader, Cookie random-order runs
  • composer phpstan:check
  • vendor/bin/rector process --dry-run
  • composer cs

Checklist:

  • Securely signed commits
  • Component(s) with PHPDoc blocks, only if necessary or adds value (without duplication)
  • Unit testing, with >80% coverage
  • User guide updated
  • Conforms to style guide

@memleakd
memleakdforce-pushed the refactor/random-tests-commands branch 14 times, most recently from 0f09214 to 39a32beCompareMay 8, 2026 08:36
@github-actionsgithub-actionsBot added the github_actions Pull requests that update Github_actions code label May 8, 2026
@memleakd
memleakdforce-pushed the refactor/random-tests-commands branch from 39a32be to c325e39CompareMay 8, 2026 08:38
@github-actionsgithub-actionsBot removed the github_actions Pull requests that update Github_actions code label May 8, 2026
@memleakd
memleakdforce-pushed the refactor/random-tests-commands branch 2 times, most recently from d2c7fda to 50c93a1CompareMay 8, 2026 09:56
@memleakd
memleakdforce-pushed the refactor/random-tests-commands branch from 50c93a1 to 811476fCompareMay 8, 2026 10:07
@memleakdmemleakd changed the title test: make Commands tests random-order safetest: make random component execution saferMay 8, 2026
Comment threadsystem/Cache/FactoriesCache/FileVarExportHandler.php
Comment threadsystem/Cache/FactoriesCache/FileVarExportHandler.php
Comment threadsystem/Commands/Database/ShowTableInfo.php Outdated
Comment threadtests/system/Commands/Generators/ScaffoldGeneratorTest.php
memleakd added 2 commits May 9, 2026 08:46
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

@paulbalandanpaulbalandan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking at the random order testing! Few comments:

Comment threadsystem/Database/BaseConnection.php Outdated
Comment threadtests/system/Commands/Database/MigrateStatusTest.php
Comment threadtests/system/Commands/MigrationIntegrationTest.php Outdated
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
@memleakdmemleakd mentioned this pull request May 9, 2026
5 tasks

@michalsnmichalsn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please replace the silent returns with log_message() calls. We can't just return and say nothing bad happened here - we have to at least log an error so this leaves some trace for developers.

Comment threadsystem/Cache/FactoriesCache/FileVarExportHandler.php
Comment threadsystem/Cache/FactoriesCache/FileVarExportHandler.php
Comment threadsystem/Cache/FactoriesCache/FileVarExportHandler.php
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Comment threadsystem/Log/Handlers/FileHandler.php
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>

@michalsnmichalsn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@paulbalandan
paulbalandan merged commit 83d509c into codeigniter4:developMay 11, 2026
55 checks passed
@memleakd
memleakd deleted the refactor/random-tests-commands branch May 11, 2026 17:35
@memleakd

Copy link
Copy Markdown
ContributorAuthor

Thank you @neznaika0@paulbalandan@michalsn

paulbalandan added a commit that referenced this pull request May 22, 2026
* docs: add changelog and upgrade for v4.7.3 (#10068)
* chore: migrate SCSS from deprecated `@import` usage (#10066)
* docs: clarify `Model::find()` note for null argument (#10072)
* chore: upload as artifacts the debug files of failing random execution tests (#10074)
* test: indicate components that already pass random execution tests (#10073)
* chore: fix wrong trigger name for manually runnable workflow (#10077)
* chore: upgrade to `gvenzl/oracle-free` (#10075)
* docs: fix formatting in Time library guide (#10078)
* docs: update 014.php (#10083)
* chore: resolve PHPStan nullCoalesce and isset errors on Config properties (#10081)
* chore: resolve PHPStan nullCoalesce and isset errors on Config properties
* fix tests
* fix: make Autoloader composer path injectable to fix parallel test race condition (#10082)
* chore(deps-dev): update rector/rector requirement
Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.
Updates `rector/rector` to 2.4.0
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.3.9...2.4.0)
---
updated-dependencies:
- dependency-name: rector/rector
dependency-version: 2.4.0
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps-dev): update rector/rector requirement
Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.
Updates `rector/rector` to 2.4.1
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.0...2.4.1)
---
updated-dependencies:
- dependency-name: rector/rector
dependency-version: 2.4.1
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: remove useless @var
* refactor: add full testing to `logs:clear` command (#10090)
* fix: store SPL closures in register() so unregister() can remove them (#10097)
* refactor: add full testing for `debugbar:clear` command (#10093)
* refactor: pass `--do-not-cache-result` to prevent shared cache corruption (#10098)
Co-authored-by: John Paul E Balandan <paulbalandan@gmail.com>
* refactor: add full testing for `cache:clear` command (#10094)
* chore: re-comment transiently failing component tests (#10095)
* test: group commands tests similar to `system/Commands/` (#10096)
* chore(deps): bump actions/github-script in / (#10100)
Bumps [actions/github-script](https://github.com/actions/github-script) in `/` from 8.0.0 to 9.0.0.
Updates `actions/github-script` from 8.0.0 to 9.0.0
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](actions/github-script@ed59741...3a2844b)
---
updated-dependencies:
- dependency-name: actions/github-script
dependency-version: 9.0.0
dependency-type: direct:production
update-type: version-update:semver-major
dependency-group: github_actions
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: ensure output buffer is closed after use of `command()` (#10099)
* chore: fix labeler workflow (#10104)
* chore: fix labeler workflow
* revert now to use pull_request_target
* chore: refactor phpunit config file (#10102)
* chore: fixes for php-cs-fixer and psalm (#10105)
* fix: preserve null values in Validation::getValidated() (#10101)
* test: refactor tests on `BaseCommand` and `Commands` (#10103)
* fix: Rename phpunit.xml.dist (#10111)
* fix: refactor inconsistent behavior on `CLI::write()` and `CLI::error()` (#10106)
* test: fix command tests that may hang on linux due to sudo (#10107)
* refactor: rename `-h` option of `routes` command as `--handler` (#10113)
* fix: ensure calling `env` command with options only would not throw (#10114)
* docs: Improve guide (#10109)
* docs: Update "Managing your Applications"
* docs: Update "Composer Installation"
* docs: Update "Worker Mode"
* docs: Update "Testing"
* fix: Move next line
* refactor: start only required services (#10115)
* chore(deps): bump actions/upload-artifact in / (#10116)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) in `/` from 7.0.0 to 7.0.1.
Updates `actions/upload-artifact` from 7.0.0 to 7.0.1
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@bbbca2d...043fb46)
---
updated-dependencies:
- dependency-name: actions/upload-artifact
dependency-version: 7.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: github_actions
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore(deps): bump actions/cache in / (#10117)
Bumps [actions/cache](https://github.com/actions/cache) in `/` from 5.0.4 to 5.0.5.
Updates `actions/cache` from 5.0.4 to 5.0.5
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](actions/cache@6682284...27d5ce7)
---
updated-dependencies:
- dependency-name: actions/cache
dependency-version: 5.0.5
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: github_actions
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: fix label-pr verification step (#10118)
* chore: fix label-pr verification step
* revert to pull request target
* chore(deps-dev): update rector/rector requirement
Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.
Updates `rector/rector` to 2.4.2
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.1...2.4.2)
---
updated-dependencies:
- dependency-name: rector/rector
dependency-version: 2.4.2
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore: fix transient random test failures (#10122)
* chore(deps): bump actions/setup-node in / (#10123)
Bumps [actions/setup-node](https://github.com/actions/setup-node) in `/` from 6.3.0 to 6.4.0.
Updates `actions/setup-node` from 6.3.0 to 6.4.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](actions/setup-node@53b8394...48b55a0)
---
updated-dependencies:
- dependency-name: actions/setup-node
dependency-version: 6.4.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: github_actions
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: suppress stty stderr leak in `CLI::generateDimensions()` when stdin is not a TTY (#10124)
* refactor: further rename `--handler` to `--sort-by-handler` for `routes` (#10125)
* test: optimize AutoReview tests (#10127)
* refactor: UX: `ClearLogs::execute()` error message is misleading after interactive `'n'` (#10126)
* docs: document Axios header configuration for AJAX (#10069)
Added Axios information regarding the X-Requested-With header.
* docs: refactor AJAX request and clarify framework examples (#10129)
* docs: fix indentation on `4.7.2` and `4.7.3` changelogs (#10131)
* docs: add version switcher to docs page (#10135)
* fix: reset Kint CSP state in worker mode (#10139)
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* refactor: simplify `FileLocator::listFiles()` (#10142)
* fix: make Time::createFromTimestamp locale-independent (#10151)
* chore(deps): bump actions/labeler in / (#10161)
Bumps [actions/labeler](https://github.com/actions/labeler) in `/` from 6.0.1 to 6.1.0.
Updates `actions/labeler` from 6.0.1 to 6.1.0
- [Release notes](https://github.com/actions/labeler/releases)
- [Commits](actions/labeler@634933e...f27b608)
---
updated-dependencies:
- dependency-name: actions/labeler
dependency-version: 6.1.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: github_actions
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix: SQLSRV driver's `decrement()` method (#10155)
Co-authored-by: Michal Sniatala <michal@sniatala.pl>
Co-authored-by: John Paul E. Balandan, CPA <paulbalandan@gmail.com>
* fix: suppress tput stderr leak when TERM is not present (#10167)
* fix: support third-party loggers in toolbar logs collector (#10173)
* fix: PostgreSQL Builder's `increment()` and `decrement()` methods not working for numeric columns (#10172)
* test: fix random-order failures in Config, Honeypot, and Test (#10168)
* chore: use single class per file when possible on tests/ directory
* chore: add return array iterable doc and regenerate baseline
* refactor: reduce PHPStan child return type baseline (#10165)
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* refactor: remove PHPStan callable signature baseline (#10166)
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* fix: preserve cached table list shape (#10179)
* fix: preserve cached table list shape
* docs: add changelog entry for cached table list fix
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
---------
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* fix: harden regex matching on `key:generate` command (#10183)
* chore: apply `withRootFiles()` on rector config (#10188)
* chore: apply withRootFiles() on rector config
* chore: run cs fix
* test: make random component execution safer (#10169)
* test: make random component execution safer
* test: remove unnecessary normalization
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* fix: stabilize cached table names for random tests
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* test: address review feedbacks
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* fix: log factories cache write failures
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* fix: document best-effort log chmod
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
---------
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
* chore: add Structarmed to QA (#10180)
* chore: Add Structarmed to QA
* chore: use latest structarmed 0.3.1
* chore: bump structarmed to 0.3.2
* bump structarmed to 0.3.3 to properly fix cache on CI
* chore: bump to structarmed 0.3.4 to fix very long list progressbar
* chore: bump to structarmed 0.3.5
* chore: bump structarmed to 0.4.0
* chore: remove ignore platform php 8.5 on test-structarmed workflow
* chore: use php 8.5 in php-versions
* use tools: composer under Setup PHP
* docs: fix Bitnami link (#10190)
* fix: restore deep dot-notation traversal in `Language::getLine()` (#10189)
* fix: make frankenphp-worker.php template idempotent on watcher restart (#10191)
* chore: bump structarmed to 0.4.5
* chore: skip system/ThirdParty
* fix: `Entity::normalizeValue()` must handle `UnitEnum` before `toArray()` (#10137)
* chore: remove checkout step for base branch (#10194)
* chore(deps-dev): update rector/rector requirement
Updates the requirements on [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.
Updates `rector/rector` to 2.4.3
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.2...2.4.3)
---
updated-dependencies:
- dependency-name: rector/rector
dependency-version: 2.4.3
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* chore(deps): bump shivammathur/setup-php in / (#10201)
Bumps [shivammathur/setup-php](https://github.com/shivammathur/setup-php) in `/` from 2.37.0 to 2.37.1.
Updates `shivammathur/setup-php` from 2.37.0 to 2.37.1
- [Release notes](https://github.com/shivammathur/setup-php/releases)
- [Commits](shivammathur/setup-php@accd612...7c071df)
---
updated-dependencies:
- dependency-name: shivammathur/setup-php
dependency-version: 2.37.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: github_actions
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Merge pull request #10198 from samsonasik/migrate-deptrac-to-structarmed
chore: migrate from `deptrac` to `structarmed`
* chore(deps-dev): update boundwize/structarmed requirement (#10202)
Updates the requirements on [boundwize/structarmed](https://github.com/boundwize/structarmed) to permit the latest version.
Updates `boundwize/structarmed` to 0.5.5
- [Release notes](https://github.com/boundwize/structarmed/releases)
- [Commits](boundwize/structarmed@0.5.4...0.5.5)
---
updated-dependencies:
- dependency-name: boundwize/structarmed
dependency-version: 0.5.5
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix(config): recognize disabled zlib compression values (#10193)
* fix: escape `--host` option in `serve` command (#10203)
* chore: bump phpstan to ^2.1.55 and fix callable docblock notice (#10219)
* chore(deps-dev): update boundwize/structarmed requirement (#10218)
Updates the requirements on [boundwize/structarmed](https://github.com/boundwize/structarmed) to permit the latest version.
Updates `boundwize/structarmed` to 0.6.8
- [Release notes](https://github.com/boundwize/structarmed/releases)
- [Commits](boundwize/structarmed@0.5.5...0.6.8)
---
updated-dependencies:
- dependency-name: boundwize/structarmed
dependency-version: 0.6.8
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Merge commit from fork
* fix: validate client extension in ext_in upload rule
* add changelog and upgrade notes
* chore(deps-dev): bump the composer-dependencies group with 2 updates (#10224)
Updates the requirements on [boundwize/structarmed](https://github.com/boundwize/structarmed) and [rector/rector](https://github.com/rectorphp/rector) to permit the latest version.
Updates `boundwize/structarmed` to 0.6.15
- [Release notes](https://github.com/boundwize/structarmed/releases)
- [Commits](boundwize/structarmed@0.6.8...0.6.15)
Updates `rector/rector` to 2.4.4
- [Release notes](https://github.com/rectorphp/rector/releases)
- [Commits](rectorphp/rector@2.4.3...2.4.4)
---
updated-dependencies:
- dependency-name: boundwize/structarmed
dependency-version: 0.6.15
dependency-type: direct:development
dependency-group: composer-dependencies
- dependency-name: rector/rector
dependency-version: 2.4.4
dependency-type: direct:development
dependency-group: composer-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Prep for 4.7.3 release (#10227)
---------
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: memleakd <121398829+memleakd@users.noreply.github.com>
Co-authored-by: Michal Sniatala <michal@sniatala.pl>
Co-authored-by: Toto <totoprayogo1916@gmail.com>
Co-authored-by: Robson Jonathas <68930311+robsonjonathas@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Abdul Malik Ikhsan <samsonasik@gmail.com>
Co-authored-by: neznaika0 <ozornick.ks@gmail.com>
Co-authored-by: Asad <asadkhan4230@gmail.com>
Co-authored-by: memleakd <121398829+memleakd@users.noreply.github.com>
Co-authored-by: Vansh Patel <developer.patelvansh@gmail.com>
Co-authored-by: maniaba <61078470+maniaba@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@memleakd@michalsn@neznaika0@paulbalandan