Bug Report
Introduced in: commit 4f1eb3f (PR #5438) — fix: run-workers --by suite parallelization broken by test file sorting
What was fixed in #5438
testFiles.sort() was moved from loadTests() to run() to fix worker suite distribution. Good fix — but it introduced a regression for --shuffle users.
The regression
In lib/codecept.js, the execution order is now:
// loadTests() — line ~217if(this.opts.shuffle){this.testFiles=shuffle(this.testFiles)// ✅ shuffled here}// run() — line 293 (added by #5438)this.testFiles.sort()// ❌ always sorts — overwrites the shuffle!// then:mocha.files=this.testFiles// receives sorted, not shuffled, filessort() in run() is unconditional — it runs regardless of whether --shuffle was specified. This means --shuffle is silently ignored on every run after this commit.
Steps to reproduce
# Run twice with --shuffle — expect different order each time
npx codeceptjs run --shuffle
npx codeceptjs run --shuffle
# Both runs will produce identical alphabetical order — shuffle has no effect
Expected behavior
--shuffle randomizes test execution order. Each run should produce a different sequence.
Actual behavior
Both runs produce identical alphabetical order. The shuffle applied in loadTests() is overwritten by sort() in run().
Suggested fix
Guard the sort with a shuffle check:
// run() — only sort if shuffle is not activeif(!this.opts.shuffle){this.testFiles.sort()}This preserves alphabetical order for normal runs while respecting shuffle when requested.
Happy to submit a PR if the fix looks right to maintainers.
Bug Report
Introduced in: commit
4f1eb3f(PR #5438) — fix: run-workers --by suite parallelization broken by test file sortingWhat was fixed in #5438
testFiles.sort()was moved fromloadTests()torun()to fix worker suite distribution. Good fix — but it introduced a regression for--shuffleusers.The regression
In
lib/codecept.js, the execution order is now:sort()inrun()is unconditional — it runs regardless of whether--shufflewas specified. This means--shuffleis silently ignored on every run after this commit.Steps to reproduce
Expected behavior
--shufflerandomizes test execution order. Each run should produce a different sequence.Actual behavior
Both runs produce identical alphabetical order. The shuffle applied in
loadTests()is overwritten bysort()inrun().Suggested fix
Guard the sort with a shuffle check:
This preserves alphabetical order for normal runs while respecting shuffle when requested.
Happy to submit a PR if the fix looks right to maintainers.