Uh oh!
There was an error while loading. Please reload this page.
Fix Behat code coverage pipeline (#218) - #219
Merged
Merged
Conversation
The Behat coverage path produced no usable data, so Codecov's totals were PHPUnit-only and any change covered solely by Behat reported 0% patch coverage. Three defects, all in the Behat coverage path: 1. `CoverageContext::setup()` called `Filter::includeFile()` with the `src` *directory*. `includeFile()` realpaths and registers a single file key, so a directory matches no source file and `PcovDriver::stop()` intersects the waiting file list down to nothing. Nothing was ever recorded. The filter is now populated with every `src/**/*.php` path via `SebastianBergmann\FileIterator\Facade` (`src/Resources/config` excluded to match the PHPUnit source scope and codecov.yml's ignore list). 2. Clover XML was written to a `.cov` filename for `phpcov merge` to consume. `phpcov merge` never reads Clover. 3. Writing a real `.cov` would not have fixed it either. CI downloads the latest `phpcov.phar` (13.x), whose `MergeCommand` uses `CodeCoverage\Serialization\Unserializer` and requires a `<?php // phpunit/php-code-coverage serialization format N` header line. The installed php-code-coverage (12.5.7) has no `Serialization` namespace - its `Report\PHP` writes a bare serialized object, which that unserializer rejects. The merge step was unfixable without a version bump. `CoverageContext` now writes Clover XML straight to `build/logs/behat/clover.xml` (overridable via `BEHAT_COVERAGE_CLOVER`), which Codecov already consumes, and the `phpcov merge` step is removed. It is replaced by a verification step that fails the job if the report is missing or near-empty - the old step had `continue-on-error: true` and reported success while producing no file, which is what hid the problem. Measured locally over the full suite (451 scenarios): 4402/6548 covered statements in `src` (67.23%), versus 27.64% from PHPUnit alone. Union: 82.07%.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@## main #219 +/- ##
=============================================
+ Coverage 28.54% 83.07% +54.53%
Complexity 2542 2542 =============================================
Files 253 253 Lines 7399 7399 =============================================
+ Hits 2112 6147 +4035 + Misses 5287 1252 -4035
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes#218.
What was wrong
The Behat coverage pipeline recorded nothing, so Codecov's
behatflag contributed no data and project totals were PHPUnit-only. Any change covered only by Behat scenarios reported 0% patch coverage.Three defects, not two. The issue identified the first two from static analysis; the third only shows up when you look at what CI actually downloads.
1. The coverage filter included nothing.
CoverageContext::setup()calledFilter::includeFile()with thesrcdirectory. Confirmed againstvendor/phpunit/php-code-coverage/src/Filter.php:includeFile()realpaths its argument and stores it as a single key in$files;isExcluded()then does!isset($this->files[$filename]), so no source file ever matches.PcovDriver::stop()intersects pcov's waiting file list with$filter->files()and gets an empty array. Nothing was recorded.2. Clover XML was written to a
.covfilename.phpcov mergedoes not read Clover.3. Writing a real
.covwould not have fixed it either. The issue suggested(new PHP())->process(...). That does not work with the phpcov CI downloads.ci.ymlwgets the latestphpcov.phar— currently 13.1.0 — whoseMergeCommanddelegates toSebastianBergmann\CodeCoverage\Serialization\Merger. ItsUnserializer::unserialize()requires the first line of each.covto match^<\?php \/\/ phpunit\/php-code-coverage serialization format (\d+)$, and throwsFileCouldNotBeReadExceptionotherwise. The installed php-code-coverage is 12.5.7, which has noSerializationnamespace at all — itsReport\PHPwrites a bare<?php return \unserialize(...), which that unserializer rejects. So the merge step was unfixable without a dependency bump.Filterin 12.5.7 also has noincludeDirectory()method (onlyincludeFile()/includeFiles()), so the issue's suggested one-word fix would not have compiled.What changed
CoverageContextpopulates the filter with everysrc/**/*.phppath viaSebastianBergmann\FileIterator\Facade(already installed as a php-code-coverage dependency).src/Resources/configis excluded to match the PHPUnit source scope andcodecov.yml's ignore list.CoverageContextwrites Clover XML directly tobuild/logs/behat/clover.xml— the path Codecov already picks up — overridable viaBEHAT_COVERAGE_CLOVER. The intermediate.covis gone.Merge code coverage reportsstep is removed fromci.ymland replaced withVerify code coverage report is not empty, which fails the job if the report is missing, empty, or has under 1000 covered statements. The old step hadcontinue-on-error: trueand reported success while producing no file — that is what hid this for so long.-d pcov.enabled=1added to the coverage Behat invocation so the run does not depend on setup-php's ini defaults.CLAUDE.mdrecording why there is nophpcov mergestep, so it does not get reintroduced.No
src/code changed, so the Infection gate is untouched.How it was verified
Locally, PHP 8.5.9 with pcov.
Small subset first —
features/user/verify_email_address.featureunder--profile=default-coverage, 7 scenarios:ResendVerifyEmailAddressAction.phpis the file whose 0% patch coverage on #217 surfaced this. Before the fix the report did not exist at all.Full suite under the coverage profile — 451 scenarios, 3374 steps, all passing:
294 files is exactly
find src -name '*.php' -not -path 'src/Resources/config/*' | wc -l, so uncovered files are included too and the percentage is not inflated by only counting executed files.Comparing the two clover reports line by line over
src:The 27.64% figure matches Codecov's current ~28.5% project total, which corroborates that the
behatflag has been contributing nothing. Expect the project total to rise substantially once this lands; I have not tried to predict where it settles since Codecov's carry-forward and ignore rules differ from this local calculation.No regressions:
vendor/bin/behat(default profile, full): 451 scenarios, 3374 steps, all passingvendor/bin/phpunit: 579 tests, 1480 assertions, 113 PHPUnit Notices (pre-existing), 0 riskyvendor/bin/phpunit --configuration=phpunit.coverage.xml.dist(the config Infection's initial run uses): same 579 / 0 riskyvendor/bin/php-cs-fixer fix: 0 of 413 files changedBehat coverage: 4402/6548 covered statements across 294 files, exit 0CI result on this PR
All jobs green (Scrutinizer excepted — chronically failing on this repo, unrelated).
The new verification step printed, in the
Behat (Symfony 7.4) (PHP 8.5)job:— identical to the local run — and Codecov ingested
build/logs/behat/clover.xmlunder thebehatflag.Codecov reports project coverage 28.54% -> 83.07% (+54.53%), with the
behatflag appearing for the first time at 68.72%; thephpunitflag is unchanged at 28.54%. Hits went from 2112 to 6147 with no change to the line count, i.e. the same code, previously unmeasured.