Skip to content

Fix Behat code coverage pipeline (#218) - #219

Merged
silverbackdan merged 1 commit into
mainfrom
fix/behat-coverage-pipeline
Aug 14, 2026
Merged

Fix Behat code coverage pipeline (#218)#219
silverbackdan merged 1 commit into
mainfrom
fix/behat-coverage-pipeline

Conversation

@silverbackdan

@silverbackdansilverbackdan commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Fixes#218.

What was wrong

The Behat coverage pipeline recorded nothing, so Codecov's behat flag contributed no data and project totals were PHPUnit-only. Any change covered only by Behat scenarios reported 0% patch coverage.

Three defects, not two. The issue identified the first two from static analysis; the third only shows up when you look at what CI actually downloads.

1. The coverage filter included nothing.CoverageContext::setup() called Filter::includeFile() with the srcdirectory. Confirmed against vendor/phpunit/php-code-coverage/src/Filter.php: includeFile() realpaths its argument and stores it as a single key in $files; isExcluded() then does !isset($this->files[$filename]), so no source file ever matches. PcovDriver::stop() intersects pcov's waiting file list with $filter->files() and gets an empty array. Nothing was recorded.

2. Clover XML was written to a .cov filename.phpcov merge does not read Clover.

3. Writing a real .cov would not have fixed it either. The issue suggested (new PHP())->process(...). That does not work with the phpcov CI downloads. ci.yml wgets the latestphpcov.phar — currently 13.1.0 — whose MergeCommand delegates to SebastianBergmann\CodeCoverage\Serialization\Merger. Its Unserializer::unserialize() requires the first line of each .cov to match ^<\?php \/\/ phpunit\/php-code-coverage serialization format (\d+)$, and throws FileCouldNotBeReadException otherwise. The installed php-code-coverage is 12.5.7, which has no Serialization namespace at all — its Report\PHP writes a bare <?php return \unserialize(...), which that unserializer rejects. So the merge step was unfixable without a dependency bump.

Filter in 12.5.7 also has no includeDirectory() method (only includeFile() / includeFiles()), so the issue's suggested one-word fix would not have compiled.

What changed

  • CoverageContext populates the filter with every src/**/*.php path via SebastianBergmann\FileIterator\Facade (already installed as a php-code-coverage dependency). src/Resources/config is excluded to match the PHPUnit source scope and codecov.yml's ignore list.
  • CoverageContext writes Clover XML directly to build/logs/behat/clover.xml — the path Codecov already picks up — overridable via BEHAT_COVERAGE_CLOVER. The intermediate .cov is gone.
  • The Merge code coverage reports step is removed from ci.yml and replaced with Verify code coverage report is not empty, which fails the job if the report is missing, empty, or has under 1000 covered statements. The old step had continue-on-error: true and reported success while producing no file — that is what hid this for so long.
  • -d pcov.enabled=1 added to the coverage Behat invocation so the run does not depend on setup-php's ini defaults.
  • Brief note in CLAUDE.md recording why there is no phpcov merge step, so it does not get reintroduced.

No src/ code changed, so the Infection gate is untouched.

How it was verified

Locally, PHP 8.5.9 with pcov.

Small subset first — features/user/verify_email_address.feature under --profile=default-coverage, 7 scenarios:

files=294 statements=6548 coveredstatements=1614 (24.65%)
ResendVerifyEmailAddressAction.php -> statements=20 covered=13
VerifyEmailAddressAction.php -> statements=11 covered=11
UserRepository.php -> statements=48 covered=10
files with >0 covered statements: 134

ResendVerifyEmailAddressAction.php is the file whose 0% patch coverage on #217 surfaced this. Before the fix the report did not exist at all.

Full suite under the coverage profile — 451 scenarios, 3374 steps, all passing:

files=294 statements=6548 covered=4402 (67.23%)

294 files is exactly find src -name '*.php' -not -path 'src/Resources/config/*' | wc -l, so uncovered files are included too and the percentage is not inflated by only counting executed files.

Comparing the two clover reports line by line over src:

covered / statements
PHPUnit alone1810 / 654927.64%
Behat alone4402 / 654867.23%
Union5375 / 654982.07%

The 27.64% figure matches Codecov's current ~28.5% project total, which corroborates that the behat flag has been contributing nothing. Expect the project total to rise substantially once this lands; I have not tried to predict where it settles since Codecov's carry-forward and ignore rules differ from this local calculation.

No regressions:

  • vendor/bin/behat (default profile, full): 451 scenarios, 3374 steps, all passing
  • vendor/bin/phpunit: 579 tests, 1480 assertions, 113 PHPUnit Notices (pre-existing), 0 risky
  • vendor/bin/phpunit --configuration=phpunit.coverage.xml.dist (the config Infection's initial run uses): same 579 / 0 risky
  • vendor/bin/php-cs-fixer fix: 0 of 413 files changed
  • The CI verification step's shell + PHP was executed locally against the real report and prints Behat coverage: 4402/6548 covered statements across 294 files, exit 0

CI result on this PR

All jobs green (Scrutinizer excepted — chronically failing on this repo, unrelated).

The new verification step printed, in the Behat (Symfony 7.4) (PHP 8.5) job:

Behat coverage: 4402/6548 covered statements across 294 files

— identical to the local run — and Codecov ingested build/logs/behat/clover.xml under the behat flag.

Codecov reports project coverage 28.54% -> 83.07% (+54.53%), with the behat flag appearing for the first time at 68.72%; the phpunit flag is unchanged at 28.54%. Hits went from 2112 to 6147 with no change to the line count, i.e. the same code, previously unmeasured.

The Behat coverage path produced no usable data, so Codecov's totals were
PHPUnit-only and any change covered solely by Behat reported 0% patch coverage.
Three defects, all in the Behat coverage path:
1. `CoverageContext::setup()` called `Filter::includeFile()` with the `src`
*directory*. `includeFile()` realpaths and registers a single file key, so a
directory matches no source file and `PcovDriver::stop()` intersects the
waiting file list down to nothing. Nothing was ever recorded. The filter is
now populated with every `src/**/*.php` path via
`SebastianBergmann\FileIterator\Facade` (`src/Resources/config` excluded to
match the PHPUnit source scope and codecov.yml's ignore list).
2. Clover XML was written to a `.cov` filename for `phpcov merge` to consume.
`phpcov merge` never reads Clover.
3. Writing a real `.cov` would not have fixed it either. CI downloads the
latest `phpcov.phar` (13.x), whose `MergeCommand` uses
`CodeCoverage\Serialization\Unserializer` and requires a
`<?php // phpunit/php-code-coverage serialization format N` header line. The
installed php-code-coverage (12.5.7) has no `Serialization` namespace - its
`Report\PHP` writes a bare serialized object, which that unserializer
rejects. The merge step was unfixable without a version bump.
`CoverageContext` now writes Clover XML straight to `build/logs/behat/clover.xml`
(overridable via `BEHAT_COVERAGE_CLOVER`), which Codecov already consumes, and
the `phpcov merge` step is removed. It is replaced by a verification step that
fails the job if the report is missing or near-empty - the old step had
`continue-on-error: true` and reported success while producing no file, which is
what hid the problem.
Measured locally over the full suite (451 scenarios): 4402/6548 covered
statements in `src` (67.23%), versus 27.64% from PHPUnit alone. Union: 82.07%.
@codecov

codecovBot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.07%. Comparing base (471fb43) to head (6a568af).

Additional details and impacted files
@@ Coverage Diff @@## main #219 +/- ##
=============================================
+ Coverage 28.54% 83.07% +54.53% 
Complexity 2542 2542 =============================================
Files 253 253 Lines 7399 7399 =============================================
+ Hits 2112 6147 +4035 + Misses 5287 1252 -4035 
FlagCoverage Δ
behat68.72% <ø> (?)
phpunit28.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Behat code coverage produces no data: filter misuse + Clover written to a .cov, so codecov totals are PHPUnit-only

1 participant

@silverbackdan