Make the test runners cheaper to run repeatedly and quieter to read back

[swissspidy]

Three independent changes coming out of the discussion in wp-cli/wp-cli#6161 about what it costs an AI agent — or anyone iterating in a terminal — to work in a WP-CLI repository. Happy to split them into separate pull requests if that reads better.

Pairs with wp-cli/.github#285, which rewrites the composer test guidance in AGENTS.md and points the CI Gherkin job at composer lint-gherkin.

1. Quieter output, opt-in

Two environment variables, both unset by default, so nothing changes for existing users or for CI:

• NO_COLOR (no-color.org) stops the runners from forcing ANSI color on. run-linter-tests passed --colors and run-php-unit-tests passed --color=always unconditionally, so every captured log carried escape sequences whether or not anything was going to render them. Where a project's own config can turn color back on — phpunit.xml with colors="true" — the flag is set to an explicit never rather than omitted.

• WP_CLI_TEST_QUIET switches the reporters to their most compact form:

  • PHP_CodeSniffer → -q --report=emacs, one file:line:col line per violation, no progress ticker
  • PHPStan → --no-progress --error-format=raw, one file:line:message line per error, no redrawing progress bar and no box-drawing result table

  Behat is deliberately untouched: its progress output is already minimal, and its step definition snippets are how a typo in an existing step surfaces, so they are a diagnostic rather than noise.

Also documents, in the README, things the runners already supported but nobody had written down: narrowing a Behat run to a single scenario with features/x.feature:12, --tags=, --stop-on-failure, and composer behat-rerun.

2. Cache the WP_VERSION lookup

run-behat-tests resolved WP_VERSION over the network on every single invocation, whether you were running the full suite or re-running one scenario for the fifth time while iterating on a fix.

It also made two separate requests for what is one question. The wp-versions artifact already carries a status per release with the current one marked latest, so the extra call to api.wordpress.org was redundant. Both the latest resolution and the X.Y → latest-patch resolution now come out of that single file.

It is cached under the system temp directory, following the wp-cli-test-* naming the FeatureContext core download cache already uses. Lifetime defaults to a day and is configurable through WP_CLI_TEST_WP_VERSION_CACHE_TTL, where 0 fetches every run. Net effect: at most one request per run, and none at all on a warm cache.

Two behavior changes fall out of it, both of which look like improvements but are worth calling out explicitly:

• A run without connectivity now falls back to the last known copy. Previously it continued with an empty WP_VERSION, which silently disabled filtering of the @require-wp-* tags.
• When there is nothing to fall back to, that is now reported rather than being silent.

WP_VERSION=X.Y.0 still normalizes to X.Y and stops there, rather than resolving on to the newest patch — that spelling asks for the initial release specifically.

3. Bring the Gherkin linting into the test suite

The feature files are linted on every pull request, but the check exists only inside the reusable CI workflow and its ruleset lives in wp-cli/.github. Contributors cannot run it locally at all — not "it is inconvenient", but there is no config file in the repository to run it against. So composer test passing does not mean the build passes, and the way to find out is to push.

This moves it next to the other suites:

• .gherkin-lintrc ships with this package as the shared default ruleset, carried over unchanged from wp-cli/.github. A project that needs different rules overrides it by committing its own.
• composer lint-gherkin runs it, and it joins composer test and the setup instructions.
• CI can then call that script instead of reimplementing the invocation, which leaves one place to change the rules.

Uses gherkin-lint-plus. It is a Node package, so it runs through npx and needs Node.js 20 or later; where npx is absent it reports that it is skipping rather than failing a suite that is otherwise entirely PHP. That is a deliberate trade — a hard failure would break composer test for every PHP-only contributor across ~40 repositories — and CI, where Node is always present, still enforces it.

The version is pinned in a package.json that exists for no other purpose: it is private, has no scripts, and nothing runs npm install against it. The pin lives there rather than in the shell script because a version string in a shell script is invisible to Dependabot. Picking the updates up needs the npm entry added in wp-cli/.github#285.

One wrinkle worth recording: the linter writes its report to STDERR and colors it unconditionally, honoring neither NO_COLOR nor the absence of a terminal, and stylish is its only output format. So the runner strips the escape sequences from that stream itself when NO_COLOR is set, preserving STDOUT and the exit code.

Testing

The Gherkin linting is verified end to end, since Node was available where I was working:

• Both this package's four feature files and wp-cli/wp-cli's thirty-five pass cleanly under the ported ruleset, so adopting this does not start with a wall of pre-existing violations.
• Positive control on a deliberately broken feature file: file-name, no-unnamed-scenarios, indentation and use-and are all caught, exit code 1, while no-trailing-spaces correctly stays quiet because the shared config disables it. The fork reads the existing ruleset the same way gherkin-lint did, indentation option keys included.
• NO_COLOR=1 output is stripped of escape sequences with the exit code preserved; the default run keeps its colors; a clean tree prints nothing and exits 0; an explicit path argument overrides the features default; a package with no features directory skips and exits 0; a package.json with the pin removed fails with a message rather than silently installing the latest release.

The version resolution was exercised against a stubbed curl and the real artifact: cold cache, warm cache with curl removed from PATH entirely (zero requests), stale entry with the network down, a malformed response, latest → 7.0.4, 6.8 → 6.8.8, 6.9.0 → 6.9, 7.0.0 → 7.0, and trunk and exact versions passing through untouched. All the shell is syntax-checked and composer validate passes.

What I could not do is run the PHP suites. composer install does not complete in the environment I am working in — phpstan/phpstan is dist-only and its dist URL is on api.github.com, which is blocked here. So the flags in the first commit (--report=emacs, --error-format=raw, --color=never) are unverified by execution and rest on the documented CLI surface of each tool. That is the part of this pull request that most needs CI, or a second pair of eyes.

Refs wp-cli/wp-cli#6161

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3fd7eb23-d0fc-4285-8afa-b2c242ea34eb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!