feat(provider-tck): emit a conformance report whose results are Cucumber Messages

[aepfli]

Stacked on #409 — base is feat/provider-tck, so the diff here is only the report emitter. Part of open-feature/spec#424; the schema is open-feature/spec#425.

This PR was reworked after review. It originally defined a per-scenario result format in the report schema. It no longer does: the results are carried in Cucumber Messages, the ndjson protocol, and the schema is an envelope that references them. The rest of this description describes what that means and what was deleted.

Setting PROVIDER_TCK_REPORT_DIR makes each suite write two files: an envelope at <dir>/<name>.json and the results it points at at <dir>/<name>.ndjson.

$ PROVIDER_TCK_REPORT_DIR=./reports uv run pytest tools/openfeature-provider-tck/tests
provider-tck [controllable-in-memory]: report written to reports/controllable-in-memory.json with results in controllable-in-memory.ndjson (1 failed, 24 passed, 4 skipped)
provider-tck [in-memory]: report written to reports/in-memory.json with results in in-memory.ndjson (1 failed, 23 passed, 5 skipped)

94 passed, 9 skipped, 2 xfailed

The envelope:

{
  "schemaVersion": "1",
  "provider": { "name": "In-Memory Provider", "language": "python", "configuration": "in-memory" },
  "sdk": { "name": "openfeature-sdk", "version": "0.8.4" },
  "tck": {
    "implementation": "python-sdk-contrib/tools/openfeature-provider-tck",
    "version": "0.1.0",
    "specRevision": "dfa16586d91ca020ef1b3b82a7c972d833ff8f29"
  },
  "declaration": { "declared": ["@events", "@object", "@strict-numeric-typing"] },
  "results": {
    "format": "cucumber-messages",
    "location": "in-memory.ndjson",
    "digest": "sha256:c7e12a28e86c8a715331ce134b9787732f9dc04d2f89423910e99e71e3771db6"
  },
  "backend": { "description": "the Python SDK's InMemoryProvider, rebuilt per scenario" },
  "knownDeviations": [
    {
      "issue": "https://github.com/open-feature/python-sdk/issues/619",
      "summary": "python-sdk: a boolean satisfies an Integer request. …"
    }
  ]
}

Why the results are not our format

Per-scenario outcomes, tags, Scenario Outline row identity and the executed feature source are all already specified by Cucumber Messages, which is maintained, cross-language, schema'd, and emitted natively by cucumber-jvm. Defining them again in the report schema created a second format to maintain and version, and two places for the same fact to disagree.

What was deleted

scenarios[]. Now TestCase / TestCaseStarted / TestStepStarted / TestStepFinished / TestCaseFinished, keyed by ids the protocol defines.

The four-value outcome enum. Cucumber's own seven statuses replace it. The not-declared / not-applicable split it existed for was never a property of the run: it follows from the envelope's declaration and the scenario's tags, both of which are present, so it is stated once instead of once per scenario. declaration.notApplicable keeps the distinction the enum was drawing, where it belongs — as an input to reading the results rather than a per-scenario fact.

example. This is the clearest case. It was added days ago so an outline row could be identified, and four implementations each reinvented it — one of them, this one, by reverse-engineering how its runner maps a pickle back to a table row. Messages has carried that identity all along: a pickle's astNodeIds are [scenario id, table row id], and the row id resolves in the GherkinDocument to exactly the cells the feature file wrote.

tck.assetsTree. The stream carries the executed feature Source verbatim, so a consumer can read the questions that were actually asked rather than trusting a hash to stand for them. That is strictly stronger than what the tree hash was carrying, and it is one fewer thing for the build to get right. The generated spec_revision.json is now just {"specRevision": "…"}.

What stays OpenFeature-specific, and why

The declaration is an input, not a summary. A skipped scenario in the payload says the question was not put to this provider; only the declaration says whether that is because the provider declines the capability. It cannot be derived from the results, which is exactly why it has to be stated.

No standard results format identifies the tested subject. Messages' Meta records the runtime, the OS and the CPU — what produced the answers, not what was being asked about. So provider, sdk, tck and backend stay in the envelope.

knownDeviations. A gap the provider acknowledges is a claim about the provider, not a result. The Python suite marks one scenario xfail(strict=True) against python-sdk#619; the payload still reports that scenario as FAILED, and the envelope carries the acknowledgement beside it. An adoption declares it with TckConfig.known_deviations. Recording it in the envelope is what lets a consumer tell a tracked gap from a surprise without the result itself being weakened, which is the trade the previous version got wrong by carrying the issue reference as prose inside a reason.

Dependencies added

pytest-bdd emits no Cucumber Messages. It ships cucumber_json.py, the legacy Cucumber JSON format, and nothing for the ndjson protocol — so messages.py assembles the stream. Two libraries, each doing the half it owns:

• cucumber-messages (34.2.0, new) — the official Python types, published from cucumber/messages itself. Zero dependencies, requires-python >=3.9, actively released. Used for the execution messages: Meta, TestCase, TestCaseStarted, TestStepFinished, TestStepResult, TestRunStarted/Finished, Hook.
• gherkin-official (>=29, promoted from a transitive dependency of pytest-bdd to a direct one) — the reference Gherkin parser. It produces the gherkinDocument and pickle payloads, which are Messages: emitting Messages ndjson is what that library exists for, so its output is used as it comes rather than round-tripped through another representation that could quietly drop a field it does not model.

uv.lock is regenerated, and uv sync --frozen verified against it. Only cucumber-messages is genuinely new to the lock.

The feature files are parsed a second time, by this package. pytest-bdd parses them with gherkin-official too but converts the result into dataclasses of its own that do not carry the AST node ids — and those ids are what a pickle refers to and what makes one outline row distinguishable from another. Four small files, once per session.

Truthful skips, and per-step results

Appendix F requires that a scenario skipped for an undeclared capability is reported as skipped with the reason and never as passed. pytest, unlike godog, already reports a skip honestly, and Cucumber's SKIPPED says the same thing, so this is the easy part in Python — but it is checked explicitly rather than assumed.

Step results come from pytest-bdd's step hooks (pytest_bdd_after_step, pytest_bdd_step_error, pytest_bdd_step_func_lookup_error) rather than from the scenario's verdict. Marking all eight steps of a scenario FAILED because the scenario failed would be saying something untrue about the seven that passed and the ones never reached. The failing row of the type-mismatch matrix comes out as:

test-case-7-setup     PASSED
test-case-7-0         PASSED
test-case-7-1         PASSED
test-case-7-2         PASSED
test-case-7-3         FAILED   flag 'boolean-flag' resolved to True (bool), expected 1 (int)
test-case-7-4         SKIPPED
test-case-7-5         SKIPPED
test-case-7-6         SKIPPED
test-case-7-teardown  PASSED

with the assertion carried as a TestStepResult.exception of type AssertionError, which is what that field is for.

Every test case also carries a before- and after-hook TestStep, with a Hook message declaring each. pytest runs a scenario in three phases and only the middle one executes Gherkin steps: the capability gate skips during setup, and a provider that fails to shut down fails during teardown. Neither has a pickle step to attach a result to, so without hooks a gated skip would have to borrow the first step's result and a teardown failure would be invisible behind a row of passed steps. Cucumber models exactly this.

That also closes a gap worth naming. A consumer derives a test case's outcome as the worst of its steps. A verdict no step accounts for — a strict xfail that passes, which pytest fails while every step passed — would be lost on the way out, so it is attached to the after-hook. There is a unit test for it.

Scenarios are enumerated at collection and resolved only at the end of the session, unchanged from before and still load-bearing: a scenario skipped by a marker never runs a fixture, so an emitter that learned of a scenario when its fixtures ran would leave it out of the stream entirely.

TckConfig also gains not_applicable={Capability.X: "why"} for a capability that cannot hold rather than one the provider declines. The suite treats the two identically — the scenarios are skipped either way — but the report keeps them apart, and a configuration that puts a capability in both, or gives no reason, is rejected at construction.

Verification

Run on WSL Ubuntu, Python 3.10 (the repo's pinned version), against the schema as it stands on feat/provider-tck-report-schema at 46ebc218. CI does not run on this PR: build.yml triggers only on pull requests targeting main, and this one targets feat/provider-tck. Everything below is local.

Package test suite. 94 passed, 9 skipped, 2 xfailed. ruff check, ruff format --check and mypy (whose scope includes tests) are all clean.

Envelope validation. Both envelopes validate with jsonschema as Draft 2020-12 against conformance-report.schema.json, zero errors, and results.digest matches a SHA-256 recomputed over the ndjson bytes:

Suite	Envelope	Digest	Payload
in-memory	VALID	matches	661 messages
controllable-in-memory	VALID	matches	661 messages

Payload validation. Both streams validate clean — 661 of 661 messages, zero errors — against the published Cucumber Messages JSON schema at tag v34.2.0 (jsonschema/src/*.schema.json from cucumber/messages, resolved locally through a referencing registry so the relative $refs work). Message census, identical for both suites:

meta 1, testRunStarted 1, hook 2, source 4, gherkinDocument 4, pickle 29,
testCase 29, testCaseStarted 29, testStepStarted 266, testStepFinished 266,
testCaseFinished 29, testRunFinished 1

Every scenario accounted for. 29 TestCase messages against 29 scenarios from pytest's own --collect-only, in both suites, with (uri, scenario name, Examples row) distinct across all 29. test_every_collected_scenario_appears_exactly_once checks the total against --collect-only rather than a number written down beside it, so adding a scenario to the specification cannot leave it passing while the payload loses one.

Outcome counts, computed the way a consumer must — worst step status per test case:

Suite	PASSED	SKIPPED	FAILED
in-memory	23	5	1
controllable-in-memory	24	4	1

Per step, for in-memory: 220 PASSED, 45 SKIPPED, 1 FAILED. Of the 45, 42 belong to the five gated scenarios (their before-hooks included, which is where the reason is) and three are the steps of the failing scenario that were never reached.

Gated skips are truthful. Every scenario carrying a tag absent from declaration.declared is SKIPPED, at every step, in both suites — none is reported as passed. Checked against the reason the gate actually gave, so the derivation from the two documents agrees with reality rather than merely being possible:

SKIPPED  ['@configuration-change']  A configuration change is signalled and applied
         provider does not declare capability @configuration-change. Declared: @events @object @strict-numeric-typing
SKIPPED  ['@lifecycle']             A provider reaching its backend becomes ready
SKIPPED  ['@lifecycle','@unavailable']  A provider that cannot reach its backend reports an error
SKIPPED  ['@lifecycle','@unavailable']  A provider that cannot reach its backend still returns code defaults
SKIPPED  ['@stale']                 Losing the backend makes the provider stale, regaining it makes it ready again

And the distinction survives: the scenario the generated test suite skips with a plain marker is also SKIPPED, but carries no undeclared tag, so it is not attributable to a capability.

The payload disagrees with the runner, deliberately. pytest exits zero — the one scenario the SDK fails is xfailed — and the stream reports it FAILED. That is asserted, not just observed.

The eleven rows. Requesting the wrong type returns the code default produces eleven test cases sharing one scenario name, told apart by the second entry of the pickle's astNodeIds (the id of the TableRow it was compiled from), which resolves in the GherkinDocument to the cells the feature file wrote:

astNodeIds=[25, 9]   PASSED   {key: string-flag,  requested: Boolean, default: false}
astNodeIds=[25, 10]  PASSED   {key: string-flag,  requested: Integer, default: 1}
astNodeIds=[25, 11]  PASSED   {key: string-flag,  requested: Float,   default: 0.1}
astNodeIds=[25, 12]  PASSED   {key: wrong-flag,   requested: Boolean, default: false}
astNodeIds=[25, 15]  PASSED   {key: boolean-flag, requested: String,  default: fallback}
astNodeIds=[25, 16]  FAILED   {key: boolean-flag, requested: Integer, default: 1}
astNodeIds=[25, 17]  PASSED   {key: boolean-flag, requested: Float,   default: 0.1}
astNodeIds=[25, 20]  PASSED   {key: integer-flag, requested: Boolean, default: false}
astNodeIds=[25, 21]  PASSED   {key: integer-flag, requested: String,  default: fallback}
astNodeIds=[25, 22]  PASSED   {key: float-flag,   requested: Boolean, default: false}
astNodeIds=[25, 23]  PASSED   {key: float-flag,   requested: String,  default: fallback}

25 is the Scenario node all eleven share, which is right: the scenario is one scenario. The rows the tests recover from the stream this way are compared against the three Examples tables read out of the Gherkin by hand, rather than against the parser that produced them.

The same holds for rows the capability gate skipped, since row identity comes from the pickle rather than from the run: with @object undeclared, all four rows of Requesting a structured flag as a scalar returns the code default are present, SKIPPED, and individually identified.

Tags on an Examples block. Still covered. Gherkin lets an Examples block carry its own tags, so two rows of one outline can differ in which capability gates them, and those tags are on neither the scenario, the feature nor the rule. No canonical feature file does this yet, so the test writes its own: with one row of a three-row outline gated, all three appear, the gated one is SKIPPED and carries @object, and its two siblings pass.

The executed source is present and correct. Every Source message's data compares equal to the feature file on disk, and the set of Source uris equals the set of uris the test cases name.

Cross-platform. pathlib and list-argv subprocess throughout. Feature uris are normalised to forward slashes: pytest-bdd builds its relative filename with os.path.join, so on Windows it arrives backslash-separated, and the same string has to appear in the Source, the GherkinDocument and every Pickle or nothing ties them together — a report emitted on Windows would otherwise not be comparable with one emitted on Linux. There is a test for it. The emitter itself was developed and verified on Linux; the Windows path of that normalisation is covered by the unit test rather than by a Windows run.

Packaging. uv build produces a wheel carrying spec_revision.json alongside the four feature files, the canonical flag set and control-api.yaml, and declaring the two new requirements.

Failure to write still fails the run loudly, verified by pointing PROVIDER_TCK_REPORT_DIR at a path under a regular file. A run that asked for a report and silently did not get one is how a publishing pipeline ends up serving a stale result forever. A scenario that ran and matches no pickle fails the session the same way and names the scenario, because that is the one failure mode this format exists to rule out.

Things worth raising, rather than glossing

The capability rollup is gone entirely, not moved. The previous version summarised a verdict per capability. Both defects raised against it — a failed capability with no reason, and a declared-but-untested capability reported green — disappear with it, but so does the summary. A consumer wanting "did @object pass" now computes it from the payload: the scenarios carrying that tag, and their statuses. That is more work for the consumer and one fewer thing to disagree with the results. Whether the schema should offer a derived summary is worth deciding deliberately rather than by inheritance.

There is still no overall verdict in the envelope. TestRunFinished.success in the payload is false when any scenario failed, which is more than the previous version had, but a consumer must open the payload to see it. Worth deciding whether the envelope wants a top-level verdict.

The stream is assembled by this package, not by the runner. If pytest-bdd ever emits Messages natively, messages.py should shrink to a shim. Two places where that assembly makes a judgement a native emitter would not have to: the join from a pytest node to a pickle goes through (uri, scenario name, Examples row), because pytest-bdd's parse and this one are separate; and test-case/step ids are synthesised (test-case-7-3) rather than being ids the runner already owns. Both are internally consistent and neither is observable across runs, but they are not the ids cucumber-js would emit.

Meta.protocolVersion is read from the installed cucumber-messages version, not pinned in code, so it cannot go stale after a dependency bump. It reports the library version rather than a protocol version negotiated with anything.

Not done here: CI does not upload the reports as artifacts. That belongs with whatever consumes them, and can follow.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}