feat(provider-tck): add the Python conformance suite for OpenFeature providers - #409
Draft
aepfli wants to merge 11 commits into
Draft
feat(provider-tck): add the Python conformance suite for OpenFeature providers#409aepfli wants to merge 11 commits into
aepfli wants to merge 11 commits into
Conversation
…providers
A conformance suite any Python provider can adopt to verify it implements the
provider contract of the specification, and the Python implementation of the
cross-language suite defined in Appendix F. It runs the same Gherkin, the same
canonical flag set and the same control API as the Go and Java implementations.
It uses pytest-bdd, the runner the flagd provider and the flagd testkit already
use, so an adopting package gains no new test framework.
Adoption is one fixture and one call. The step definitions ship as a pytest
plugin registered through a pytest11 entry point, so there is no conftest.py to
write and nothing to import for the vocabulary - pytest-bdd resolves steps
through the fixture system, and fixtures from an installed plugin are visible
everywhere. The feature files and flag set are packaged with the distribution,
so adopting needs no git submodule.
Capability gating uses pytest.skip from an autouse fixture, so a scenario whose
capability was not declared is reported as skipped with the reason attached
rather than silently passing. The gate keys off the node's markers rather than
its requested fixtures: pytest-bdd resolves a step's fixtures lazily, so
tck_config is not in request.fixturenames at setup time, and guarding on that
silently disabled the gate.
Two self-test suites, plus unit tests for what the Gherkin cannot assert about
itself: the SDK's InMemoryProvider, and the TCK's own updatable one. The second
exists because the first cannot exercise the configuration-change path at all.
Findings, both confirmed by running the suite:
* A boolean satisfies an Integer request. The client type-checks with
isinstance(value, int) and bool subclasses int in Python, so boolean-flag
requested as an Integer returns True with reason STATIC and no error code.
This is Python-specific - the identical scenario passes in every other
language - which is a fair argument for having more than one
implementation. Tracked as open-feature/python-sdk#619, and marked
xfail(strict=True) so it stays visible and un-hides itself once fixed.
* InMemoryProvider cannot update its flag set, which Appendix A requires of
an SDK in-memory provider. Only half the machinery is missing, since
AbstractProvider already supplies emit_provider_configuration_changed, so
ControllableInMemoryProvider is a small subclass rather than a
reimplementation and should port back as a method. Tracked as
open-feature/python-sdk#620.
Verified locally: 56 passed, 7 skipped, 2 xfailed; ruff and mypy --strict clean.
Part of open-feature/spec#417
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
This was referenced Aug 24, 2026
Two things CI caught that local verification did not. `ruff format` is a separate pre-commit hook from `ruff check`, and only the latter was run locally. Nine files needed reformatting; the changes are cosmetic line-wrapping only. More importantly, the package was not being tested in CI at all. The build matrix is gated on dorny/paths-filter and its filter list had no entry for tools/openfeature-provider-tck, so no change under that path expanded the matrix and the suite never ran. The locally reported 56 passed / 7 skipped / 2 xfailed was local-only. Adding the filter block, mirroring the one for tools/openfeature-flagd-core, turns it on. Verified after formatting: 56 passed, 7 skipped, 2 xfailed; ruff check and mypy --strict still clean. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
`uv sync --frozen` in the build workflow validates the lockfile against the manifests, and the previous commit added openfeature-provider-tck to the workspace root's dependencies and [tool.uv.sources] without regenerating the lock. That breaks the build job for *every* package, not just this one. It was latent until now only because the paths-filter had no entry for this package, so no build job ran at all. Enabling the filter in the previous commit would have surfaced it as a red build. The regeneration also picks up openfeature-provider-flagd 0.5.1 -> 0.5.2, which the lock had missed when that release landed. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
TckConfig.ready_timeout was documented but never read by anything, so a provider that hung while connecting would hang the whole pytest session with no useful message, and the documented knob did nothing. api.set_provider initialises synchronously and has no timeout of its own, so the bound comes from running it on a worker thread and giving up on the result. The worker is deliberately not cancelled -- Python cannot interrupt a thread blocked in a socket call -- and is left to finish or die with the process, which is acceptable because a timeout already means the scenario is failing. A config field that claims to do something it does not is exactly the kind of quiet untruth this suite exists to catch, so it is fixed rather than removed. Verified: 56 passed, 7 skipped, 2 xfailed; ruff and mypy --strict clean. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…he tests
TckConfig.capabilities was annotated frozenset[Capability], but the README
tells adopters to write `capabilities={Capability.EVENTS, ...}` -- a set
literal. Anyone copying the documented example and running mypy got an
incompatible-argument error from the suite's own documentation. It is now
annotated Collection[Capability], which is what __post_init__ already accepted:
a set, a list or a generator all normalise to a frozenset on construction.
The reason this was invisible is the second half of the fix. mypy was
configured `files = "src"`, so the tests were never checked -- and the tests
are the reference adoption, the thing an adopting provider copies. They are now
in scope, which is what would have caught the annotation in the first place.
Verified: mypy clean over src and tests (17 files), ruff format and check
clean, 56 passed / 7 skipped / 2 xfailed.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
aepfli
force-pushed
the
feat/provider-tck
branch
from
August 24, 2026 12:13
a8e003a to
15a57bb
Compare
…dule The feature files, the canonical flag set and the control-API document are owned by open-feature/spec, not by this repository. Committing copies of them here forks the definition of conformance -- the one thing this suite exists to prevent -- and leaves no machine-checkable record of which spec revision the copies came from. Replace them with a git submodule at tools/openfeature-provider-tck/spec, pinned at dfa16586 (spec#423), plus a build-time copy. The copies are gitignored and carry a DO-NOT-EDIT marker, so the pin is now the only record of the revision and the two cannot drift apart unnoticed. An adopter installing this package still needs no submodule: the copies are force-included into the wheel and the sdist, and the sdist excludes the submodule itself so it carries the four assets rather than the whole spec repository. Only a contributor to this package needs the submodule, and `poe test` syncs it first. This mirrors what openfeature-flagd-api-testkit already does for the flagd test harness. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
lifecycle.feature was gated by @events, which was wrong in both directions. An SDK dispatches PROVIDER_READY around initialize for any provider (openfeature/provider/_registry.py), so a provider that declares @events passes the readiness scenario without demonstrating anything -- a NoOpProvider passes it identically. The gate made the scenario vacuous for exactly the providers it admitted. Conversely a stateless provider such as OFREP has a real initialisation to verify but no event stream of its own to declare @events for, so the gate shut it out of a scenario it should be held to. The spec revision pinned by the submodule retags the feature to @lifecycle and adds the capability to Appendix F. Add the matching enum member; plugin.py registers the marker by iterating the enum, so nothing else changes. Neither in-memory self-test declares it. They have no backend to reach, so their readiness scenario was passing vacuously too, and a skip with a reason is the honest outcome. 54 passed, 9 skipped, 2 xfailed. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This was referenced Aug 30, 2026
…rcion The capability vocabulary and the spec submodule pin belong to this PR, so the rename does too. It was written on the report branch, which is a sibling of the flagd and OFREP adoptions rather than an ancestor -- so the adoptions could not see it, and renaming their references there would have broken them against this base. Moving it down is what lets every branch above share one vocabulary. `Capability.STRICT_NUMERIC_TYPING` becomes `Capability.NUMERIC_COERCION`, marker `numeric-coercion`, and the submodule moves to dc4d7ae8 so the executed feature files carry the renamed tag. That bump also brings two unrelated spec changes: the lifecycle readiness scenario is renamed, and control-api.yaml gains the requirement that POST /start not return until the seeded state is served. The framing is corrected at the same time, because it was wrong rather than merely stale. Both the README and the capability's own docstring asserted that "the specification requires TYPE_MISMATCH when the requested type cannot be satisfied" and concluded that not declaring the capability was "an admission of a known bug". OpenFeature defines one numeric type deliberately -- `number` is "of unspecified type or size", and differentiating integers from floats is an optional language idiom -- so no requirement governs this, and the second claim followed from the first. The rule tested here is borrowed from flagd's numeric coercion ADR, which is scoped to flagd's own implementations; a provider behaving differently is not violating the specification. The gap in the provider contract is open-feature/spec#430, and flagd's own instance is open-feature/flagd#1996. That also makes the capability genuinely optional rather than a concession to a defect, which is the opposite of what the old text said. The report branch's own files stay with it: test_report.py does not exist here, and neither do the `not_applicable` and `known_deviations` configuration fields the rename also touched. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
A TckConfig is two things. It is the configuration a run needs, and it is the set
of claims an adopter makes about their provider -- which is what turns a skipped
scenario from a hole in the run into a recorded answer. The second role was
incomplete, and the gaps were all the same shape: something an adopter has to be
able to say that the vocabulary gave them no way to say.
`not_applicable={Capability.X: "why"}` is a capability that *cannot* hold rather
than one the adopter chose not to declare. The suite treats the two identically,
because the scenarios are skipped either way, but collapsing them misrepresents
whole languages: @numeric-coercion is unsatisfiable in JavaScript, which has no
integer type, and recording that as a choice shows every JavaScript provider as
declining something none of them can have.
`known_deviations` acknowledges a gap against something the specification does
not treat as optional, with somewhere it is tracked. An acknowledgement and not
an excuse: the scenario still fails and the suite still fails with it. What it
adds is that the gap was known rather than a surprise.
And a capability is now either declarable or reserved. @targeting and @caching
gate no scenario, so declaring one cannot be verified, cannot produce a skip, and
says only that something was claimed and nothing examined -- so declaring one is
refused at construction, where the adopter's own code is still on the stack to
say which line to fix. The default is DECLARABLE_CAPABILITIES rather than the
whole enum, because "declare everything, then narrow it" is the advice and
therefore the one place a reserved tag gets declared by accident: that is how one
implementation's published report came to assert both of them.
`capability_for_tag` and `control_api` are here for a consumer that is not. A
reporter outside this package has to tell a capability-gating tag from a merely
organisational one, and has to be able to ask a control how it drove the backend.
Nothing in this commit calls either; that is the point. Two branches sit on this
one, and neither should be able to change what the other compiles against.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
A provider is rarely only a provider. flagd has `fractional` targeting, another
vendor has a proprietary rollout rule, and pinning those used to mean a second
harness beside the conformance suite: a second backend lifecycle, a second set
of fixtures, a second thing to keep working.
An adopter's scenarios now run inside the canonical suite instead -- same
session, same provider registration, same backend control. Almost nothing was
needed to make that happen, because pytest already scans: it collects
`conftest.py` on its own and pytest-bdd resolves step definitions through the
fixture system, so a step an adopter writes beside their test module is already
in scope for the scenarios generated into it. The only thing pytest cannot find
by itself is the feature files, because the canonical ones live inside the
installed distribution. `feature_paths()` returns both -- the packaged assets,
and a `tck-extensions` directory beside the calling module if there is one --
so an adoption gains one call and no configuration:
scenarios(*feature_paths())
An extension must never be able to stand in for a canonical scenario. Java's
suite found that a same-named feature file in a second classpath root replaced
the canonical one outright and the run went green having asked the adopter's
questions; Python has a narrower route to the same place, because pytest-bdd
names a feature file by its parent directory joined to its own name and
`tck-extensions/features/errors.feature` therefore arrives under the uri the
canonical `errors.feature` already occupies.
So the uri a feature file is identified by is derived from where the file is:
`features/` for the packaged assets and nothing else, `extensions/` for anything
below a `tck-extensions` directory -- the same prefix the Go and JavaScript
suites mount extensions under, so a consumer holding reports from several
languages applies one rule. The two cases the derivation cannot rule out are
reported rather than raised, because the scenarios are the adopter's to run and
it is publishing them as the specification's that has to be refused: a file of
the adopter's own that would reach the reserved `features/` prefix, and two
feature files that would share one uri, which nothing recording a run can hold
because it keeps one copy of a feature file per uri.
Whatever refuses to publish is not here. The derivation and both problems are
public, and the self-test reads a generated adoption back through pytest's own
JUnit XML rather than through a conformance report, which this package does not
write.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…e steps Follow the conformance assets to open-feature/spec@15fe861, which adds metadata.feature, three shutdown scenarios to lifecycle.feature, the falsy-value and integer-precision scenarios to evaluation.feature, the lossless half of @numeric-coercion to errors.feature, and six flags to the canonical set. Steps: - "the error message should be empty" reads the last evaluation's error_message and accepts None or "". - "the provider is shut down" and "the provider is initialized again" call the registered provider's own shutdown() and initialize() directly, not through the SDK, so a scenario can shut down twice and an evaluation afterwards reaches the instance that was brought back. Each call is recorded as a LifecycleRecord with its duration and anything it raised; "no exception should have been thrown" now reads those records alongside the evaluation's, so there is one mechanism rather than two. A call that outlasts ready_timeout is given up on and recorded as a TimeoutError. - "the shutdown should have completed within {int}ms" bounds the most recent shutdown, parsed the way the event step's bound is. - "the provider metadata name should not be empty" asks the provider for get_metadata() and requires a non-blank string. Capabilities and flags: - @large-integers is a declarable capability. Python's int is unbounded, so both in-memory self-tests declare it. - The six new flags are transcribed into canonical_flag_set(), and a test checks the transcription against canonical-flags.json value for value and Python type for Python type, so 10.0 stays a float and false, 0 and "" stay values. - The in-memory self-tests stop declaring @numeric-coercion: the SDK's InMemoryProvider hands values back untouched and the client's type check is isinstance-based, so 10.0 requested as an integer is a TYPE_MISMATCH rather than 10. The lossless scenarios exist to catch exactly that, and the capability is optional, so the honest declaration is to leave it out. Recorded as finding 3 in the README. The lifecycle steps have no canonical scenario running them here, because neither in-memory suite declares @lifecycle, so test_lifecycle_steps drives them against a recording provider instead. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #410
Part of open-feature/spec#417 — the cross-language tracking issue for the provider conformance suite. Java is the reference (java-sdk-contrib#1830); Go is in review (go-sdk-contrib#940).
What this is
A conformance suite any Python provider can adopt to verify it implements the provider contract — the Python implementation of Appendix F, running the same Gherkin scenarios, against the same canonical flag set, driven through the same control API as every other language's TCK. That shared basis is the point: "conformant" only means something if the question is identical everywhere.
It uses pytest-bdd, the same runner
openfeature-provider-flagdandopenfeature-flagd-api-testkitalready depend on, so an adopting package gains no new test framework.The adoption surface
One fixture and one call:
No
conftest.py, and nothing to import for the steps. The vocabulary ships as a pytest plugin registered through apytest11entry point. This is the nicest of the three implementations so far, and it is not an accident — pytest-bdd resolves steps through the fixture system, and fixtures from an installed plugin are visible to every test.The feature files and canonical flag set are packaged with the distribution, so adopting needs no git submodule.
Four things I probed rather than assumed
Having a working toolchain this time, each of these was verified against pytest-bdd 8.1 before the design depended on it:
@configuration-change?getattr(pytest.mark, tag)handles thempytest.skip()from an autouse fixture report skipped with the reason?scenarios()accept an absolute path into an installed package?conftest.pyTwo things that only showed up by running it, both now fixed and commented:
PytestUnknownMarkWarning— noise at best, a hard failure under-W error. The plugin registers them inpytest_configure.request.fixturenames. pytest-bdd resolves a step's fixtures lazily as each step runs, sotck_configis not infixturenamesat setup time, and@unavailablescenarios ran against a config that never declared it. The gate now keys off the node's markers, which are on the item itself.That second one is exactly the failure mode the suite exists to prevent — a gate that looks right and quietly passes everything — so it is pinned by a test.
Capabilities
A scenario whose capability was not declared is reported as skipped, with the reason — never as passed:
Self-tests
test_in_memory_conformanceInMemoryProvidertest_controllable_conformanceControllableInMemoryProvidertest_in_process_controlInProcessControlThere is no multi-provider suite because Python has no multi-provider — worth noting as its own gap.
Findings
1. A boolean satisfies an Integer request — and this is Python-only
boolean-flagevaluated throughget_integer_detailsreturnsTrue, reasonSTATIC, no error code. The specification requires the code default andTYPE_MISMATCH.boolis a subclass ofintin Python, soisinstance(True, int)isTrueand the check passes. The application gets a value that behaves as1, with nothing to indicate anything went wrong.The identical scenario passes in Java and Go. No suite in another language could ever have caught this — which is a fair advertisement for the "multiple implementations" argument, and arrived on the suite's first real run.
Tracked as open-feature/python-sdk#619. The self-test marks that one row
xfail(strict=True)with a pointer to the issue, so it stays visible in the report and fails the moment it starts passing, which forces the marker's removal when the SDK is fixed.2. The in-memory provider cannot update its flag set
Appendix A requires it, and Python's copies its mapping in the constructor and exposes nothing to change it. Same class of gap as go-sdk#530, found independently in a second SDK.
Only half the machinery is missing —
AbstractProvideralready suppliesemit_provider_configuration_changed— soControllableInMemoryProviderhere is a small subclass, not a reimplementation: every resolution decision is still the SDK's. It should port back as a method. Tracked as open-feature/python-sdk#620.Verification
pytest testsruff check(repo config)mypy --strictRun in a clean venv against
openfeature-sdk0.8.4.Known gaps
openfeature-flagd-api-testkitalready does for the flagd harness.Open questions
tools/openfeature-provider-tckthe right home, alongside the flagd testkit?xfail(strict=True)for a known SDK deviation is a local answer to spec#417's open question 4 ("is a known-deviations concept needed?"). Does that shape look right before it becomes a pattern?ControllableInMemoryProviderlive here at all, or should the SDK fix land first and this package depend on it?