feat: add Appendix F, provider conformance (TCK) - #423
Conversation
OpenFeature's central promise is that swapping providers does not change application behaviour, and nothing verifies it. Every provider tests itself differently, against its own backend, with its own harness, so "implements the provider contract" is an unverified claim. Add the language-agnostic artifacts a shared provider conformance suite needs, plus an appendix explaining how they fit together for implementers in other languages. A new appendix rather than an extension of Appendix B, because the two answer different questions. Appendix B validates an SDK using an in-memory provider as a stand-in; this validates a provider against a real backend. Appendix F also needs more than Gherkin: the scenarios, the flag set they assume and the control API that produces outages are three co-dependent artifacts, and an OpenAPI document has no home under "Gherkin Suites". Both appendices now cross-reference each other. Assets live under specification/assets/provider-tck/ rather than alongside the existing SDK gherkin — those already contain an evaluation.feature that means something different. Marked experimental and explicitly non-normative. Nothing here is expressed as a numbered requirement yet; whether the control API contract and the capability tag vocabulary should become normative sections is a TSC decision, and is recorded as an open question rather than assumed. The first implementation is tools/provider-tck in open-feature/java-sdk-contrib, in review alongside this. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueComment |
…pec submodule
The Gherkin, the canonical flag set and the control API document are not Java
artifacts. They are language-agnostic definitions of the provider contract that
every language's TCK must agree on byte for byte, and they only lived in this
module because the proof of concept had to start somewhere.
They now live in open-feature/spec as Appendix F, under
specification/assets/provider-tck/, and are copied in from the `spec` git
submodule at generate-resources — the same mechanism tools/flagd-api-testkit
already uses for the flagd test harness. The copies are git-ignored and carry a
do-not-edit note; changes belong in the spec repo and arrive here by bumping the
submodule.
Consumers are unaffected: the artifacts are still packaged into the release JAR,
@SelectClasspathResource("features") still resolves, and nobody needs a submodule
of their own. Verified byte-identical after the round trip.
The in-memory CI job now checks out submodules, since without them there is no
suite to run.
DEPENDS ON open-feature/spec#423. The submodule is pinned to that PR's branch
commit rather than to a commit on the spec repo's main branch. That is reachable,
so CI can fetch it, but it must be re-pinned to main once #423 merges and before
this lands.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…pec submodule
The Gherkin, the canonical flag set and the control API document are not Java
artifacts. They are language-agnostic definitions of the provider contract that
every language's TCK must agree on byte for byte, and they only lived in this
module because the proof of concept had to start somewhere.
They now live in open-feature/spec as Appendix F, under
specification/assets/provider-tck/, and are copied in from the `spec` git
submodule at generate-resources — the same mechanism tools/flagd-api-testkit
already uses for the flagd test harness. The copies are git-ignored and carry a
do-not-edit note; changes belong in the spec repo and arrive here by bumping the
submodule.
Consumers are unaffected: the artifacts are still packaged into the release JAR,
@SelectClasspathResource("features") still resolves, and nobody needs a submodule
of their own. Verified byte-identical after the round trip.
The in-memory CI job now checks out submodules, since without them there is no
suite to run.
DEPENDS ON open-feature/spec#423. The submodule is pinned to that PR's branch
commit rather than to a commit on the spec repo's main branch. That is reachable,
so CI can fetch it, but it must be re-pinned to main once #423 merges and before
this lands.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
lifecycle.feature was gated by @events, which is wrong in both directions. Too strict: a stateless HTTP provider such as OFREP emits no events of its own and so cannot declare @events, yet the readiness scenario is not really about events at all. Too lax: every SDK synthesises PROVIDER_READY for a provider with no initialisation step -- the Go SDK says so in a comment, "a provider without state handling capability can be assumed to be ready immediately" -- so a provider that declares @events passes the readiness scenario without demonstrating anything. A no-op provider passes it identically. The two are different claims. @events is "emits lifecycle events at all"; @lifecycle is "performs an initialisation that reaches its backend, with an observable outcome". A provider may do either without the other. Found by adopting the suite in the OFREP providers for Go and Java, where two independent adoptions withheld @events for the same reason and neither could express what they actually lacked. Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
lifecycle.feature was tagged @events, which is wrong in both directions.
Too strict: a stateless provider such as OFREP emits no events of its own
and cannot declare @events, yet it still initialises against a backend and
still owes the lifecycle contract.
Too lax: FeatureProviderStateManager emits PROVIDER_READY/PROVIDER_ERROR
around initialize for any provider, whether or not it is an EventProvider.
So a provider that does no initialisation of its own reaches READY exactly
as NoOpProvider would, and the readiness scenario passes vacuously.
Adds Capability.LIFECYCLE ("@lifecycle") -- performs an initialisation that
reaches its backend, with an observable outcome -- and re-vendors
lifecycle.feature verbatim from the spec assets, where the feature-level tag
is now @lifecycle (spec dfa16586, PR open-feature/spec#423).
flagd declares LIFECYCLE in both resolver modes: RPC does a round trip and
in-process syncs the whole ruleset during initialisation, so the scenarios
assert something real there.
Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…odule The feature files, canonical flag set and control-API document are owned by open-feature/spec. Vendoring them here made this repository a second place the definition of conformance could drift, which is precisely what the suite exists to prevent. They are now a pinned submodule at libs/shared/provider-tck/spec and the copies are gone. Adopters are unaffected, and that is the constraint the change had to respect: the rollup asset globs copy the artifacts out of the submodule and into the published package, so installing @openfeature/provider-tck from npm still needs no submodule and no particular repository layout. resolveAssetDir therefore has to satisfy two layouts -- the packaged copy next to the bundle, and the submodule under the library root -- and tries both. The spec calls the feature directory `gherkin`; the package keeps the name the API talks about. Contributors do need the submodule: without it no feature file loads at all. `nx test` and `nx package` depend on a pullSpec target that initialises it, and CI already checks out with `submodules: recursive`. Prettier is pointed at the submodule instead of the old vendored paths so it never rewrites artifacts that are consumed byte for byte by every language's TCK. The .gitattributes normalising those files to LF goes with them; the equivalent lives upstream, where the files now do. Pinned to dfa1658 (open-feature/spec#423). Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
…served Found by adopting the suite in the Python OFREP provider, which is the first stateless adopter. The launchpad returns from /start as soon as flagd answers its readiness probe, roughly 40ms before its file sources reach the flag store. An evaluation in that window gets FLAG_NOT_FOUND for a flag the configuration plainly defines. Every previous adopter is a provider that blocks during initialisation -- streaming, or syncing a ruleset -- which absorbs the window silently. A stateless provider evaluates over HTTP with no initialisation at all, so it has nothing to hide the warm-up behind: the first run failed 22 of 24 scenarios and read as a catastrophically broken provider rather than as a racing testbed. The contract said /start MUST reseed flag state, but never said it returns only once that state is being served. Those should be the same instant. Leaving it implicit means every stateless adopter reimplements the same defensive probe, and the ones that do not get a suite that looks broken for reasons that have nothing to do with the provider under test. Part of #417 Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
4661068 to
36392a9
Compare
| @@ -0,0 +1,82 @@ | |||
| { | |||
There was a problem hiding this comment.
I wonder if the canonical flags should also be gherkin? They could be a very large "given", I guess.
There was a problem hiding this comment.
i can see the motivition behind it. but i am not sure it will be easier to read or understand or parse. Especially a json is easily transformed into another JSON for eg. CLI's to prefill databases etc. Not so sure about the gherkin for this purpose.
| Scenario: An integer flag resolves as an integer | ||
| # Paired with the float scenario below and with the narrowing scenario in errors.feature. | ||
| # Together they pin down that the two numeric types stay distinct rather than both being | ||
| # funnelled through one numeric representation. | ||
| Given a Integer-flag with key "integer-flag" and a default value "1" | ||
| When the flag was evaluated with details | ||
| Then the resolved details value should be "10" | ||
| And the error-code should be "" | ||
| And no exception should have been thrown | ||
|
|
||
| Scenario: A float flag resolves as a float | ||
| Given a Float-flag with key "float-flag" and a default value "0.1" | ||
| When the flag was evaluated with details | ||
| Then the resolved details value should be "0.5" | ||
| And the error-code should be "" | ||
| And no exception should have been thrown |
There was a problem hiding this comment.
What about integer precision? I think a lot of our SDKs support resolutions above 32 bit (though some like JS are practically capped at 2⁵³... it might be worth testing at least up to 2⁵³ -1
There was a problem hiding this comment.
fully agree, but for now, i would love to focus on the method, rather than the tests. We should agree and finalize the basic process. Afterwards we can add more tests to ensure compatibility
|
I have to say, the Java impl is remarkably small and clean in Java. I think there's a lot of value here. |
|
I really like the idea, and the impl is clean (though TBH the issue and PR description is very wordy and maybe could be compressed and made less verbose) it took me a long time to get through 😅 . I have 2 things I think I'd want to understand additionally:
|
Co-authored-by: Todd Baert <todd.baert@dynatrace.com> Signed-off-by: Simon Schrottner <simon@schrottner.at>
|
I am in total favor for extensibility. This would even help our providers. We could try to define defaults in java, like a default resource path and a default step path, add those to the test, and have a simply way for the beginning. But also I am not sure, if this would be something which we should delay to the next version. For now, there is the idea, but do we need it immediately to ensure conformance. It sounds like a nice to have feature. A good iteration on the tck, i feel like it could bloat the current efforts. wdyt? I measured what the Java classpath actually does here — the defaults idea works, with one catchFour measurements against the real module:
The third is the catch. If the default resource path is just plus one extra glue package on the base suite ( Net cost: two lines in If it does land, I'd word it in Appendix F as an outcome rather than a mechanism — an adopter can add scenarios and steps that run in the same backend phase without redeclaring the canonical set — since it's pure convention in Java and Python, while Go and JS have no runtime scanning and need one registration callback. Worth separating from the above either way: the suite should verify its canonical scenarios actually ran. That is not about extensibility — the same footgun already exists as |
What
Adds Appendix F: Provider Conformance (TCK) — a language-agnostic conformance suite for validating OpenFeature providers, plus the three artifacts it rests on.
Marked experimental and explicitly non-normative. This is a proof of concept and the scenario set is a representative subset, not exhaustive coverage.
Why
OpenFeature's central promise is that swapping providers does not change application behaviour, and nothing currently verifies it. Every provider tests itself differently, against its own backend, with its own harness — so "implements the provider contract" is an unverified claim, and a behavioural difference between two providers gets discovered by the application that trips over it.
Why a new appendix rather than extending Appendix B
They answer different questions:
Appendix F also needs more than Gherkin. The scenarios, the flag set they assume, and the control API that produces outages are three co-dependent artifacts — a feature file that evaluates
boolean-flagis meaningless without the flag definition, and a disconnect scenario is meaningless without the endpoint that produces the disconnect. An OpenAPI document has no home under "Gherkin Suites".Both appendices now cross-reference each other.
Contents
specification/appendix-f-provider-conformance.md— the appendix, written for implementers in other languages: what the artifacts are, the control-API contract and its two invariants, the capability tag vocabulary, and the six things a language implementation ownsspecification/assets/provider-tck/gherkin/— four feature files (evaluation, errors, events, lifecycle)specification/assets/provider-tck/flags/canonical-flags.json— the flag set every scenario assumesspecification/assets/provider-tck/openapi/control-api.yaml— what a backend under test must exposespecification/assets/provider-tck/README.md— a landing page for the directory, matching the convention ofassets/gherkin/README.mdAssets live under
assets/provider-tck/rather than alongside the existing SDK Gherkin, becauseassets/gherkin/evaluation.featurealready exists and means something different.A
.gitattributesnormalises these files to LF. They are consumed byte for byte by every language's TCK and copied verbatim into published build artifacts, so a Windows checkout must not produce a different packaged file than a Linux one.Design decisions worth reviewing
Capabilities. Not every provider implements every optional part of the contract. Scenarios exercising an optional capability carry a tag; a provider declares what it supports and the rest are reported as skipped with a reason, never as passed. The rule behind it: a conformance suite that quietly goes green on scenarios it did not run is worse than no suite at all.
@strict-numeric-typingis not really optional. The spec requiresTYPE_MISMATCHwhen the requested type cannot be satisfied, and narrowing0.5to0loses information silently. It is a capability only so a provider with this defect can adopt the suite today and see the gap reported explicitly. Not declaring it is an admission of a known bug.No container restarts. Unavailability is simulated inside the running stack via the control API. Orchestrators assign host ports dynamically and cannot reliably preserve them across a restart, so restarting silently invalidates every provider pointed at the old port — and the failure looks like a flaky provider.
In-process control is a narrow carve-out. Providers with no backend (in-memory, environment-variable, file-based) may be driven in-process. Providers with a backend must use the control API; reaching an external backend through a side channel produces a suite that passes while proving nothing.
Open questions
GET /last-evaluationor similar). Until then a provider that silently drops the context passes. The@targetingtag is reserved for those scenarios.Reference implementation
tools/provider-tckin open-feature/java-sdk-contrib#1830, adopted by the flagd provider for both its RPC and in-process resolvers. A follow-up PR there switches that module to consume these files from this repo via submodule, and verified the artifacts round-trip byte-identical.Which of #417's open questions this answers
Q1, directory layout — proposes
specification/assets/provider-tck/{gherkin,flags,openapi}/. The subdirectory is forced rather than chosen:assets/gherkin/evaluation.featurealready exists and tests an SDK, not a provider.Q4,
@strict-numeric-typingas capability vs hard requirement — keeps it a capability so providers with the defect can adopt today, but the appendix says in prose that it is not an optional feature and that not declaring it is an admission of a known bug. The "known deviations" idea from #417 is left open.Left open, unchanged by this PR: Q2 versioning, Q3 canonical flag-set format, Q5 context passthrough, Q6 reason values, Q7 conformance claims. Q5 is also recorded in the appendix's own "Open questions" section.
What the suite has found so far
Two real defects, from the outside, which is the argument for the whole exercise:
MultiProviderswallows child provider events — it extendsEventProviderbut never subscribes to its children, soPROVIDER_CONFIGURATION_CHANGED,PROVIDER_ERRORandPROVIDER_STALEnever reach the client. Independently reproduced by running the suite against it; already tracked as open-feature/java-sdk#1882 (gap 1, High), where it had been found by hand-comparing against the js-sdk reference.