You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Branches = true/false branches for each keyword (except for keywords that don't branch such as annotation-only keywords like title and description)
Suggestion
Set coverage thresholds of
100% for lines
100% for keywords (statements)
100% for subschemas (functions)
no threshold for branches
Rationale
Lines and Statements
The natural threshold for lines and statements is 100% - if we have a line/keyword in our schema that we don't touch in at least one test case, then someone could remove that line/keyword and the automated PR status check for schema tests would still be green.
It also helps spotting test cases that fail for the wrong reason, such as the two test cases that are now being fixed by
the corresponding two branches can be covered with a "pass" test case providing an object, and a "fail" test case providing an array, which leaves both good cases (providing a boolean) or bad cases (providing a number, a string, or null) uncovered.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
The new JSON Schema Test Coverage tool we are now using allows to set separate coverage thresholds for
titleanddescription)Suggestion
Set coverage thresholds of
Rationale
Lines and Statements
The natural threshold for lines and statements is 100% - if we have a line/keyword in our schema that we don't touch in at least one test case, then someone could remove that line/keyword and the automated PR status check for schema tests would still be green.
It also helps spotting test cases that fail for the wrong reason, such as the two test cases that are now being fixed by
These were intended to cover schema lines
OpenAPI-Specification/src/schemas/validation/schema.yaml
Lines 604 to 608 in 805084d
and failed before reaching that part of the schema due to another unintended failure condition.
Subschemas (Functions)
Subschema coverage includes
unevaluatedPropertieswhich takes a subschema as its argument, andpropertieswhich takes a map as its argument where the map values are subschemas.This means that we can only be sure to have covered all properties in our schema if we strive for subschema (function) coverage of 100%.
Branches
Covering 100% of all branches seems less useful because for a keyword use such as
OpenAPI-Specification/src/schemas/validation/schema.yaml
Lines 844 to 846 in 805084d
the corresponding two branches can be covered with a "pass" test case providing an object, and a "fail" test case providing an array, which leaves both good cases (providing a boolean) or bad cases (providing a number, a string, or null) uncovered.
All reactions