Hypothesis reported failures are sometimes hard to diagnose, even though our assertions go to incredible lengths to give useful diagnostics. It is often not easy to reproduce an exact set of parameters which trigger a test failure.
As suggested in #379, we want to emit on failure the exact inputs which triggered it. [1]
A typical test case looks like this:
@given(arg=hypothesis_strategy)
def test_something(arg):
arg2 = further_hypothesis_magic
out = xp.function_under_test(arg1, arg2)
assertions_on_out
and a "low-tech" solution is to convert it into
@given(arg=hypothesis_strategy)
def test_something(arg):
arg2 = further_hypothesis_magic
repro_snippet = f"xp.function_under_test({arg1!r}, {arg2!r})" # exactly the r.h.s. of `out = ` below
try:
# as before
out = xp.function_under_test(arg1, arg2)
assertions_on_out
except Exception as exc:
# attach a failing snippet and reraise
exc.add_note(repro_snippet)
raise
See #384 for a worked example.
[1] gh-379 gives two ideas: declare "dependencies" of each test, and emit a "reproducible snippet" on failure. This issue only tracks progress towards the latter.
The plan is transform all tests in this manner:
- a test module per commit
- list the commits in
.git-blame-ignore-revs for git blame to skip the whitespace changes.
Here's the tracker:
These test modules are different enough to not need "repro snippets", as their use of hypothesis is minimal to not at all:
- test_signatures.py : does not use hypothesis
- test_inspection_functions.py: does not use hypothesis
- test_has_names.py
- test_constants.py
- test_special_cases.py
cc @cbourjau @mtsokol
Hypothesis reported failures are sometimes hard to diagnose, even though our assertions go to incredible lengths to give useful diagnostics. It is often not easy to reproduce an exact set of parameters which trigger a test failure.
As suggested in #379, we want to emit on failure the exact inputs which triggered it. [1]
A typical test case looks like this:
and a "low-tech" solution is to convert it into
See #384 for a worked example.
[1] gh-379 gives two ideas: declare "dependencies" of each test, and emit a "reproducible snippet" on failure. This issue only tracks progress towards the latter.
The plan is transform all tests in this manner:
.git-blame-ignore-revsforgit blameto skip the whitespace changes.Here's the tracker:
These test modules are different enough to not need "repro snippets", as their use of hypothesis is minimal to not at all:
cc @cbourjau @mtsokol