Add fuzz test, property-based test, and table test for PV function by paulzuradzki · Pull Request #100 · numpy/numpy-financial

paulzuradzki · 2024-01-07T21:11:43Z

Description

This change request adds testing to the pv / present value function.

Tests
- Adds fuzz tests and property-based tests (PBT) using Hypothesis
- Adds more example-based tests (with pytest.mark.parametrize)
- The hope is that these tests can be extended to other functions as well.
Add exception handling for InvalidOperation, TypeError, DivisionByZero, Overflow. ~~I opted to mirror numpy and return -0.0. Arguably, we would want a runtime error and can ignore it in the fuzz tests.~~

Context:

We use fuzz tests for extreme value testing (e.g., empty values, very low, very high, or zero) using a randomized input generator.
We use property-based tests to verify that certain conditions hold true across a range of inputs.
I used type hints and Hypothesis Ghostwriter CLI feature to help write the strategy.
- We can generate random np.ndarray of small dimensions, but this was a bit slow. I opted to test array-input directly (tests/test_financial.py::TestPV::test_pv_examples[rates_as_array).

Testing

If you want to select only the related tests, you can run them like so.

$ pytest tests/test_financial.py::TestPV -vv
tests/test_financial.py::TestPV::test_pv PASSED [ 9%]
tests/test_financial.py::TestPV::test_pv_decimal PASSED [ 18%]
tests/test_financial.py::TestPV::test_pv_examples[default_fv_and_when] PASSED [ 27%]
tests/test_financial.py::TestPV::test_pv_examples[specify_fv_and_when] PASSED [ 36%]
tests/test_financial.py::TestPV::test_pv_examples[when_1] PASSED [ 45%]
tests/test_financial.py::TestPV::test_pv_examples[when_1_and_fv_1000] PASSED [ 54%]
tests/test_financial.py::TestPV::test_pv_examples[fv>0] PASSED [ 63%]
tests/test_financial.py::TestPV::test_pv_examples[negative_rate] PASSED [ 72%]
tests/test_financial.py::TestPV::test_pv_examples[rates_as_array] PASSED [ 81%]
tests/test_financial.py::TestPV::test_pv_fuzz PASSED [ 90%]
tests/test_financial.py::TestPV::test_pv_interest_rate_sensitivity PASSED [100%]
====================================================================================== 11 passed in 0.82s ======================================================================================

paulzuradzki · 2024-01-07T21:12:47Z

Here are some of the example errors caught in fuzz testing. We can handle or ignore these cases.

InvalidOperation

rate=fraction_to_decimal(Fraction(0, 1)),
nper=0,
pmt=Decimal('NaN'),
fv=Decimal('NaN'),
when='end',
rate=fraction_to_decimal(Fraction(0, 1)),
nper=array([0], dtype=int8),
pmt=Decimal('NaN'),
fv=Decimal('NaN'),
when='end',

ValueError


Trying example: test_pv_fuzz(
self=<tests.test_financial.TestPV object at 0x11179c550>,
rate=0.0,
nper=array([False, False]),
pmt=array([False, False, False]),
fv=1.9565376279601164e+16,
when='begin',
)
> return -(fv + pmt * fact) / temp
E ValueError: operands could not be broadcast together with shapes (3,) (2,)

DivisionByZero

E Falsifying example: test_pv_fuzz(
E self=<tests.test_financial.TestPV object at 0x109ba8e50>,
E rate=1,
E nper=Decimal('-Infinity'),
E pmt=0,
E fv=1,
E when='end', # or any other generated value
E )
numpy_financial/_financial.py:607: DivisionByZero

Overflow

 | decimal.Overflow: [<class 'decimal.Overflow'>]
| Falsifying example: test_pv_fuzz(
| self=<tests.test_financial.TestPV object at 0x12059c790>,
| rate=fraction_to_decimal(Fraction(15150, 123698847)),
| nper=fraction_to_decimal(Fraction(141745421979347, 7539)),
| pmt=Decimal('NaN'),
| fv=0,
| when='end',
| )
+---------------- 2 ----------------
| Traceback (most recent call last):
| File "/Users/pzuradzki/repos/numpy-financial/tests/test_financial.py", line 143, in test_pv_fuzz
| npf.pv(rate, nper, pmt, fv, when)
| File "/Users/pzuradzki/repos/numpy-financial/numpy_financial/_financial.py", line 605, in pv
| temp = (1 + rate) ** nper
| ~~~~~~~~~^~~~~~~~~
| decimal.Overflow: [<class 'decimal.Overflow'>]
| Falsifying example: test_pv_fuzz(
| self=<tests.test_financial.TestPV object at 0x12059c790>,
| rate=fraction_to_decimal(Fraction(3, 1)),
| nper=fraction_to_decimal(Fraction(1660965, 1)),
| pmt=Decimal('NaN'),
| fv=Decimal('NaN'),
| when='end',
| )
+------------------------------------

paulzuradzki · 2024-01-07T21:17:32Z


-def pv(rate, nper, pmt, fv=0, when='end'):
+def pv(
+ rate: Union[int, float, Decimal, np.ndarray],


I added type hints here, but I see that they're not conventional in the repo. NP to remove the type hints if preferred for consistency; else, can keep it as optional.
Using Union since the package supports Python 3.9. Starting in Python 3.10, we can do int|float|Decimal|np.ndarray.

Type hints are great. Adding type hints is actually part of what I'll be working on once I have the broadcasting done.

paulzuradzki · 2024-01-07T21:21:04Z

@@ -90,13 +93,167 @@ def test_decimal_with_when(self):


 class TestPV:


fwiw, I would remove the test classes and rely on function naming convention (ex: test_pv*). Out of scope.

I agree on principle, however having them in classes makes it easier to run groups of tests for each function.

paulzuradzki · 2024-01-07T21:22:25Z

+ },
+ }
+
+ # Randomized input strategies for fuzz tests & property-based tests


Providing a "strategy" limits the values that the fuzzer (Hypothesis) will supply. The strategies that we re-use are defined here.

Kai-Striega · 2024-01-07T22:51:45Z

Thanks for taking the time to contribute. This looks excellent!

I'm a bit short of time today/tomorrow. I'll try to get a full review done ASAP as there are a couple of other people touching these tests.

FYI I'm currently working on a broadcasting + numba rewrite so many of these errors should be fixed after that.

Kai-Striega · 2024-01-07T23:01:00Z



 [tool.poetry.group.test.dependencies]
+hypothesis = "^6.92.2"


I think adding hypothesis is the right idea here, however I don't have much experience with it. It might take me a while to thoroughly understand this

No problem. Here are some links that I found helpful depending how far down the rabbit hole you want to go with it.
What is Property Based Testing? - Hypothesis
Quick start guide — Hypothesis 6.92.2 documentation
Beyond Units by Hillel Wayne
Choosing properties for property-based testing | F# for fun and profit (fsharpforfunandprofit.com)

Thanks! I've read the Hypothesis documentation and done some basic property based tests, but not much beyond that.

paulzuradzki · 2024-01-07T23:14:24Z

+ expected = float(npf.pv(rate=rate + 0.1, nper=nper, pmt=pmt, when=when))
+
+ # As interest rate increases, present value decreases
+ assert round(result, 4) >= round(expected, 4)


I tried to use something like result >= pytest.approx(expected), but pytest.approx doesn't work with floats and greater-than-or-equal-to operator.

remove logging

paulzuradzki · 2024-01-07T23:32:47Z

+ Decimal("-127128.1709461939327295222005"),
+ )
+
+ @pytest.mark.parametrize("test_case", test_cases.values(), ids=test_cases.keys())


ids= paramater gets us test case labels.
With ids
$ pytest tests/test_financial.py::TestPV::test_pv_examples -vv tests/test_financial.py::TestPV::test_pv PASSED [ 9%] tests/test_financial.py::TestPV::test_pv_decimal PASSED [ 18%] tests/test_financial.py::TestPV::test_pv_examples[default_fv_and_when] PASSED [ 27%] tests/test_financial.py::TestPV::test_pv_examples[specify_fv_and_when] PASSED [ 36%] tests/test_financial.py::TestPV::test_pv_examples[when_1] PASSED [ 45%] tests/test_financial.py::TestPV::test_pv_examples[when_1_and_fv_1000] PASSED [ 54%] tests/test_financial.py::TestPV::test_pv_examples[fv>0] PASSED [ 63%] tests/test_financial.py::TestPV::test_pv_examples[negative_rate] PASSED [ 72%] tests/test_financial.py::TestPV::test_pv_examples[rates_as_array] PASSED [ 81%] tests/test_financial.py::TestPV::test_pv_fuzz PASSED [ 90%] tests/test_financial.py::TestPV::test_pv_interest_rate_sensitivity PASSED [100%]
Without ids (no labels)
$ pytest tests/test_financial.py::TestPV::test_pv_examples -vv tests/test_financial.py::TestPV::test_pv_examples[test_case0] PASSED [ 27%] tests/test_financial.py::TestPV::test_pv_examples[test_case1] PASSED [ 36%] tests/test_financial.py::TestPV::test_pv_examples[test_case2] PASSED [ 45%] tests/test_financial.py::TestPV::test_pv_examples[test_case3] PASSED [ 54%] tests/test_financial.py::TestPV::test_pv_examples[test_case4] PASSED [ 63%] tests/test_financial.py::TestPV::test_pv_examples[test_case5] PASSED [ 72%] tests/test_financial.py::TestPV::test_pv_examples[test_case6] PASSED [ 81%]

Kai-Striega · 2024-03-11T23:46:00Z

@paulzuradzki sorry for not being in touch. I moved states and then had some health issues. I'm back now.

This work looks great and is definitely a candidate for merging. I'll get a formal review done this week if you are still interested in contributing.

add PBT and table tests for PV func
34eebab

paulzuradzki force-pushed the tests/add-hypothesis-pbt-and-fuzz-tests branch from aaccf23 to 34eebabCompare January 7, 2024 21:14