Parameterized testing with any Python test framework

Parameterized testing in Python sucks.

parameterized fixes that. For everything. Parameterized testing for nose, parameterized testing for py.test, parameterized testing for unittest.

# test_math.pyfromnose.toolsimportassert_equalfromparameterizedimportparameterized, parameterized_classimportunittestimportmath@parameterized([ (2, 2, 4), (2, 3, 8), (1, 9, 1), (0, 9, 0),])deftest_pow(base, exponent, expected):
assert_equal(math.pow(base, exponent), expected)
classTestMathUnitTest(unittest.TestCase):
@parameterized.expand([ ("negative", -1.5, -2.0), ("integer", 1, 1.0), ("large fraction", 1.6, 1), ])deftest_floor(self, name, input, expected):
assert_equal(math.floor(input), expected)
@parameterized_class(('a', 'b', 'expected_sum', 'expected_product'), [ (1, 2, 3, 2), (5, 5, 10, 25),])classTestMathClass(unittest.TestCase):
deftest_add(self):
assert_equal(self.a+self.b, self.expected_sum)
deftest_multiply(self):
assert_equal(self.a*self.b, self.expected_product)
@parameterized_class([ { "a": 3, "expected": 2 }, { "b": 5, "expected": -4 },])classTestMathClassDict(unittest.TestCase):
a=1b=1deftest_subtract(self):
assert_equal(self.a-self.b, self.expected)

With nose (and nose2):

$ nosetests -v test_math.py
test_floor_0_negative (test_math.TestMathUnitTest) ... ok
test_floor_1_integer (test_math.TestMathUnitTest) ... ok
test_floor_2_large_fraction (test_math.TestMathUnitTest) ... ok
test_math.test_pow(2, 2, 4, {}) ... ok
test_math.test_pow(2, 3, 8, {}) ... ok
test_math.test_pow(1, 9, 1, {}) ... ok
test_math.test_pow(0, 9, 0, {}) ... ok
test_add (test_math.TestMathClass_0) ... ok
test_multiply (test_math.TestMathClass_0) ... ok
test_add (test_math.TestMathClass_1) ... ok
test_multiply (test_math.TestMathClass_1) ... ok
test_subtract (test_math.TestMathClassDict_0) ... ok
----------------------------------------------------------------------
Ran 12 tests in 0.015s
OK

As the package name suggests, nose is best supported and will be used for all further examples.

With py.test (version 2.0 and above):

$ py.test -v test_math.py
============================= test session starts ==============================
platform darwin -- Python 3.6.1, pytest-3.1.3, py-1.4.34, pluggy-0.4.0
collecting ... collected 13 items
test_math.py::test_pow::[0] PASSED
test_math.py::test_pow::[1] PASSED
test_math.py::test_pow::[2] PASSED
test_math.py::test_pow::[3] PASSED
test_math.py::TestMathUnitTest::test_floor_0_negative PASSED
test_math.py::TestMathUnitTest::test_floor_1_integer PASSED
test_math.py::TestMathUnitTest::test_floor_2_large_fraction PASSED
test_math.py::TestMathClass_0::test_add PASSED
test_math.py::TestMathClass_0::test_multiply PASSED
test_math.py::TestMathClass_1::test_add PASSED
test_math.py::TestMathClass_1::test_multiply PASSED
test_math.py::TestMathClassDict_0::test_subtract PASSED
==================== 12 passed, 4 warnings in 0.16 seconds =====================

With unittest (and unittest2):

$ python -m unittest -v test_math
test_floor_0_negative (test_math.TestMathUnitTest) ... ok
test_floor_1_integer (test_math.TestMathUnitTest) ... ok
test_floor_2_large_fraction (test_math.TestMathUnitTest) ... ok
test_add (test_math.TestMathClass_0) ... ok
test_multiply (test_math.TestMathClass_0) ... ok
test_add (test_math.TestMathClass_1) ... ok
test_multiply (test_math.TestMathClass_1) ... ok
test_subtract (test_math.TestMathClassDict_0) ... ok
----------------------------------------------------------------------
Ran 8 tests in 0.001s
OK

(note: because unittest does not support test decorators, only tests created with @parameterized.expand will be executed)

With green:

$ green test_math.py -vvv
test_math
TestMathClass_1
. test_method_a
. test_method_b
TestMathClass_2
. test_method_a
. test_method_b
TestMathClass_3
. test_method_a
. test_method_b
TestMathUnitTest
. test_floor_0_negative
. test_floor_1_integer
. test_floor_2_large_fraction
TestMathClass_0
. test_add
. test_multiply
TestMathClass_1
. test_add
. test_multiply
TestMathClassDict_0
. test_subtract
Ran 12 tests in 0.121s
OK (passes=9)

Installation

$ pip install parameterized

Compatibility

Yes (mostly).

	Py3.7	Py3.8	Py3.9	Py3.10	Py3.11	PyPy3	@mock.patch
nose	yes	yes	yes	yes	no§	no§	yes
nose2	yes	yes	yes	yes	yes	yes	yes
py.test 2	no*	no*	no*	no*	no*	no*	no*
py.test 3	yes	yes	yes	yes	no*	no*	yes
py.test 4	no**	no**	no**	no**	no**	no**	no**
py.test fixtures	no†	no†	no†	no†	no†	no†	no†
unittest (@parameterized.expand)	yes	yes	yes	yes	yes	yes	yes
unittest2 (@parameterized.expand)	yes	yes	yes	yes	no§	no§	yes

§: nose and unittest2 - both of which were last updated in 2015 - sadly do not appear to support Python 3.10 or 3.11.

*: py.test 2 does not appear to work under Python 3 (#71), and py.test 3 does not appear to work under Python 3.10 or 3.11 (#154).

**: py.test 4 is not yet supported (but coming!) in issue #34

†: py.test fixture support is documented in issue #81

Dependencies

(this section left intentionally blank)

Exhaustive Usage Examples

The @parameterized and @parameterized.expand decorators accept a list or iterable of tuples or param(...), or a callable which returns a list or iterable:

fromparameterizedimportparameterized, param# A list of tuples@parameterized([ (2, 3, 5), (3, 5, 8),])deftest_add(a, b, expected):
assert_equal(a+b, expected)
# A list of params@parameterized([param("10", 10),param("10", 16, base=16),])deftest_int(str_val, expected, base=10):
assert_equal(int(str_val, base=base), expected)
# An iterable of params@parameterized(param.explicit(*json.loads(line))forlineinopen("testcases.jsons"))deftest_from_json_file(...):
...
# A callable which returns a list of tuplesdefload_test_cases():
return [
("test1", ),
("test2", ),
]
@parameterized(load_test_cases)deftest_from_function(name):
...

Note that, when using an iterator or a generator, all the items will be loaded into memory before the start of the test run (we do this explicitly to ensure that generators are exhausted exactly once in multi-process or multi-threaded testing environments).

The @parameterized decorator can be used test class methods, and standalone functions:

fromparameterizedimportparameterizedclassAddTest(object):
@parameterized([ (2, 3, 5), ])deftest_add(self, a, b, expected):
assert_equal(a+b, expected)
@parameterized([ (2, 3, 5),])deftest_add(a, b, expected):
assert_equal(a+b, expected)

And @parameterized.expand can be used to generate test methods in situations where test generators cannot be used (for example, when the test class is a subclass of unittest.TestCase):

importunittestfromparameterizedimportparameterizedclassAddTestCase(unittest.TestCase):
@parameterized.expand([ ("2 and 3", 2, 3, 5), ("3 and 5", 3, 5, 8), ])deftest_add(self, _, a, b, expected):
assert_equal(a+b, expected)

Will create the test cases:

$ nosetests example.py
test_add_0_2_and_3 (example.AddTestCase) ... ok
test_add_1_3_and_5 (example.AddTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK

Note that @parameterized.expand works by creating new methods on the test class. If the first parameter is a string, that string will be added to the end of the method name. For example, the test case above will generate the methods test_add_0_2_and_3 and test_add_1_3_and_5.

The names of the test cases generated by @parameterized.expand can be customized using the name_func keyword argument. The value should be a function which accepts three arguments: testcase_func, param_num, and params, and it should return the name of the test case. testcase_func will be the function to be tested, param_num will be the index of the test case parameters in the list of parameters, and param (an instance of param) will be the parameters which will be used.

importunittestfromparameterizedimportparameterizeddefcustom_name_func(testcase_func, param_num, param):
return"%s_%s"%(
testcase_func.__name__,
parameterized.to_safe_name("_".join(str(x) forxinparam.args)),
)
classAddTestCase(unittest.TestCase):
@parameterized.expand([ (2, 3, 5), (2, 3, 5), ], name_func=custom_name_func)deftest_add(self, a, b, expected):
assert_equal(a+b, expected)

Will create the test cases:

$ nosetests example.py
test_add_1_2_3 (example.AddTestCase) ... ok
test_add_2_3_5 (example.AddTestCase) ... ok
----------------------------------------------------------------------
Ran 2 tests in 0.001s
OK

The param(...) helper class stores the parameters for one specific test case. It can be used to pass keyword arguments to test cases:

fromparameterizedimportparameterized, param@parameterized([param("10", 10),param("10", 16, base=16),])deftest_int(str_val, expected, base=10):
assert_equal(int(str_val, base=base), expected)

If test cases have a docstring, the parameters for that test case will be appended to the first line of the docstring. This behavior can be controlled with the doc_func argument:

fromparameterizedimportparameterized@parameterized([ (1, 2, 3), (4, 5, 9),])deftest_add(a, b, expected):
""" Test addition. """assert_equal(a+b, expected)
defmy_doc_func(func, num, param):
return"%s: %s with %s"%(num, func.__name__, param)
@parameterized([ (5, 4, 1), (9, 6, 3),], doc_func=my_doc_func)deftest_subtraction(a, b, expected):
assert_equal(a-b, expected)

$ nosetests example.py
Test addition. [with a=1, b=2, expected=3] ... ok
Test addition. [with a=4, b=5, expected=9] ... ok
0: test_subtraction with param(*(5, 4, 1)) ... ok
1: test_subtraction with param(*(9, 6, 3)) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK

Finally @parameterized_class parameterizes an entire class, using either a list of attributes, or a list of dicts that will be applied to the class:

fromyourapp.modelsimportUserfromparameterizedimportparameterized_class@parameterized_class([ { "username": "user_1", "access_level": 1 }, { "username": "user_2", "access_level": 2, "expected_status_code": 404 },])classTestUserAccessLevel(TestCase):
expected_status_code=200defsetUp(self):
self.client.force_login(User.objects.get(username=self.username)[0])
deftest_url_a(self):
response=self.client.get('/url')
self.assertEqual(response.status_code, self.expected_status_code)
deftearDown(self):
self.client.logout()
@parameterized_class(("username", "access_level", "expected_status_code"), [ ("user_1", 1, 200), ("user_2", 2, 404)])classTestUserAccessLevel(TestCase):
defsetUp(self):
self.client.force_login(User.objects.get(username=self.username)[0])
deftest_url_a(self):
response=self.client.get("/url")
self.assertEqual(response.status_code, self.expected_status_code)
deftearDown(self):
self.client.logout()

The @parameterized_class decorator accepts a class_name_func argument, which controls the name of the parameterized classes generated by @parameterized_class:

fromparameterizedimportparameterized, parameterized_classdefget_class_name(cls, num, params_dict):
# By default the generated class named includes either the "name"# parameter (if present), or the first string value. This example shows# multiple parameters being included in the generated class name:return"%s_%s_%s%s"%(
cls.__name__,
num,
parameterized.to_safe_name(params_dict['a']),
parameterized.to_safe_name(params_dict['b']),
)
@parameterized_class([ { "a": "hello", "b": " world!", "expected": "hello world!" }, { "a": "say ", "b": " cheese :)", "expected": "say cheese :)" },], class_name_func=get_class_name)classTestConcatenation(TestCase):
deftest_concat(self):
self.assertEqual(self.a+self.b, self.expected)

$ nosetests -v test_math.py
test_concat (test_concat.TestConcatenation_0_hello_world_) ... ok
test_concat (test_concat.TestConcatenation_0_say_cheese__) ... ok

Using with Single Parameters

If a test function only accepts one parameter and the value is not iterable, then it is possible to supply a list of values without wrapping each one in a tuple:

@parameterized([1, 2, 3])deftest_greater_than_zero(value):
assertvalue>0

Note, however, that if the single parameter is iterable (such as a list or tuple), then it must be wrapped in a tuple, list, or the param(...) helper:

@parameterized([ ([1, 2, 3], ), ([3, 3], ), ([6], ),])deftest_sums_to_6(numbers):
assertsum(numbers) ==6

(note, also, that Python requires single element tuples to be defined with a trailing comma: (foo, ))

Using with @mock.patch

parameterized can be used with mock.patch, but the argument ordering can be confusing. The @mock.patch(...) decorator must come below the @parameterized(...), and the mocked parameters must come last:

@mock.patch("os.getpid")classTestOS(object):
@parameterized(...)@mock.patch("os.fdopen")@mock.patch("os.umask")deftest_method(self, param1, param2, ..., mock_umask, mock_fdopen, mock_getpid):
...

Note: the same holds true when using @parameterized.expand.

Migrating from nose-parameterized to parameterized

To migrate a codebase from nose-parameterized to parameterized:

1. Update your requirements file, replacing nose-parameterized with parameterized.

2. Replace all references to nose_parameterized with parameterized:

   $ perl -pi -e 's/nose_parameterized/parameterized/g' your-codebase/
   

3. You're done!

FAQ

What happened to Python 2.X, 3.5, and 3.6 support?

As of version 0.9.0, parameterized no longer supports Python 2.X, 3.5, or 3.6. Previous versions of parameterized - 0.8.1 being the latest - will continue to work, but will not receive any new features or bug fixes.

What do you mean when you say "nose is best supported"?

There are small caveates with py.test and unittest: py.test does not show the parameter values (ex, it will show test_add[0] instead of test_add[1, 2, 3]), and unittest/unittest2 do not support test generators so @parameterized.expand must be used.

Why not use @pytest.mark.parametrize?

Because spelling is difficult. Also, parameterized doesn't require you to repeat argument names, and (using param) it supports optional keyword arguments.

Why do I get an AttributeError: 'function' object has no attribute 'expand' with @parameterized.expand?

You've likely installed the parametrized (note the missing e) package. Use parameterized (with the e) instead and you'll be all set.

What happened to nose-parameterized?

Originally only nose was supported. But now everything is supported, and it only made sense to change the name!