Support the pedantic API

[art049]

Fixes#77

We'll merge both after your PR @not-matthias

[adriencaccia]

Nice!

It would make sense to test the benchmark.pedantic function in tests/test_pytest_plugin_walltime.py

[art049]

Yep I'll add a nominal test for both!

[codspeed-hq]

CodSpeed Instrumentation Performance Report

Merging #81 will not alter performance

_{Comparing cod-955-allow-stateful-benchmark-replaying-setup-teardown-with (96fe457) with master (22d4d33)}

Summary

✅ 57 untouched benchmarks

[codspeed-hq]

CodSpeed Walltime Performance Report

Merging #81 will degrade performances by 7.67%

_{Comparing cod-955-allow-stateful-benchmark-replaying-setup-teardown-with (96fe457) with master (22d4d33)}

Summary

⚡ 7 improvements
❌ 21 regressions
✅ 29 untouched benchmarks

⚠️Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	test_iir_filter_process	3.1 µs	3 µs	+2.63%
❌	test_iir_filter_set_coefficients[a_coeffs0-b_coeffs0]	842.8 ns	881.7 ns	-4.41%
❌	test_make_allpass	5.1 µs	5.3 µs	-3.62%
❌	test_make_bandpass	5.3 µs	5.6 µs	-4.13%
❌	test_make_highpass	5.4 µs	5.7 µs	-5.44%
❌	test_make_lowpass	5.3 µs	5.6 µs	-5.03%
⚡	test_combination_lists[0-0]	1.9 µs	1.9 µs	+2.34%
❌	test_combination_sum[candidates0-8]	10.7 µs	11 µs	-3.04%
❌	test_depth_first_search[4]	30.3 µs	30.9 µs	-2.07%
❌	test_generate_all_combinations[0-0]	1.3 µs	1.3 µs	-2.49%
❌	test_generate_all_combinations[4-2]	7 µs	7.3 µs	-3.77%
❌	test_generate_all_combinations[5-4]	9.9 µs	10.2 µs	-3.56%
❌	test_generate_all_permutations[sequence0]	158.2 µs	161.7 µs	-2.15%
⚡	test_generate_sum_of_subsets_soln[nums0-9]	25.5 µs	24.3 µs	+4.79%
❌	test_get_valid_pos[pos0-4]	2.8 µs	2.9 µs	-3.26%
⚡	test_hamilton_cycle[graph0]	28.2 µs	27.6 µs	+2.41%
⚡	test_match_word_pattern[aba-GraphTreesGraph]	104.8 µs	101.1 µs	+3.61%
❌	test_minimax[scores0]	10.4 µs	10.8 µs	-4.37%
❌	test_solve_maze[maze0]	22.8 µs	23.9 µs	-4.56%
❌	test_solve_power_sum[13-2]	6.9 µs	7 µs	-2.71%
...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

[flying-sheep]

Please revert the typing changes. E.g. args: tuple is just wrong.

[flying-sheep]

this used to be correct, now it makes no sense at all.

what you changed it to is “each positional argument is a tuple, and each keyword argument is a dict”. So this is valid according to your typing:

benchmark(print, (), (1,2), file={})

but this is invalid according to your typing:

benchmark(print, "a", file="out.txt")