Shrinker improvements

[jmid]

This PR combines QCheck(1) shrinker improvements to list, string, functions, and int.
The improvements arose from studying the benchmark outputs from #177 across a range of equivalent tests.
Here it became clear that both the QCheck(1) and QCheck2 shrinkers each had some sore spots.
As a first step, I set out to get QCheck(1) up to speed with QCheck2's performance on the tests where its shrinkers were lacking behind.

I don't expect to merge the PR in the current form, partly because it includes an int shrinker along the lines of the somewhat controversial #173, but

• I wanted to convey the bigger picture
• I was curious to hear feedback from potential users.

@vch9 : if you (or others) still have QCheck(1) tests (needing shrinking) I'd be curious to hear if you observe a difference
when pinning https://github.com/jmid/qcheck/tree/shrinker-improvements

Whereas these improvements have been obtained over artificial benchmark programs, I've been bitten in the past by shrinker performance in bigger developments, e.g.,

• in the shrinker for our Wasm generator where it made a difference to reduce integers to zero quickly (in the common case most integer values are irrelevant).
• in the shrinker for multicoretests, where the non-deterministic behavior is extra challenging. Here reproducing a failing run O(n log n) times has a much lower probability than O(log n) times, which means that a algorithmically faster shrinker performing fewer, well-chosen reduction attempts will produce smaller counterexamples than a "more exhaustive" one.

The core is an improved list_spine shrinker. As an added bonus this implicitly improves function shrinking.
Secondly, we can reuse the improved list shrinker for strings - by breaking them up in char lists - and then use the char-shrinker to reduce the individual chars. Overall, these three were identified as QCheck(1) sore spots in #177.

Despite the reduced complexity we are able to arrive at the same counterexamples - or simpler(!) as should
be clear from the expect-test logs.

Here's the new and simpler shrinker of list spines using recursion:

letrec list_spinelyield=letrec splitllenacc=match len,l with| _,[]|0,_ -> List.rev acc, l
|_,x::xs -> split xs (len-1) (x::acc) inmatch l with|[] -> ()| [_] -> yield []| [x;y] -> yield []; yield [x]; yield [y]
|_::_ ->
let len =List.length l inlet xs,ys = split l ((1+ len) /2) []in
yield xs;
list_spine xs (funxs' -> yield (xs'@ys))

The base cases with 0,1, or 2 elements are not surprising.
In the inductive case, the yield xs represents dropping the last half of the list (akin to bisection). The yield from the recursive call in the last line combines a reduced front half with an untouched second half ys.

So how well does this fare? It turns out to work reasonably well. Here's an example:

utop # Shrink.list_spine [1;2;3;4;5;6;7;8] (funxs -> Printf.printf "%s\n"Print.(listint xs));;
[1; 2; 3; 4]
[1; 2; 5; 6; 7; 8]
[3; 4; 5; 6; 7; 8]
[1; 3; 4; 5; 6; 7; 8]
[2; 3; 4; 5; 6; 7; 8]

As an added bonus, there are both children containing and excluding the list head. This is useful, since sometimes it is central to keep it in a counterexample and sometimes it is not.

The list of children does not grow much longer on an input list twice as long:

utop # Shrink.list_spine [1;2;3;4;5;6;7;8;9;10;11;12;13;14;15;16] (funxs -> Printf.printf "%s\n"Print.(listint xs));;
[1; 2; 3; 4; 5; 6; 7; 8]
[1; 2; 3; 4; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
- : unit=()

In comparison, here's the output from the same lines run with the current list_spine shrinker:

#Shrink.list_spine [1;2;3;4;5;6;7;8] (funxs -> Printf.printf "%s\n"Print.(listint xs));;
[5; 6; 7; 8]
[1; 6; 7; 8]
[1; 2; 7; 8]
[1; 2; 3; 8]
[1; 2; 3; 4]
[3; 4; 5; 6; 7; 8]
[1; 4; 5; 6; 7; 8]
[1; 2; 5; 6; 7; 8]
[1; 2; 3; 6; 7; 8]
[1; 2; 3; 4; 7; 8]
[1; 2; 3; 4; 5; 8]
[1; 2; 3; 4; 5; 6]
[2; 3; 4; 5; 6; 7; 8]
[1; 3; 4; 5; 6; 7; 8]
[1; 2; 4; 5; 6; 7; 8]
[1; 2; 3; 5; 6; 7; 8]
[1; 2; 3; 4; 6; 7; 8]
[1; 2; 3; 4; 5; 7; 8]
[1; 2; 3; 4; 5; 6; 8]
[1; 2; 3; 4; 5; 6; 7]
- : unit=()#Shrink.list_spine [1;2;3;4;5;6;7;8;9;10;11;12;13;14;15;16] (funxs -> Printf.printf "%s\n"Print.(listint xs));;
[9; 10; 11; 12; 13; 14; 15; 16]
[1; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 16]
[1; 2; 3; 4; 5; 6; 7; 8]
[5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12]
[3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14]
[2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 7; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 8; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 9; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 10; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 11; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 12; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 13; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 14; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 15; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 16]
[1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15]
- : unit=()

Because of the repeated chunk reduction (each O(log n) chunk size is attempted at a sequence of positions) the complexity of the current list_spine is something like O(n log n), whereas the new one is O(log n).
This is just for one run though. After a successful reduction, each shrinker is restarted.

[jmid]

Below follows the output from a rewamped shrinker performance benchmark run of #177.

In total QCheck goes from 67.071s to 8.064s - not a bad improvement.
This is primarily achieved by the improvements to the list and function shrinkers, leaving only one benchmark (fold_left test, fun first) to take more than 1 second. The latter is further improved by combining the present PR with #240.

 iteration seed 1234 iteration seed 8743 iteration seed 6789 total
Shrink test name Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
big bound issue59 - skipped as generator is stateful, making it non-repeatable
long_shrink 0.006 89/292 0.922 3039/3099 0.002 84/276 0.615 3068/3127 0.001 82/268 0.675 3063/3124 0.009 2.212
ints arent 0 mod 3 0.000 1/1 0.000 2/2 0.000 1/1 0.000 1/1 0.000 1/1 0.000 88/305 0.000 0.000
ints are 0 0.000 61/123 0.000 61/123 0.000 60/121 0.000 61/122 0.000 61/123 0.000 61/123 0.000 0.000
ints < 209609 - skipped as generator is stateful, making it non-repeatable
nat < 5001 0.000 6/66 0.000 7/77 0.000 6/57 0.000 7/69 0.000 7/73 0.000 8/85 0.000 0.000
char never produces 'abcdef' 0.000 0/0 0.000 1/1 0.000 0/0 0.000 0/0 0.000 0/0 0.000 0/0 0.000 0.000
strings are empty 0.000 145/284 0.000 8/16 0.000 169/327 0.001 13/26 0.000 49/99 0.000 1/2 0.000 0.001
string never has a \000 char 0.000 8/16 0.002 22/167 0.001 17/28 0.002 56/254 0.000 6/12 0.000 15/48 0.001 0.004
string never has a \255 char 0.000 14/28 0.001 59/318 0.000 15/24 0.003 97/529 0.001 21/38 0.003 41/194 0.001 0.006
strings have unique chars 0.000 13/31 0.000 18/30 0.002 15/30 0.002 24/52 0.000 5/14 0.000 15/20 0.002 0.002
pairs have different components 0.000 0/6 0.000 0/6 0.000 0/6 0.000 0/6 0.000 0/10 0.000 0/10 0.000 0.000
pairs have same components 0.000 62/124 0.000 63/125 0.000 61/122 0.000 62/123 0.000 62/124 0.000 63/125 0.000 0.000
pairs have a zero component 0.000 122/307 0.000 122/306 0.000 121/304 0.000 122/306 0.000 116/295 0.002 123/308 0.000 0.002
pairs are (0,0) 0.000 62/124 0.000 63/125 0.000 61/122 0.000 62/123 0.000 62/124 0.000 63/125 0.000 0.000
pairs are ordered 0.000 93/1162 0.000 94/1217 0.000 87/985 0.000 85/865 0.000 91/1149 0.000 94/1326 0.000 0.000
pairs are ordered reversely 0.000 62/124 0.000 62/124 0.000 62/124 0.000 62/124 0.000 62/124 0.000 62/124 0.000 0.000
pairs sum to less than 128 0.000 56/126 0.000 56/126 0.000 58/131 0.000 59/138 0.000 57/131 0.000 57/130 0.000 0.000
pairs lists rev concat 0.002 84/283 0.010 83/168 0.001 78/280 0.005 75/152 0.000 68/253 0.000 67/136 0.003 0.015
pairs lists no overlap 0.000 24/44 0.003 27/60 0.000 17/37 0.002 18/41 0.000 8/22 0.000 11/28 0.001 0.004
triples have pair-wise different components 0.000 3/15 0.000 3/15 0.000 3/3 0.000 3/3 0.000 3/3 0.000 3/3 0.000 0.000
triples have same components 0.000 63/126 0.000 64/127 0.000 63/126 0.000 64/128 0.000 57/114 0.000 62/122 0.000 0.000
triples are ordered 0.000 63/126 0.000 3/4 0.000 62/123 0.000 3/4 0.000 63/126 0.000 91/1021 0.000 0.000
triples are ordered reversely 0.000 63/125 0.000 64/126 0.000 63/126 0.000 124/247 0.000 63/125 0.000 65/127 0.000 0.000
quadruples have pair-wise different components 0.000 4/4 0.000 4/4 0.000 4/4 0.000 4/4 0.000 4/11 0.000 4/11 0.000 0.000
quadruples have same components 0.000 124/310 0.000 126/313 0.000 113/287 0.000 115/292 0.000 124/310 0.000 123/307 0.000 0.000
quadruples are ordered 0.000 64/127 0.000 5/6 0.000 63/124 0.000 4/5 0.000 58/115 0.000 5/6 0.000 0.000
quadruples are ordered reversely 0.000 64/126 0.000 66/128 0.000 64/127 0.000 126/250 0.000 64/126 0.000 66/128 0.000 0.000
forall (a, b) in nat: a < b 0.000 6/16 0.000 6/16 0.000 6/15 0.000 6/15 0.000 4/7 0.000 4/7 0.000 0.000
forall (a, b, c) in nat: a < b < c 0.000 3/7 0.000 3/7 0.000 6/22 0.000 7/28 0.000 3/3 0.000 3/3 0.000 0.000
forall (a, b, c, d) in nat: a < b < c < d 0.000 4/4 0.000 4/4 0.000 4/4 0.000 4/4 0.000 4/4 0.000 4/4 0.000 0.000
forall (a, b, c, d, e) in nat: a < b < c < d < e 0.000 5/5 0.000 5/5 0.000 5/5 0.000 5/5 0.000 5/5 0.000 5/5 0.000 0.000
forall (a, b, c, d, e, f) in nat: a < b < c < d 0.000 6/6 0.000 6/6 0.000 6/6 0.000 6/6 0.000 6/6 0.000 6/6 0.000 0.000
forall (a, b, c, d, e, f, g) in nat: a < b < c < 0.000 7/7 0.000 7/7 0.000 7/7 0.000 7/7 0.000 7/7 0.000 7/7 0.000 0.000
forall (a, b, c, d, e, f, g, h) in nat: a < b < 0.000 8/8 0.000 8/8 0.000 8/8 0.000 8/8 0.000 7/7 0.000 7/7 0.000 0.000
forall (a, b, c, d, e, f, g, h, i) in nat: a < b 0.000 9/9 0.000 9/9 0.000 9/9 0.000 9/9 0.000 8/8 0.000 8/8 0.000 0.000
bind ordered pairs 0.000 2/2 0.000 1/1 0.000 2/2 0.000 1/1 0.000 2/2 0.000 1/1 0.000 0.000
bind list_size constant 0.000 11/31 0.000 12/26 0.000 13/35 0.000 12/25 0.000 12/33 0.000 11/21 0.000 0.000
lists are empty 0.001 9/12 0.000 8/16 0.002 14/17 0.003 13/26 0.000 1/3 0.000 1/2 0.002 0.004
lists shorter than 10 0.000 16/81 0.000 16/30 0.000 21/91 0.004 21/42 0.000 14/81 0.000 15/29 0.000 0.004
lists shorter than 432 0.102 417/4881 1.049 412/457 0.139 404/4728 0.662 405/450 0.027 407/4793 0.067 419/447 0.269 1.778
lists shorter than 4332 0.063 13/67 3.616 4022/4087 0.075 10/35 3.637 4020/4067 0.014 8/55 2.415 4013/4055 0.151 9.668
lists equal to duplication 0.152 20/23 0.420 4/7 0.000 3/6 0.000 3/6 0.011 18/21 0.122 17/35 0.163 0.542
lists have unique elems 0.000 8/18 0.000 11/22 0.004 12/25 0.008 17/30 0.000 8/22 0.000 10/20 0.004 0.008
tree contains only 42 0.000 2/2 0.000 2/2 0.000 1/1 0.000 2/2 0.000 2/2 0.000 2/2 0.000 0.000
fail_pred_map_commute 0.000 134/779 0.000 16/59 0.000 109/621 0.000 14/65 0.000 115/603 0.001 117/373 0.001 0.001
fail_pred_strings 0.000 1/4 0.000 1/4 0.000 1/4 0.000 1/4 0.000 1/3 0.000 2/5 0.000 0.000
fold_left fold_right 0.000 17/63 0.000 22/73 0.001 31/91 0.006 58/139 0.000 22/80 0.000 39/95 0.001 0.006
fold_left fold_right uncurried 0.002 31/136 0.043 325/984 0.022 34/86 9.235 4811/8969 0.000 3/17 0.000 2/13 0.025 9.279
fold_left fold_right uncurried fun last 0.000 12/42 0.000 25/86 0.001 30/86 0.002 54/176 0.000 22/81 0.001 40/93 0.001 0.003
fold_left test, fun first 0.017 272/534 0.001 15/28 0.760 165/506 4.041 45/11947 6.652 548/20645 1.074 168/27748 7.430 5.115
8.064 28.657

[jmid]

OK, using the merged shrinker performance benchmark I've now cherry-picked each of the four shrinker improvements and measured each of them.

• 5a67e37 - simpler list shrinker with better complexity
• d731af8 - use improved list shrinker for strings
• 43b3962 - improved function shrinker
• c19ccd0 - improved int shrinker

Executive summary:

• the improved list, string, and function shrinkers collectively reduce the runtime of the benchmark from 78.402s to 5.456s on my laptop
• adding the int shrinker improvement on top reduces the runtime from an average of 5.448s to 5.3289s across 10 runs

Whereas the latter represents a statistical significant improvement (measured with ministat) it is not as drastic as I thought before measuring.
The duplicate 0 testing of the improved int shrinker is unsatisfying, but so is the duplicate output of the current int shrinker:

qcheck/test/core/QCheck_unit_tests.ml

Lines 20 to 24 in 824dafb

	lettest_int()=
	List.iter (alco_check Alcotest.int (trace_false Shrink.int) "on repeated failure")
	[ ("int 100", 100, [50; 75; 88; 94; 97; 99; 99]); (*WTF?*)
	("int 1000", 1000, [500; 750; 875; 938; 969; 985; 993; 997; 999; 999]); (*WTF?*)
	("int (-26)", -26, [-13; -20; -23; -25; -25]) ]; (*WTF?*)

We should address this separately. I therefore move to merge the first three commits.

Measurement details

To keep the tables focused below I've deleted benchmark entries with a total QCheck1 shrinker time below 0.1 seconds.

Here's first the performance of current master. Note how the focused tables highlight string, list, and function benchmarks.

 iteration seed 1234 iteration seed 8743 iteration seed 6789 total
Shrink test name Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
string never has a \255 char 0.001 249/316 0.001 59/318 0.092 4466/4520 0.002 97/529 0.377 9260/9365 0.002 41/194 0.469 0.005
strings have unique chars 0.003 248/269 0.000 18/30 0.919 4465/4536 0.002 24/52 0.000 14/34 0.000 15/20 0.922 0.002
lists shorter than 432 6.343 1696/5118102 1.049 412/457 6.108 1612/4863421 0.981 405/450 6.087 1667/5037661 0.126 419/447 18.537 2.156
lists shorter than 4332 2.172 13/190735 3.331 4022/4087 1.455 11/126052 3.580 4020/4067 1.410 7/126607 2.408 4013/4055 5.037 9.318
lists equal to duplication 0.145 20/23 0.470 4/7 0.000 7/13 0.000 3/6 0.021 20/25 0.091 17/35 0.165 0.561
fold_left fold_right uncurried 2.496 97/80630 0.044 376/1550 0.144 38/390 2.160 2064/8057 0.000 5/20 0.000 4/17 2.640 2.204
fold_left test, fun first 0.001 40/57 0.001 15/28 40.082 191/44563 3.318 47/9773 10.359 223/75912 0.002 36/64 50.442 3.321
78.402 19.754

Step 1 Here's the performance of the simpler list shrinker. Note how the two lists shorter than benchmarks now run much faster.
On the other hand, the change causes fold_left test, fun first to really slow down because of the 93854 and 524691 shrink attempts with seed 8743 and 6789, respectively.
This latter test combines both lists, strings, and functions.

 iteration seed 1234 iteration seed 8743 iteration seed 6789 total
Shrink test name Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
string never has a \255 char 0.001 249/316 0.001 59/318 0.100 4466/4520 0.003 97/529 0.413 9260/9365 0.002 41/194 0.514 0.005
strings have unique chars 0.003 248/269 0.000 18/30 0.917 4465/4536 0.001 24/52 0.000 14/34 0.000 15/20 0.920 0.002
lists shorter than 432 0.161 1632/19461 0.903 412/457 0.194 1677/20004 1.016 405/450 0.076 1735/20729 0.112 419/447 0.431 2.032
lists shorter than 4332 0.085 13/67 3.474 4022/4087 0.128 10/35 3.538 4020/4067 0.009 8/55 2.529 4013/4055 0.222 9.541
lists equal to duplication 0.157 26/35 0.419 4/7 0.000 6/12 0.000 3/6 0.009 25/35 0.044 17/35 0.166 0.463
fold_left fold_right uncurried 0.003 25/125 0.046 376/1550 0.023 34/119 2.255 2064/8057 0.000 5/20 0.000 4/17 0.025 2.301
fold_left test, fun first 0.002 41/60 0.001 15/28 1488.644 1230/93854 3.423 47/9773 2123.311 2759/524691 0.002 36/64 3611.956 3.426
3614.426 19.923

Step 2 Here's the performance of using the improved list shrinker for strings. This reduces the runtime of the two string tests to just 0.001s. Note how for strings have unique chars with seed 8743 this reduces 4465 succesful shrink attempts out of 4536 down to just 15 out of 30.

The string improvement also helps a bit on fold_left test, fun first which is now down to 67879 and 124382 shrink attempts with seed 8743 and 6789, respectively.

 iteration seed 1234 iteration seed 8743 iteration seed 6789 total
Shrink test name Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
string never has a \255 char 0.000 14/28 0.001 59/318 0.001 15/24 0.005 97/529 0.001 21/38 0.003 41/194 0.001 0.008
strings have unique chars 0.000 13/31 0.000 18/30 0.001 15/30 0.003 24/52 0.000 5/14 0.000 15/20 0.001 0.003
lists shorter than 432 0.158 1632/19461 0.893 412/457 0.200 1677/20004 0.934 405/450 0.069 1735/20729 0.062 419/447 0.427 1.889
lists shorter than 4332 0.140 13/67 3.365 4022/4087 0.116 10/35 3.214 4020/4067 0.018 8/55 2.391 4013/4055 0.273 8.970
lists equal to duplication 0.160 26/35 0.382 4/7 0.000 6/12 0.000 3/6 0.009 25/35 0.103 17/35 0.169 0.485
fold_left fold_right uncurried 0.003 25/125 0.046 376/1550 0.023 34/119 2.162 2064/8057 0.000 5/20 0.000 4/17 0.026 2.209
fold_left test, fun first 0.017 275/543 0.001 15/28 669.151 1006/67879 3.552 47/9773 110.429 726/124382 0.002 36/64 779.596 3.555
780.559 19.211

Step 3 Here's the performance of adding the improved function shrinker on top.
This last one restores decent performance for fold_left test, fun first which is now down to 1551 and 1483 shrink attempts with seed 8743 and 6789, respectively.

 iteration seed 1234 iteration seed 8743 iteration seed 6789 total
Shrink test name Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
string never has a \255 char 0.000 14/28 0.001 59/318 0.000 15/24 0.005 97/529 0.001 21/38 0.003 41/194 0.002 0.008
strings have unique chars 0.000 13/31 0.000 18/30 0.001 15/30 0.003 24/52 0.000 5/14 0.000 15/20 0.002 0.003
lists shorter than 432 0.160 1632/19461 0.904 412/457 0.197 1677/20004 0.909 405/450 0.067 1735/20729 0.060 419/447 0.425 1.873
lists shorter than 4332 0.135 13/67 3.534 4022/4087 0.116 10/35 3.196 4020/4067 0.020 8/55 2.364 4013/4055 0.270 9.094
lists equal to duplication 0.156 26/35 0.381 4/7 0.000 6/12 0.000 3/6 0.009 25/35 0.103 17/35 0.165 0.483
fold_left fold_right uncurried 0.002 44/199 0.044 376/1550 0.032 55/170 2.106 2064/8057 0.000 5/15 0.000 4/17 0.034 2.151
fold_left test, fun first 0.016 275/543 0.000 15/28 4.206 305/1551 3.254 47/9773 0.272 296/1483 0.002 36/64 4.494 3.257
5.456 18.971

Step 4 Here's the performance of adding the improved int shrinker on top.
Note how lists shorter than 432 has its total shrink attempt reduced from 19461, 20004, and 20729 down to 4881, 4728, 4793 total attempts.

 iteration seed 1234 iteration seed 8743 iteration seed 6789 total
Shrink test name Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s #succ/#att Q2/s #succ/#att Q1/s Q2/s
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
string never has a \255 char 0.000 14/28 0.001 59/318 0.000 15/24 0.003 97/529 0.001 21/38 0.002 41/194 0.002 0.005
strings have unique chars 0.000 13/31 0.000 18/30 0.001 15/30 0.002 24/52 0.000 5/14 0.000 15/20 0.001 0.002
lists shorter than 432 0.107 417/4881 0.831 412/457 0.143 404/4728 0.871 405/450 0.021 407/4793 0.060 419/447 0.271 1.762
lists shorter than 4332 0.133 13/67 3.335 4022/4087 0.083 10/35 3.583 4020/4067 0.010 8/55 2.265 4013/4055 0.226 9.183
lists equal to duplication 0.166 20/23 0.385 4/7 0.000 3/6 0.000 3/6 0.010 18/21 0.101 17/35 0.176 0.487
fold_left fold_right uncurried 0.002 39/233 0.043 376/1550 0.026 49/161 2.119 2064/8057 0.000 2/14 0.000 4/17 0.028 2.163
fold_left test, fun first 0.017 272/534 0.001 15/28 4.225 305/1558 3.208 47/9773 0.274 288/1434 0.002 36/64 4.516 3.211
5.241 18.957

Statistical test For reference here's the total timings of 10 runs of step 3:

5.456
5.508
5.441
5.442
5.406
5.414
5.417
5.454
5.435
5.507

and 10 runs of step 4:

5.241
5.232
5.269
5.240
5.376
5.289
5.405
5.413
5.416
5.408

and the output of ministat:

x shrink_bench.log-stepwise-step3-improved-function-shrinker-timings
+ shrink_bench.log-stepwise-step4-improved-int-shrinker-timings
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|+ ++ + + + * + * * x x xx xx|
| |___________________________________________________A______________________________M_____________________||__________________M___A______________________| |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 10 5.406 5.508 5.442 5.448 0.035458896
+ 10 5.232 5.416 5.376 5.3289 0.081052041
Difference at 95.0% confidence
-0.1191 +/- 0.0587783
-2.18612% +/- 1.0789%
(Student's t, pooled s = 0.062557)

[jmid]

I will rebase this PR shortly. For reference, here's the considered int shrinker replacement:

(* inspired by QCheck2's int shrinker algorithm (non-exhaustive) *)letintxyield=let curr =ref0in(*to return 0 repeatedly *)(*was: let curr = ref (x/2) *)(* try some divisors *)while!curr <> x do
yield !curr;
let half_diff = (x -!curr)/2in(*was: let half_diff = (x/2) - (!curr/2) in *)if half_diff =0then curr := x
else curr :=!curr + half_diff
done

Edit: To avoid too much forcing and overwriting I'll submit a separate PR instead.

[vch9]

Unrelated to your merge request but what is yield? As x is continuously shrinking I guess it it's in charge of both marking the numbers of steps but also stop when the shrinker found the minimal value?

[vch9]

The int shrinker improvement makes sense imo. Starting at 0 should help in the vast majority of cases, it also fix the case where the last shrinked element is repeated. LGTM

[jmid]

Closing as the 3 dominant improvements have been included as part of #242.
Note: during review PR #242 was further extended with an improved char shrinker, which brought the QCheck(1) shrinker benchmark time down to 2 secs.