Flaky test-benchmark-path

[BridgeAR]

Seems like this test some times times out.

https://ci.nodejs.org/job/node-test-binary-arm/12927/RUN_SUBSET=6,label=pi3-raspbian-jessie/console

20:28:04 not ok 15 parallel/test-benchmark-path
20:28:04 ---
20:28:04 duration_ms: 240.61
20:28:04 severity: fail
20:28:04 stack: |-
20:28:04 timeout

[joyeecheung]

Looks more like a bug within the benchmark tools

[apapirovski]

I think this one got resolved at some point? At least that's my recollection.

[gireeshpunathil]

started to see this again, for example: https://ci.nodejs.org/job/node-test-commit-aix/nodes=aix61-ppc64/15848/consoleFull

I will see if I can recreate locally.

[gireeshpunathil]

06:32:20 not ok 235 parallel/test-benchmark-path
06:32:20 ---
06:32:20 duration_ms: 120.191
06:32:20 severity: fail
06:32:20 stack: |-
06:32:20 timeout

[Trott]

Failing on AIX in parallel with a timeout suggests that perhaps it should be moved to sequential.

I see on successful runs that it still can take 25 seconds or so, and that's a lot for a test in parallel. Get the machine busy doing something else and have this test compete with just the wrong other tests in parallel, and it times out. Or at least, that would be the first theory.

So I'd say move this to sequential. Or check to see if there's a way to reduce the time it takes to run this test. But I did that and I don't think there's anything to do there unless we want to combine benchmarks into fewer files. It seems like launching processes is where the time is taken up.

PR to move this test to sequential: #21393

[gireeshpunathil]

Agree that it is a manifestation of parallelism, not much to do with the test as such. Running independantly, it comes out in ~9 seconds.

Do we know what are the tunables which decide the number of tests that run in parallel? I am just wondering if there is a mismatch in those tunables in relation to the resource limits in the failing AIX box: for example, if the number of tests are derived as a function of the number of CPUs but the amount of usable memory / user processes etc. are otherwise constrained through ulimit settings, the process can be seen as timing out? /cc @mhdawson

[gireeshpunathil]

On a related note: what is the rationale behind running benchmarks in parallel? what inference one can derive, and what purpose they solve?