Create a stress test that more deterministically reproduces #20067, #25026 and #15014.

[juj]

Create a stress test that more deterministically reproduces flaky test errors.

Ran the test #20067 20 times, with the following results:

Failed in 5.167s
Failed in 2.002s
Failed in 2.324s
Failed in 2.564s
Passed (no repro), took 12.276s
Failed in 3.277s
Passed (no repro), took 12.955s
Failed in 5.717s
Failed in 2.092s
Failed in 3.836s
Failed in 2.001s
Failed in 3.475s
Failed in 1.601s
Failed in 5.307s
Failed in 1.757s
Failed in 4.716s
Failed in 2.036s
Failed in 3.001s
Failed in 3.600s
Failed in 7.481s

giving a repro rate of ~90% on that test.

[sbc100]

This seems like maybe useful command to be able to run on a JS file in general.

Imagine I have out/test/foo.js from the last test I ran. I'd quite like to be able to run this type of stress test by doing something like ./test/stress_test.py node out/test/foo.js. This could potentially run the test forever until it fails on as many cores as you like.

I myself have often done a poor mans version of this using counter=0 while node out/test/foojs; echo $count; ((counter++)); done.. this will run the test forever printing how many iterations it got to.

I also find the using cpu pinning can help reproduce a lot of these failure. @brendandahl recently used this, can you remind the command for it? The downside is that its linux-only (at least the command we use is).

It also might be nice to be able to run any given test in this stress test mode instead of these hardcoded 3 tests.

[brendandahl]

taskset -c 0-1

[brendandahl]

To reproduce and run the tests a ton of times without compiling. I've been doing this:

1. run the test once with --save-dir to build the test
2. Comment out the compile step, and run the test again with --save-dir and --repeat X.

I've considered adding a --compile-once flag or something, but haven't gotten around to it.

[juj]

What I do is run the failing test once (where it typically passes), so that out/test/failing_test.js contains the test that breaks.

Then I write a stress.py script:

import subprocess
import threading
import multiprocessing
import sys

COMMAND = ['node', 'out/test/failing_test.js']
stop_flag = threading.Event()

def worker():
    while not stop_flag.is_set():
        try:
            result = subprocess.run(COMMAND, capture_output=True, timeout=60)
            output = result.stdout.decode(errors="ignore") + result.stderr.decode(errors="ignore")
            if result.returncode != 0 or 'done.' not in output:
                stop_flag.set()
                print(f"Command failed with exit code {result.returncode}. Output: {output}")
        except Exception as e:
            print(f"Error running command: {e}")
            stop_flag.set()

def main():
    threads = []
    for _ in range(multiprocessing.cpu_count()):
        t = threading.Thread(target=worker, daemon=True)
        t.start()
        threads.append(t)
    for t in threads:
        t.join()

if __name__ == "__main__":
    main()

and run that until it stops. (adapting the expectation to the test case particulars)

This PR is though not about manually running, but improving the CI coverage to deterministically catch the errors.

[sbc100]

Are you saying that we run to have some tests that will always run as "stress tests", in that they run N times and in parallel? How will we decide which tests we want to run in the mode?

If so, maybe it would be better to use a decorator here?

i.e. @run_as_stress_test, or @with_stress_test rather than duplicating the existing tests.

Then it would be easy to add these decorators to arbitrary tests.

Either way I think this it would be good to create a separate test/stress_test.py that can be run either manually, or from the test code. Then we can all share best preactices and recommendations (e.g. the option to use taskset) by collaborating on this file.

[juj]

How will we decide which tests we want to run in the mode?

I think a good process would be to take tests that have eluded the group for a long time that they got flaky, and after dissecting the flaky, it turns out to be a design/programming flaw (and not a flaw in the test itself), so that when we land a fix, it needs to have diligent test to ensure it won't go back into the "oh just a flake" box again.

If so, maybe it would be better to use a decorator here?

I considered that, but I don't know how to achieve this. For a stress test it is extremely critical for the stress test code part to be as light as possible, preferably optimized to be a minimal repro. I.e. repeatedly parallelizing the test from the outside (the whole python test) is disappointing, since it will result in a lot of overhead that is not related to the actual race.

The best stress test case is one that has minimal amount of code left to run that is still exercising the race. That way CI resources will then be saved and the repro % occurrence greatly increased.

[sbc100]

Yes, it would not be a decorator that produces a parameterized test, but rather one that causes the inner-most js+check loop to be run in a loop.

[sbc100]

The decorator could take parameters too so you could control how much iterations of inner loop you want, and/or if you want to pin to a single CPU (which also seems to help repro these issues).

[brendandahl]

Another potentially interesting thing we could do with this is verify new tests. Back when I worked on firefox, we had a CI runner that basically did a more intensive repeat of any new tests that were added (or modified). We'd need some way to identify new tests though in CI.

[juj]

Another potentially interesting thing we could do with this is verify new tests. Back when I worked on firefox, we had a CI runner that basically did a more intensive repeat of any new tests that were added (or modified). We'd need some way to identify new tests though in CI.

Do you recall how Firefox identified new tests?

[brendandahl]

https://firefox-source-docs.mozilla.org/testing/test-verification/index.html#test-verification-in-automation