test: move heapdump tests to pummel

[Trott]

The heapdump tests take a lot more time to run than our other tests in
parallel. They are also a bit of an internal test that perhaps does not
need to be run on every commit on every platform. This change moves them
to the pummel directory where they will be run on a single platform once
a day in CI.

This shaves more than 20 seconds off make test on my laptop, FWIW.

@addaleax@Fishrock123

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• commit message follows commit guidelines

[nodejs-github-bot]

@Trott build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/2040/pipeline

[Trott]

pummel tests are not (yet) run in CI, so lite CI seems sufficient.

[addaleax]

Hm … just for context, the original motivation for these tests was increasing native code coverage. They do take 20 seconds for me as well, but only when not run in parallel, so things could be worse…

Does it make sense to maybe create a directory for tests that should be run locally only on request, and in CI always? We have a lot of tests that would qualify for that, imo, and that might be a good path to speeding up make test in general?

[Trott]

Does it make sense to maybe create a directory for tests that should be run locally only on request, and in CI always?

The temptation there would be to put all the slow tests in the CI-only directory.

My sense is that it would be better to split out unit tests from end-to-end tests. The unit tests will run fast and you can run them very frequently. The end-to-end tests will definitely run on CI but optionally locally as well. (People can choose whether they are that kind of completist or not.)

Then we can collect coverage stats for unit tests only and coverage stats for integration tests. The goal would be to get good coverage with both types of tests.

If we split out the tests based on speed only, then it's far more subjective. (What is the cut-off and on what type of machine? What of tests that are slow only on, say, Windows?) There's subjectivity with unit-test vs. integration test too, but I would expect it to be less frequent.

All that said, as resistant as I've been to the idea (e.g., @BridgeAR proposed something similar at some point, I'm pretty sure, many months ago), separating the slow tests may be a decent first step. I'm just concerned it will also be the last step...

[Trott]

Also: Arguably, we already have a place for slow tests: pummel. We just need to get it running on CI. I had said once a day, but we can have it run on every CI.

[Trott]

Lastly: I don't intend for this to be merged until pummel is running in CI (whether once a day or on every pull request). So I'll put a blocked label on this for now.

[Trott]

The pummel tests will now run nightly in node-daily-master, so removing blocked label here.

[Trott]

Lite CI: https://ci.nodejs.org/job/node-test-pull-request-lite-pipeline/2278/ ✅

Nightly job that includes pummel tests run with these changes: https://ci.nodejs.org/job/node-test-commit-custom-suites/830/ ✅

[Trott]

Landed in 87f6804.

I'll take a try at creating a make test-all or something.

[Trott]

Oh, there already is a test-all... guess I ought to see what that does....

[addaleax]

@Trott C++ coverage on https://coverage.nodejs.org/ dropped a full percent after this landed … do you think you could add pummel to the test suites we run for coverage?

[Trott]

@Trott C++ coverage on https://coverage.nodejs.org/ dropped a full percent after this landed … do you think you could add pummel to the test suites we run for coverage?

Yeah, very much something I want done. (internet and even benchmark tests too.) I'll take it up with @nodejs/build.

[Trott]

@Trott C++ coverage on https://coverage.nodejs.org/ dropped a full percent after this landed … do you think you could add pummel to the test suites we run for coverage?

Yeah, very much something I want done. (internet and even benchmark tests too.) I'll take it up with @nodejs/build.

OK, I added pummel and benchmark tests. Once #25799 lands, we can change it to run test-all-suites and it will always run all tests no matter where they are under the test directory.

[Trott]

(And I kicked off a job to make sure it works. https://ci.nodejs.org/view/All/job/node-test-commit-linux-coverage-daily/21/)

[Trott]

Adding pummel and benchmark tests increased C++ coverage back to 91%. Oddly, JS coverage is down to 92.75% which seems wrong unless some big chunk of largely-untested code landed today, which I don't think happened.... Have to look into that...