Test runner run docs example will always exit with exit code 0

[Filipoliko]

Affected URL(s)

https://nodejs.org/api/test.html#runoptions

Description of the problem

Implementing run function from node:test module as described in docs will lead to incorrect exit code handling for failing tests.

Example from doc:

// runner.mjs
import { run } from 'node:test';
import path from 'node:path';
import { tap } from 'node:test/reporters';
import process from 'node:process';

run({ files: [path.resolve('./tests/test.mjs')] })
 .compose(tap)
 .pipe(process.stdout);

// ./tests/test.mjs
import { test } from 'node:test';

test('test', () => {
 throw 'error';
})

Running node runner.mjs will lead to following output.

TAP version 13
# Subtest: test
not ok 1 - test
 ---
 duration_ms: 0.614625
 failureType: 'testCodeFailure'
 error: 'error'
 code: 'ERR_TEST_FAILURE'
 ...
1..1
# tests 1
# suites 0
# pass 0
# fail 1
# cancelled 0
# skipped 0
# todo 0
# duration_ms 56.791208

with exit code 0, but since the test failed, it should be 1.

Running node --test --test-reporter=tap ./tests/test.mjs leads to same output, but exit code 1 as expected.

I am not sure if this is a docs issue, or an actual bug, but I wasn't able to figure out, how to handle exit code when implementing test runner programmatically. If there is a way, could somebody please elaborate on how to do this? And maybe even update the documentation in case somebody else runs into the same issue?

Thanks in advance for any answers :)

MacOS 13.4.1
Node.js 20.4.0

[MoLow]

the run function exposed by node:test is no meant to be used with --test. although it can be used with it, it is meant to replace it - so I would say this is an expected behavior

[Filipoliko]

@MoLow Thank you for the quick reply! I am actually executing the runner.mjs script without the --test parameter. The script with --test was meant to be an example comparing how it behaves differently to run tests via Node CLI compared to the programmatic way using run function exposed by node:test. The problem is, that I could not figure out, how to exit the runner.mjs script with exit code 1 if test fails, or 0 if it passed. If I am running my tests programmatically with node runner.mjs, I always get exit code 0 even if tests fail, using the example from docs. I tried listening for some end/close/finish event, but there seems to be none. This might be just my low understanding of how to handle the end of TestsStream, but I could not find any information in Node.js docs about how to do this.

[MoLow]

you can listen to the TestStream test:fail event

// runner.mjs
import { run } from 'node:test';
import path from 'node:path';
import { tap } from 'node:test/reporters';
import process from 'node:process';

const testsStream = run({ files: [path.resolve('./tests/test.mjs')] });

testsStream.compose(tap).pipe(process.stdout);
testsStream.once('test:fail', () => {
 process.exitCode = 1;
});

[Filipoliko]

This works, thank you very much for the example. Not sure if you want to mention something like this in the docs, but this solves my problem so feel free to close the issue.

[atlowChemi]

@MoLow Won't this exit on first test failure?

[Filipoliko]

@atlowChemi It does not. It just configures the exitCode value for the current process, but does not actually exit.