Uh oh!
There was an error while loading. Please reload this page.
test_runner: add level-based diagnostic handling for reporter - #55964
test_runner: add level-based diagnostic handling for reporter#55964hpatel292-seneca wants to merge 11 commits into
Conversation
nodejs-github-bot
commented
Nov 23, 2024
Review requested:
|
hpatel292-seneca
commented
Nov 23, 2024
Hi @pmarchini, Please review and let me know if you want me to change anything. |
pmarchini
commented
Nov 23, 2024
Hey @hpatel292-seneca, I won't be able to review until Monday. I've also requested other reviews in the meantime. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@## main #55964 +/- ##
==========================================
+ Coverage 87.99% 88.54% +0.54%
==========================================
Files 653 657 +4 Lines 188091 190215 +2124 Branches 35941 36529 +588 ==========================================
+ Hits 165516 168431 +2915 + Misses 15751 14976 -775 + Partials 6824 6808 -16
🚀 New features to boost your workflow:
|
cjihrig
left a comment
There was a problem hiding this comment.
Requesting changes since this already has an approval. This also needs docs and tests.
There was a problem hiding this comment.
Let's remove the debug level. The reporter stream is not a generic logger, and we have other ways (NODE_DEBUG) of adding debug output.
There was a problem hiding this comment.
I will remove that. And I have a question so this CI https://github.com/nodejs/node/actions/runs/11983534043/job/33427085217?pr=55964 failed and it's for First commit message adheres to guidelines / lint-commit-message (pull_request) so should I change commit history??
There was a problem hiding this comment.
test_runner: add level to diagnostics
There was a problem hiding this comment.
So I am asking if I should change the commit history and force push.
There was a problem hiding this comment.
Ah. Yes, please. You'll need to avoid merge commits too, as the tooling does not handle them well.
There was a problem hiding this comment.
@pmarchini Ok so I am taking reference from this tests
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.mjs
https://github.com/nodejs/node/blob/main/test/fixtures/test-runner/output/coverage-width-80-color.snapshot
and based on that I wrote this test.
fixtures/test-runner/output/spec_reporter_diagnostic_levels.mjs
// Flags: --test-reporter=specimport{test}from'node:test';import{TestsStream}from'../../../lib/internal/test_runner/tests_stream.js';process.env.FORCE_COLOR='3';consttestsStream=newTestsStream();test('Diagnostic Levels Color Output',()=>{testsStream.diagnostic(1,{},'Info-level message','info');testsStream.diagnostic(1,{},'Warning-level message','warn');testsStream.diagnostic(1,{},'Error-level message','error');});and add this in test-runner-output.mjs
{name: 'test-runner/output/spec_reporter_diagnostic_levels.mjs',transform: specTransform,tty: true,}now I am running tools/test.py test/parallel/test-runner-output.mjs --snapshot but it's not creating snap for added test.
How I can create snaps??
There was a problem hiding this comment.
I forgot you were working in a Windows environment. Considering this, I would suggest picking a different approach. You could add a test to node/test/parallel/test-runner-coverage-thresholds.js that forces the colors via the environment variable in the spawn, and check that the error message 'coverage does not meet...' is displayed in red.
Note: you need to set the reporter to spec.
This is also because test-runner-output.mjs is intended for 'e2e' testing of the test runner's output under specific circumstances.
There was a problem hiding this comment.
@pmarchini@cjihrig Thank you so much for your help. It wouldn't be possible without your help.
I ran test cases and here is output
I printed raw output in test case and here is output
and here is test case:
test(`test failing ${coverage.flag} with red color`,()=>{constresult=spawnSync(process.execPath,['--test','--experimental-test-coverage',`${coverage.flag}=99`,'--test-reporter','spec',fixture,],{env: { ...process.env,FORCE_COLOR: '3'},});conststdout=result.stdout.toString();constredColorRegex=/\u001b\[31mℹError:\d{2}\.\d{2}%\w+coveragedoesnotmeetthresholdof99%/;assert.match(stdout,redColorRegex,'Expected red color code not found in diagnostic message');assert.strictEqual(result.status,1);assert(!findCoverageFileForPid(result.pid));});If it looks good to you I can commit it and you can review the whole PR.
There was a problem hiding this comment.
Hi @pmarchini@cjihrig@jasnell@MoLow
Could we please land this PR if no change is required??
Added a parameter to allow severity-based formatting for diagnostic messages. Defaults to 'info'. This update enables better control over message presentation (e.g., coloring) based on severity levels such as 'info', 'warn', and 'error'. Refs: nodejs#55922
Updated to process the parameter for events. Messages are now formatted with colors based on the (e.g., 'info', 'warn', 'error'). This change ensures diagnostic messages are visually distinct, improving clarity and reducing debugging effort during test runs. Refs: nodejs#55922
Enhanced to include colors for the following diagnostic levels: : blue - info : yellow - warn : red - error Refs: nodejs#55922
Updated coverage threshold checks in to use the parameter when calling. Errors now use the 'error' level for red-colored formatting. This ensures coverage errors are highlighted effectively in the output. Fixes: nodejs#55922
implemented requested change by removing debug from reporterColorMap Refs: nodejs#55964 (review)
188f357 to
038b0f8Compareupdated the documentation for the 'test:diagnostic' event to
include the new level parameter. clarified its purpose, default
value, and possible severity levels ('info', 'warn',
'error').
Fixes: nodejs#55922Add a test in to verify that the diagnostic error messages about unmet coverage thresholds are displayed in red when using the spec reporter. Fixes: nodejs#55922
hpatel292-seneca
commented
Nov 27, 2024
Hi @pmarchini@cjihrig, |
Hi, @pmarchini@cjihrig I will add one more commit for fixing the lint error. Should I go ahead and send commit? |
pmarchini
commented
Nov 27, 2024
Sure 🚀 |
Added eslint-disable comment to bypass no-control-regex. This allows testing ANSI escape sequences for red color in error messages without triggering lint errors. Fixes: nodejs#55922
hpatel292-seneca
commented
Nov 27, 2024
@pmarchini I did. Thanks |
hpatel292-seneca
commented
Nov 28, 2024
Hi @pmarchini@cjihrig, |
Hi @cjihrig, node/test/parallel/test-runner-run.mjs Line 28 in 96cd2a6 I figured out that I should test like this it('should emit diagnostic events with correct level and message',async()=>{constdiagnosticEvents=[];conststream=run({files: ['test-file.js'],// I am confused here!!reporter: 'spec',});stream.on('test:diagnostic',(event)=>{diagnosticEvents.push(event.data);});forawait(const_ofstream);assert(diagnosticEvents.length>0,'No diagnostic events were emitted');consterrorEvent=diagnosticEvents.find((e)=>e.level==='error');assert(errorEvent,'No error-level diagnostic events found');assert.match(errorEvent.message,"Error Message",'Diagnostic message format mismatch');});I am not sure how to write that test-file.js which I want to run in this test. |
cjihrig
commented
Dec 13, 2024
I think you'd want to do something like this. That will let you run the same test fixture that you're running in your test from |
hpatel292-seneca
commented
Dec 13, 2024
So I tried that like this it('should emit diagnostic events with correct level and message',async()=>{constdiagnosticEvents=[];// Run the test suite and capture eventsconststream=run({files: [join(testFixtures,'coverage.js')],reporter: 'spec',});stream.on('test:diagnostic',(event)=>{diagnosticEvents.push(event);});forawait(const_ofstream);console.log(diagnosticEvents)// here I am printing the events// Assertions on diagnostic eventsassert(diagnosticEvents.length>0,'No diagnostic events were emitted');consterrorEvent=diagnosticEvents.find((e)=>e.level==='error');});So I think I am able to capture As you can see from the output diagnostic events contain the constresult=spawnSync(process.execPath,['--test','--experimental-test-coverage',`${coverage.flag}=99`,'--test-reporter','spec',fixture,],{env: { ...process.env,FORCE_COLOR: '3'},});```
Itriedaddingflagsinrunapicallbutit's not working. |
cjihrig
commented
Dec 13, 2024
I think that's fine. We don't need to make sure the level is set to |
add a test to ensure that diagnostic events emitted by the test runner contain level parameter. Refs: nodejs#55964
hpatel292-seneca
commented
Dec 13, 2024
Hi @cjihrig, |
nodejs-github-bot
commented
Dec 13, 2024
Added eslint-disable-next-line to bypass no-unused-vars check ref: nodejs#55964
nodejs-github-bot
commented
Dec 13, 2024
nodejs-github-bot
commented
Dec 14, 2024
nodejs-github-bot
commented
Dec 14, 2024
nodejs-github-bot
commented
Dec 15, 2024
nodejs-github-bot
commented
Dec 17, 2024
| assert(!findCoverageFileForPid(result.pid)); | ||
| }); | ||
| test(`test failing ${coverage.flag} with red color`, () => { |
There was a problem hiding this comment.
hey @hpatel292-seneca, it seems this test is failing in the CI.
Could you please take a look at it?
There was a problem hiding this comment.
Hey @pmarchini, I tried reading logs on failed tests and it seems like this regex
is not matching with the output. But as we are not logging anything I am not sure what we are getting output here.
But this test is running as expected and passing locally and in GitHub Action CI.
Github Action run: https://github.com/nodejs/node/actions/runs/12320594158/job/34394971556
Below is the SS of the locally run test
There was a problem hiding this comment.
There was a problem hiding this comment.
Yes, I saw this CI failure but I am not sure what is causing the failure as this test is running as expected on my local machine. I tried reading logs on CI but I can't find anything. Is there any way I can debug those CI runs??
There was a problem hiding this comment.
Hey @hpatel292-seneca, if you follow the link I've shared, you will find the description of the error.
It seems that the expected red error does not match any part of the output.
I've restarted many times, and the error systematically appears across different OS.
There was a problem hiding this comment.
Hey @pmarchini, I got your point but the test is running as expected with red color on my machine.
For example: I am printing output in the test like this
test(`test failing ${coverage.flag} with red color`,()=>{constresult=spawnSync(process.execPath,['--test','--experimental-test-coverage',`${coverage.flag}=99`,'--test-reporter','spec',fixture,],{env: { ...process.env,FORCE_COLOR: '3'},});conststdout=result.stdout.toString();console.log(stdout); \\ HereIamprintingoutput// eslint-disable-next-line no-control-regexconstredColorRegex=/\u001b\[31mℹError:\d{2}\.\d{2}%\w+coveragedoesnotmeetthresholdof99%/;assert.match(stdout,redColorRegex,'Expected red color code not found in diagnostic message');assert.strictEqual(result.status,1);assert(!findCoverageFileForPid(result.pid));});And this is the output I am getting:
And here is the logs of failed pipeline run
actual: 'invalid tap output
✔ a test (3.3287ms)
ℹ tests 1
ℹ suites 0
ℹ pass 1
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 304.826895',
expected: /\u001b[31mℹ Error: \d{2}\.\d{2}% \w+ coverage does not meet threshold of 99%/,
So basically, it is not even printing the coverage error which I think it shoud print if I am using this flags --experimental-test-coverage, ${coverage.flag}=99, and --test-reporter', 'spec which I am using in the test.
Do you think it is a environment or configuration issue??
There was a problem hiding this comment.
I would say yes but in the CI it's also failing under Windows OS
implemented requested change by removing debug from reporterColorMap Refs: nodejs#55964 (review)
add a test to ensure that diagnostic events emitted by the test runner contain level parameter. Refs: nodejs#55964
Added eslint-disable-next-line to bypass no-unused-vars check ref: nodejs#55964
vassudanagunta
commented
Jun 27, 2025
Shouldn't this be closed given #57923? |
pmarchini
commented
Jun 28, 2025
hey @vassudanagunta, you're totally right! |

This fixes#55922
Change summary
Updated the reporter.diagnostic to accept level parameter like this
Then I updated
#handleEventlike thisAnd I am Updated
reporterColorMaplike thisand color already contain logic for this colors
I also set the reporter.diagnostic call from test.js like this (level="Error")
Here is Demo output:
