test: add windows and c++ coverage by bcoe · Pull Request #35670 · nodejs/node

bcoe · 2020-10-16T04:51:46Z

Experiment to see if collecting coverage for Windows will "just work".

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

bcoe · 2020-10-18T14:08:00Z

 run: npx envinfo
+ - name: Clone gcovr reporter
+ run: git clone -b 3.4 --depth=1 --single-branch https://github.com/gcovr/gcovr.git
+ - name: Clone patch for gcovr


@mhdawson do you remember why we're running gcovr from a branch and floating a patch? It would be nice to simplify this.

It been long enough that I don't remember the specifics. I assume it failed to run properly with out that patch. It's possible a later version of gcovr may have fixed it.

@mhdawson I've managed to get gcovr@4.2 working, installed directly from pip.
The trick was to continue running from the out/ folder, but to specify a --root pointing to the root project directory.

bcoe · 2020-10-18T19:48:36Z

@richardlau @Trott @watilde @mhdawson , I think this is ready for review:

coverage is now collected for C++ using gcov, and for Windows.
coverage reports won't show up until both the Windows and linux coverage runs finish.
I've added codecov configuration that removes the graph from the report (the default message on the PR took up a lot of space.).

Why the drop in coverage?

This is the first time we've combined C++ and JavaScript coverage, and our C++ coverage is a bit below the JavaScript thresholds -- I also noticed that codecov.io's numbers are a bit lower than our uploaded reports, I think this is because it treats partial line misses the same as misses.

What's next

Once this has been running for a week or two, and we're happy, I think we should:

remove some of the coverage logic from our Makefile, I'd be happy to do this.
it would be nice to stop floating a patch for gcovr (it looks like we have been for years.).
should we consider redirecting coverage.nodejs.org, to the reports on codecov.io, and remove a section of the website we maintain?

mhdawson · 2020-10-19T21:43:03Z

Is it expected that I have to log in to see what under and 266 more ?

It seems to want quite a bit of access to be authorized when I go to do that...

mhdawson · 2020-10-19T21:44:18Z

Does remove some of the coverage logic from our Makefile, mean that you can no longer run coverage locally?

bcoe · 2020-10-20T15:48:54Z

@mhdawson did you manage to open the reports? perhaps I can open an issue with codecov.io independently of this PR, since I don't believe this behavior is new.

I've noticed myself in the past, that when looking at Node.js' nightly reports, I sometimes get prompted to authenticate -- I wonder if it happens to folks who have a prior session with codecov.io that's expired.

bcoe · 2020-10-20T15:49:45Z

 "html",
 "text",
-"lcov"
+"cobertura"


I'm hoping this gives us better partial coverage (I noticed C++ reports branch level coverage).

targos · 2020-10-20T15:50:47Z

@bcoe It also asks me to authenticate, even in an incognito window.

bcoe · 2020-10-20T15:54:54Z

@thomasrockhu sorry to keep bothering you (I think we're on the right track to getting this configured appropriately for the project).

Folks are reporting that codecov.io is prompting them to authenticate to see the public reports:

@bcoe It also asks me to authenticate, even in an incognito window.
Is it expected that I have to log in to see what under and 266 more ?

Ideally there'd never be an authentication step for our users to view detailed reports, as the Node.js project is open source.

Is there a configuration setting I'm missing?

bcoe · 2020-10-20T18:18:02Z

@targos @mhdawson is the authentication issue a blocker, or is it okay if I open a tracking issue on codecov.io?

I seem to not be getting the authentication screen myself, either in an incognito window, or in another browser -- but, I have seen the behavior you're describing before -- I was hoping it was a bug that had been addressed, since I haven't been seeing a login prompt myself.

targos · 2020-10-20T18:55:00Z

It's not a blocker to me

mhdawson · 2020-10-21T16:59:27Z

Thanks for following up on the auth. As long as its not something that will be required long term it's not a blocker for me.

mhdawson

LGTM

nodejs-github-bot · 2020-10-21T23:23:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/33792/

bcoe · 2020-10-22T02:46:52Z

Landed in 7657f62

Collect Windows and C++ coverage. Configure codecov so that comments are more concise and are only left when coverage varies. PR-URL: #35670Fixes: #35696 Refs: #35653 Refs: #35646 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

thomasrockhu · 2020-10-22T13:18:03Z

Hi @bcoe, I noticed you opened this ticket so I'll address the issue there. Users shouldn't be seeing any problems viewing open source reports, so there's definitely an issue. Thanks for raising it up!

Collect Windows and C++ coverage. Configure codecov so that comments are more concise and are only left when coverage varies. PR-URL: #35670Fixes: #35696 Refs: #35653 Refs: #35646 Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Antoine du Hamel <duhamelantoine1995@gmail.com> Reviewed-By: Richard Lau <rlau@redhat.com> Reviewed-By: Jiawen Geng <technicalcute@gmail.com> Reviewed-By: Michael Dawson <midawson@redhat.com>

nodejs-github-bot added the meta Issues and PRs related to the general management of the project. label Oct 16, 2020

bcoe added the wip Issues and PRs that are still a work in progress. label Oct 16, 2020

bcoe changed the title ~~test: windows coverage GitHub action~~test: add windows and C++ coverageOct 17, 2020

test: add windows and C++ coverage
df981ea

bcoe force-pushed the windows-coverage-action branch from aaa926e to df981eaCompare October 17, 2020 16:13

build: improve coverage configuration
d91b81d

nodejs deleted a comment from codecov-io Oct 18, 2020

bcoe mentioned this pull request Oct 18, 2020
Coverage report left in comments is "noisy" #35696
Closed

bcoe added 4 commits October 17, 2020 22:04

chore: attempt to build with gcov
656160f

build: ensure gcovr tool exists
9eac50b

chore: tweak codecov.yml file
f869255

multi-line command had issues
f5d6272

nodejs deleted a comment from codecov-io Oct 18, 2020

bcoe commented Oct 18, 2020
View reviewed changes

Comment threadcodecov.yml

bcoe added 2 commits October 18, 2020 08:35

chore: delete more output files
40cb785

build: run test-cov rather than test-ci
fc7c05c

bcoe removed the wip Issues and PRs that are still a work in progress. label Oct 18, 2020

bcoe requested review from Trott, mhdawson, richardlau and watilde and removed request for mhdawson October 18, 2020 19:41

bcoe added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 18, 2020

bcoe changed the title ~~test: add windows and C++ coverage~~test: add windows and c++ coverageOct 18, 2020

Trott approved these changes Oct 18, 2020
View reviewed changes

richardlau reviewed Oct 19, 2020
View reviewed changes

Comment thread.github/workflows/coverage-windows.yml Outdated

bcoe requested a review from mhdawson October 20, 2020 05:09

bcoe added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 20, 2020

aduh95 approved these changes Oct 20, 2020
View reviewed changes

Comment thread.github/workflows/coverage-linux.yml
Comment thread.github/workflows/coverage-linux.yml
Comment thread.github/workflows/coverage-windows.yml
Comment thread.github/workflows/coverage-windows.yml

richardlau approved these changes Oct 20, 2020
View reviewed changes

gengjiawen approved these changes Oct 20, 2020
View reviewed changes

build: a couple more small tweaks to reports
c646f6d

bcoe commented Oct 20, 2020
View reviewed changes

nodejs deleted a comment from codecov-io Oct 20, 2020

mhdawson approved these changes Oct 21, 2020
View reviewed changes

This comment has been minimized.
Sign in to view

bcoe closed this Oct 22, 2020

bcoe deleted the windows-coverage-action branch October 22, 2020 02:46

richardlau mentioned this pull request Oct 22, 2020
Total regression in codecov coverage report looks suspicious #35759
Closed

targos mentioned this pull request Nov 3, 2020
v15.1.0 release proposal #35948
Merged

BethGriggs mentioned this pull request Dec 10, 2020
v14.15.2 proposal #36476
Merged

BethGriggs added the dont-land-on-v14.x label Jan 18, 2021

Uh oh!

Uh oh!

Conversation

bcoe commented Oct 16, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

bcoeOct 18, 2020

Choose a reason for hiding this comment

Uh oh!

mhdawsonOct 19, 2020

Choose a reason for hiding this comment

Uh oh!

bcoeOct 20, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bcoe commented Oct 18, 2020

Why the drop in coverage?

What's next

Uh oh!

Uh oh!

mhdawson commented Oct 19, 2020

Uh oh!

mhdawson commented Oct 19, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bcoe commented Oct 20, 2020

Uh oh!

bcoeOct 20, 2020

Choose a reason for hiding this comment

Uh oh!

targos commented Oct 20, 2020

Uh oh!

bcoe commented Oct 20, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bcoe commented Oct 20, 2020

Uh oh!

targos commented Oct 20, 2020

Uh oh!

mhdawson commented Oct 21, 2020

Uh oh!

mhdawson left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

nodejs-github-bot commented Oct 21, 2020

Uh oh!

bcoe commented Oct 22, 2020

Uh oh!

thomasrockhu commented Oct 22, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

bcoe commented Oct 16, 2020•
edited
Loading

bcoe commented Oct 20, 2020•
edited
Loading

thomasrockhu commented Oct 22, 2020•
edited
Loading