Uh oh!
There was an error while loading. Please reload this page.
Emit ::warning annotations for skipped tests in GitHubActionsReport - #9641
Conversation
Skipped tests were previously invisible in the GitHub Actions Annotations tab. Surface them as title-only ::warning workflow commands under the existing --report-gh-annotations knob, mirroring the RFC #9003 Tier 1 design. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Amaury Levé (Evangelink)
commented
Jul 5, 2026
/backport to rel/4.3 |
Started backporting to rel/4.3: https://github.com/microsoft/testfx/actions/runs/28752868353 |
There was a problem hiding this comment.
Pull request overview
This PR extends Microsoft.Testing.Extensions.GitHubActionsReport so that skipped tests now surface in the GitHub Actions Annotations tab. Previously only failing tests produced ::error annotations; skipped tests were silently absent. This implements the clearly-scoped Tier 1 gap from RFC #9003 (::warning for skipped tests), while Tiers 2/3 (Check Runs / sticky PR comments needing GITHUB_TOKEN) remain follow-ups.
Changes:
GitHubActionsAnnotationReporternow handlesSkippedTestNodeStatePropertyand emits a title-only::warningvia a new testableGetSkippedAnnotationhelper that mirrorsGetErrorAnnotation.- Adds resources
SkippedAnnotationTitle/NoSkipReasonFallback, regenerates all 13.xlffiles, and broadens the--report-gh-annotationsdescription to cover "failed and skipped tests" (no new CLI option). - Updates unit tests, the
HelpInfoAllExtensionsTestshelp/info expectations,PACKAGE.md, and the class doc comment.
Show a summary per file
| File | Description |
|---|---|
GitHubActionsAnnotationReporter.cs | Adds skipped-test branch + WriteSkippedAnnotationAsync / GetSkippedAnnotation emitting title-only ::warning; updates doc comment |
Resources/GitHubActionsResources.resx | Adds SkippedAnnotationTitle / NoSkipReasonFallback; broadens annotations option description |
Resources/xlf/*.xlf (13 files) | Regenerated via UpdateXlf with the two new trans-units and updated option description (correct alphabetical id order) |
GitHubActionsAnnotationReporterTests.cs | Adds two tests for the skipped-annotation reason and fallback paths, including escaping |
HelpInfoAllExtensionsTests.cs | Updates --help and --info expectations to the new option description |
PACKAGE.md | Documents skipped-test ::warning behavior and updates the options table row |
The change is additive, well-scoped, and consistent with the existing failure-annotation path: both branches use the same GetTestName, newline-prepend, and escaping conventions. The unit tests match the existing "test the pure formatter helper" convention, help/info expectations are the only place the option description is asserted and are updated, and the .xlf regeneration preserves correct alphabetical id ordering. I found no objective issues.
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 0
- Review effort level: Medium
🧪 Test quality grade — PR #9641
This advisory comment was generated automatically. Grades are heuristic
|
There was a problem hiding this comment.
Note
🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.
Summary
Clean implementation that follows established patterns. No blocking issues found.
✅ What looks good
- Correctness: Skipped test handling is correctly placed in the
failure is nullbranch ofConsumeAsync, wrapped by the existing try/catch for graceful degradation. - Escaping: Uses the same
GitHubActionsEscaper.EscapeProperty/EscapeDatapipeline as error annotations — unit tests confirm\n→%0Aand\r→%0Dhandling. - API surface:
GetSkippedAnnotationisinternal static— no public API change; noPublicAPI.Unshipped.txtentry required. - Resources/XLF: New strings added correctly with format comments;
UpdateXlfregenerated all 14 locale files. As a bonus, the regeneration incidentally fixes a pre-existing typo in pt-BR and zh-Hans XLFs (both had stale/incorrect source text forAnnotationsOptionDescription). - Testing: Two new unit tests covering explanation-present and null-explanation paths. Pattern is consistent with existing
GetErrorAnnotationtests. - ConfigureAwait(false): Correctly applied on the new async path.
📝 Minor observations (non-blocking)
- Unrelated change: The
.mcp.jsondeletion is bundled in this PR — not harmful, but ideally would be a separate commit for a cleaner history. - GitHub annotation limit awareness: GitHub Actions silently caps annotations at ~10 per type per step. Suites with many skipped tests will see truncation on the Annotations tab. Not a code defect (the reporter can't work around it), but worth a note in PACKAGE.md if users report confusion in the future.
LGTM — no changes requested.
Uh oh!
There was an error while loading. Please reload this page.
… by @Evangelink in #9641 (backport to rel/4.3) (#9644) Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a new ## G section with a GitHubActionsReport entry covering the Microsoft.Testing.Extensions.GitHubActionsReport MTP extension. The entry documents all four CLI features (log groups, failure/skip annotations, job summary, slow-test notices), MSTest.Sdk opt-in, and NativeAOT/VSTest limitations. Identified from PR #9641 (weekly full scan, 2026-07-06). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ped-test warnings) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
What
Adds skipped-test annotations to
Microsoft.Testing.Extensions.GitHubActionsReport. Previously, only failing tests produced::errorannotations; skipped tests were silently absent from the GitHub Actions Annotations tab. This change surfaces each skipped test as a title-only::warningworkflow command so intentionally/unexpectedly skipped tests are visible alongside failures.Why
While assessing RFC #9003 against the shipped extension, this was the one clearly-scoped Tier 1 (no-auth, workflow-command) gap. The RFC explicitly lists "
::warning… for skipped" in Tier 1. Tiers 2/3 (Check Runs and sticky PR comments via the GitHub API +GITHUB_TOKEN) remain a separate follow-up.How
GitHubActionsAnnotationReporternow handlesSkippedTestNodeStatePropertyand emits a title-only::warning(skipped nodes carry no exception/stack trace, so there is nofile:lineto pin to). Added a testableGetSkippedAnnotationhelper mirroringGetErrorAnnotation.--report-gh-annotationsknob (no new CLI option); broadened its description to "failed and skipped tests".SkippedAnnotationTitle/NoSkipReasonFallback; regenerated all 14.xlffiles viaUpdateXlf.HelpInfoAllExtensionsTests(--help+--info) andPACKAGE.mdto match.Behavioral notes
::warningis purely informational in GitHub Actions — it does not affect step/job pass-fail (that's driven solely by the process exit code, which this reporter never touches). The reporter'sConsumeAsyncis wrapped in try/catch that degrades to "no annotation" on any exception. The feature stays behind the--report-ghmaster switch (off by default) and the on-by-default--report-gh-annotationsknob.Verification
Microsoft.Testing.Extensions.UnitTestsbuild clean (0 warnings/errors).🤖 Assisted by GitHub Copilot CLI.