Uh oh!
There was an error while loading. Please reload this page.
Include exception message in JUnit failure/error bodies - #10286
Conversation
.NET's Exception.StackTrace omits the leading ype: message header that Java's Throwable.printStackTrace() (and Exception.ToString()) include, so writing the stack trace alone into the <failure>/<error> body dropped the single most useful piece of diagnostic information. Consumers that render the body rather than the message attribute -- GitLab CI and CircleCI most notably -- showed only a stack trace with no indication of why a test failed. The body now mirrors printStackTrace() shape, degrading gracefully when the exception type, message or stack trace is absent. The message and ype attributes are unchanged, so consumers reading them directly are unaffected. Also always emit the ype attribute. The Ant/windyroad JUnit.xsd marks it use="required" (Surefire relaxes it to optional), but MTP only supplies an exception type when the state property carried an actual Exception -- frameworks reporting via Explanation alone leave it null. Fall back to the element name so the document stays valid under the stricter schema. Fixes#10269 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a565c557-8c36-4279-ab7b-41311abcd836
There was a problem hiding this comment.
Pull request overview
Updates JUnit reports to provide complete failure diagnostics in body-rendering CI systems.
Changes:
- Adds exception type/message headers to failure and error bodies.
- Always emits the required
typeattribute. - Adds unit, acceptance, API tracking, and RFC updates.
Show a summary per file
| File | Description |
|---|---|
JUnitXmlWriter.cs | Builds diagnostic failure/error bodies and type fallbacks. |
InternalAPI.Unshipped.txt | Tracks the new internal helper. |
JUnitReportFailureBodyTests.cs | Tests body composition and XML output. |
JUnitReportTests.cs | Updates acceptance snapshots. |
016-JUnit-Report.md | Documents the revised format. |
Review details
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Medium
Uh oh!
There was an error while loading. Please reload this page.
This comment has been minimized.
This comment has been minimized.
🔴 Build Failure AnalysisRoot cause: All 6 build legs fail with the same error in Why this PR is affected: The most recent commit to This PR’s changes are unrelated — all modifications are in FixRegenerate the XLF files for dotnet msbuild src/Analyzers/MSTest.Analyzers/MSTest.Analyzers.csproj /t:UpdateXlfThen commit the updated
|
The outcome-mapping table and the XML skeleton both rendered <failure> and <error> as self-closing while the table also claimed a body, which is an impossible XML shape. Now that these elements always carry a body, show them with explicit start/end tags in both places, and correct the <skipped> row: the writer emits only the message attribute, never a body. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a565c557-8c36-4279-ab7b-41311abcd836
This comment has been minimized.
This comment has been minimized.
WriteXmlAsync_ErroredTestWithoutException_FallsBackToElementNameForTypeAttribute verified the element name and the type fallback but left the message attribute and the body content unasserted, so a regression in either would have gone undetected by this test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a565c557-8c36-4279-ab7b-41311abcd836
🧪 Test quality grade — PR #10286All 9 new test methods are in
This advisory comment was generated automatically. Grades are heuristic
|
Uh oh!
There was an error while loading. Please reload this page.
Fixes#10269
Include the exception message in
<failure>/<error>bodiesToday the JUnit report writes only
Exception.StackTraceinto the element body, keeping the message in themessageattribute. Consumers that render the body rather than the attribute — GitLab CI and CircleCI most notably — therefore show a bare stack trace with no indication of why the test failed. This is especially damaging for fluent assertion libraries, where the stack trace is mostly framework frames while the assertion message carries the actual diagnosis.This is a fidelity gap, not a stylistic choice. Canonical JUnit bodies come from Java's
Throwable.printStackTrace(), whose first line iscom.example.AssertionError: expected:<1> but was:<2>— the message is inherently part of the body. .NET'sException.StackTraceomits that header (onlyToString()includes it), which is exactly why our output looked message-less next to a genuine Surefire report. GitLab isn't being quirky; it's reading the body assuming Java semantics.The body now mirrors
printStackTrace():Each part degrades gracefully: a missing exception type drops the header prefix, a missing message drops the
:separator, and a missing stack trace yields a header-only body.Why change the default instead of adding an option
The issue proposed a
--report-junit-failure-body-formatoption. We went with changing the behavior outright:Microsoft.Testing.Extensions.JUnitReporthas not shipped a stable version (only1.0.0-alpha.*on NuGet), so there is no back-compat cost to fixing the default now. This is the one moment where it's free.JUnit.xsdnor Surefire'ssurefire-test-report.xsdconstrains the body — it's free-form text (the Ant schema's "e.g., a stack trace" is illustrative, not normative).The
messageandtypeattributes are unchanged, so consumers reading them directly are unaffected. The resulting duplication between attribute and body is exactly what every Maven/Surefire report already exhibits, so consumers rendering both have long handled it.Unlike spekt's
FailureBodyFormat=Verbose, we deliberately do not fold standard output into the failure body — it's already emitted as<system-out>per test case, and duplicating it would bloat reports for noisier suites.Always emit the
typeattributeThe Ant/windyroad
JUnit.xsdmarkstypeasuse="required"on<failure>/<error>(Surefire relaxes it to optional). MTP only supplies an exception type when the state property carried an actualException— frameworks that report a failure throughExplanationalone leave it null, so the attribute was previously omitted entirely in that case.typeis now always written, falling back to the element name (failure/error) when no exception type is available. That keeps the document valid under the stricter schema without inventing a bogus exception type name.Testing
JUnitReportFailureBodyTests) covering every combination of present/absent exception type, message and stack trace, plus thetypefallback for both<failure>and<error>, asserted against real generated XML.Microsoft.Testing.Platform.Acceptance.IntegrationTests/JUnitReportTests.cs— the<failure>/<error>elements are no longer self-closing.Docs
docs/RFCs/016-JUnit-Report.mdupdated with a "Failure and error body format" section, a "Thetypeattribute" section, and the revised outcome-mapping table.