Uh oh!
There was an error while loading. Please reload this page.
Add --list-tests json for machine-readable test discovery output - #8280
Merged
Conversation
Adds an optional json argument to the existing --list-tests flag so MTP can
emit a single JSON document on stdout describing every discovered test. The text
output and zero-arg behavior are unchanged.
CLI surface:
--list-tests (text, unchanged)
--list-tests text (explicit alias for the default)
--list-tests json (JSON document on stdout, banner/progress/summary/per-test
text suppressed; errors routed to stderr)
Schema v1 (omits absent fields, no nulls):
{ schemaVersion, tests[{ uid, displayName,
type{ assemblyFullName, namespace?, typeName,
methodName, methodArity, returnTypeFullName,
parameterTypeFullNames[] },
location{ file, lineStart, lineEnd },
traits[{key, value}],
properties[{key, value}] }] }
Implementation notes:
- `DiscoveredTestsJsonSerializer` + small `JsonStringWriter` (no
`System.Text.Json`/`Jsonite` dependency, so the same code runs on
netstandard2.0, net462, net8.0, net9.0). LF line endings; escapes U+2028 /
U+2029 and replaces lone surrogates with U+FFFD, matching
`Utf8JsonWriter`'s default behavior.
- `TerminalOutputDevice` detects JSON mode, suppresses banner/progress/per-test
text/free-form data on stdout, buffers discovered TestNodes from
`ConsumeAsync`, and emits the JSON document once at session end from the
test-host process only. Ctrl+C in JSON mode writes a single cancellation line
to stderr instead of corrupting the JSON document.
- `properties` and `traits` are arrays of `{key, value}` so duplicates
survive serialization and order is stable.
Tests:
- Unit: `DiscoveredTestsJsonSerializerTests` (11) covering empty / minimal /
type metadata / global namespace / file location / traits ordering /
properties array / duplicate property keys / special-character escaping /
lone-surrogate / valid surrogate pair / U+2028 U+2029.
- Unit: `PlatformCommandLineProviderTests` (4 new) for arity validation,
accepted values (text/json case-insensitive), invalid-value message, and the
`IsListTestsJsonOutput` helper.
- Acceptance: `MSTest.Acceptance.IntegrationTests.TestDiscoveryTests` (2 new ×
2 TFMs) parses the JSON document end-to-end (every v1 field for Test1),
asserts stderr is empty on success, and asserts invalid value fails with
`ExitCode.InvalidCommandLine`.
- Updated `HelpInfoTests` (x2) and `HelpInfoAllExtensionsTests` wildcard
expectations for the new arity (0..1) and description.
Resources updated and all 13 XLFs regenerated via `/t:UpdateXlf`.
No public API surface added.
Fixes#3221.
Related: dotnet/sdk#49754.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>Contributor
There was a problem hiding this comment.
Pull request overview
Adds machine-readable JSON output for --list-tests while preserving the existing text behavior.
Changes:
- Adds
json/textargument handling and validation for--list-tests. - Adds JSON discovery serialization and suppresses normal terminal output in JSON mode.
- Adds unit and acceptance coverage plus updated help/resource localization files.
Show a summary per file
| File | Description |
|---|---|
src/Platform/Microsoft.Testing.Platform/CommandLine/PlatformCommandLineProvider.cs | Allows optional --list-tests output format and validates accepted values. |
src/Platform/Microsoft.Testing.Platform/OutputDevice/TerminalOutputDevice.cs | Buffers discovered tests and emits JSON-only stdout in JSON mode. |
src/Platform/Microsoft.Testing.Platform/OutputDevice/DiscoveredTestsJsonSerializer.cs | Serializes discovered test nodes into schema v1 JSON. |
src/Platform/Microsoft.Testing.Platform/OutputDevice/JsonStringWriter.cs | Adds a small JSON writer for cross-target serialization. |
src/Platform/Microsoft.Testing.Platform/Resources/PlatformResources.resx | Updates CLI help text and invalid argument resource. |
src/Platform/Microsoft.Testing.Platform/Resources/PlatformResources.cs | Exposes the new validation resource to unit tests. |
src/Platform/Microsoft.Testing.Platform/Resources/xlf/* | Regenerates localized resource entries. |
test/UnitTests/Microsoft.Testing.Platform.UnitTests/CommandLine/PlatformCommandLineProviderTests.cs | Covers list-tests argument validation and JSON mode detection. |
test/UnitTests/Microsoft.Testing.Platform.UnitTests/OutputDevice/DiscoveredTestsJsonSerializerTests.cs | Covers JSON schema, escaping, ordering, and optional fields. |
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestDiscoveryTests.cs | Adds acceptance coverage for JSON discovery output and invalid format. |
test/IntegrationTests/*/HelpInfo*.cs | Updates help/info expectations for the new option description and arity. |
Copilot's findings
- Files reviewed: 25/25 changed files
- Comments generated: 1
Uh oh!
There was an error while loading. Please reload this page.
Amaury Levé (Evangelink)
left a comment
MemberAuthor
There was a problem hiding this comment.
Review Summary — PR #8280: --list-tests json
Overall this is a well-structured, well-tested feature. The custom JsonStringWriter, cross-TFM compatibility, lone-surrogate handling, banner/progress suppression, and CLI validation are all cleanly implemented. Tests cover the happy path, special character escaping, and invalid-argument rejection.
Findings
| # | Dimension | Severity | File | Finding |
|---|---|---|---|---|
| 1 | Threading/Hot-reload | MODERATE | TerminalOutputDevice.cs:379 | DisplayAfterHotReloadSessionEndAsync calls DisplayAfterSessionEndRunInternalAsync unconditionally; in a dotnet watch --list-tests json session the JSON document is emitted after every cycle and _discoveredTestsForJson is never cleared — multiple growing JSON blobs on stdout. |
| 2 | Testability/Architecture | NIT | TerminalOutputDevice.cs:420 | Console.Error.WriteLineAsync is used directly (also in the abort callback in InitializeAsync) bypassing the injected _console. This makes error/stderr output in JSON mode un-redirectable in tests. The comment acknowledges the limitation; worth tracking. |
Clean dimensions
- Algorithmic Correctness — JSON writer logic, escaping, surrogate pair handling all correct.
- Threading —
_discoveredTestsForJsonwrite path is guarded by MTP's single-consumer-per-pump invariant; the emission path uses_asyncMonitor. Correct. - Security — No path traversal, no serialization of untrusted schemas, no sensitive leakage.
- Public API — No new public API surface;
IsListTestsJsonOutputisinternal static.PublicAPI.Unshipped.txtnot dirtied. - Cross-TFM — Custom
JsonStringWriteravoids bothSystem.Text.JsonandJsonite; no#if-guarded APIs needed. - Performance —
StringBuilder-based writer; no hot-path LINQ;PropertyBag.OfType<T>returns an array,Array.Reverseis in-place. Reasonable. - CLI validation — Argument validated in
ValidateOptionArgumentsAsync;ZeroOrOnearity correct; case-insensitive comparison correct; resource string properly parametrised. - Localization — New strings in
.resx; XLFs regenerated via MSBuild, not hand-edited. - Test quality — Unit tests cover empty, minimal, full, special-char, surrogate, and validation cases. Integration tests cover JSON output schema, invalid arg, and unchanged text mode.
- Assertion style — Uses MSTest conventions consistently throughout.
- Scope discipline — PR is tightly scoped to the new
--list-tests jsonflag; no unrelated refactoring.
Generated by Expert Code Review (on open) for issue #8280 · ● 19.8M
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
- Restore ExtensionResources.pl.xlf to match the updated resx (lost during rebase earlier; was the cause of the CI XLF out-of-date failure for the TRX extension). - Help text now documents both `text` and `json` arguments (review comment by Copilot on PlatformResources.resx:389). - Hot-reload: `DisplayAfterHotReloadSessionEndAsync` now early-returns in JSON mode so `dotnet watch --list-tests json` doesn't emit multiple ever-growing JSON documents (review comment by Evangelink, MODERATE). - Extract `WriteToStandardErrorAsync` helper so the single `Console.Error` bypass of `IConsole` is centralized and easy to find when `IConsole` eventually gains stderr support (review comment by Evangelink, NIT). - Update help/info acceptance test expectations to match the new description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé (Evangelink)
commented
May 16, 2026
MemberAuthor
Pushed 884c759 to address review comments and unblock CI:
Help/info acceptance tests updated for the new description; XLFs regenerated; |
…llName MSTest's VSTestBridge (MSTestBridgedTestFramework.cs:105) intentionally emits `assemblyFullName` and `returnTypeFullName` as empty strings, with a TODO to populate them in the future. The acceptance test was asserting that these fields are non-empty, which broke on every TFM as soon as the test actually ran in CI. Weaken the assertion to verify presence (`ValueKind == String`) rather than non-emptiness, matching today's adapter behavior. The other field assertions (`typeName`, `methodName`, `methodArity`, `parameterTypeFullNames`) still pin their concrete values. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Lori (Lorilatschki)
commented
May 18, 2026
Amaury Levé (@Evangelink), we are using the |
Amaury Levé (Evangelink)
commented
May 18, 2026
MemberAuthor
It's currently only on MTP (not yet released), I will dogfood it a little then add support to SDK side. |
This was referenced May 28, 2026
This was referenced Jun 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an optional
jsonargument to the existing--list-testsflag so MTP can emit a single JSON document on stdout describing every discovered test. The text output and zero-arg behavior are unchanged.Fixes#3221.
Related: dotnet/sdk#49754.
CLI surface
--list-tests--list-tests text--list-tests json--list-tests <anything-else>ExitCode.InvalidCommandLineand a descriptive validation errorSchema v1
Flat array; absent fields are omitted (no
null), so frameworks that don't populateTestMethodIdentifierPropertyaren't broken.{ ""schemaVersion"": 1, ""tests"": [ { ""uid"": ""..."", ""displayName"": ""..."", ""type"": { ""assemblyFullName"": ""..."", ""namespace"": ""..."", // omitted for global namespace""typeName"": ""MyClass+Nested`1"", // metadata format with arity & nesting""methodName"": ""MyMethod"", ""methodArity"": 0, ""returnTypeFullName"": ""..."", ""parameterTypeFullNames"": [""..."", ...] }, ""location"": { ""file"": ""..."", ""lineStart"": 12, ""lineEnd"": 18 }, ""traits"": [ { ""key"": ""Category"", ""value"": ""Integration"" }, ... ], ""properties"": [ { ""key"": ""ExecutionId"", ""value"": ""..."" }, ... ] } ] }traitsandpropertiesare both arrays of{ key, value }(not JSON objects) so duplicate keys survive serialization and order is stable across runs.Implementation notes
DiscoveredTestsJsonSerializer+ tinyJsonStringWriter— no dependency onSystem.Text.Json(not available across all targets) orJsonite(#if !NETCOREAPPonly), so the same code path runs on netstandard2.0, net462, net8.0, net9.0.U+2028/U+2029escaped, lone surrogates replaced withU+FFFD(mirrorsUtf8JsonWriterdefaults).TerminalOutputDevicedetects JSON mode, suppresses banner / progress / per-test text / free-form output on stdout, buffersDiscoveredTestNodeStatePropertyevents fromConsumeAsync, and emits the JSON document once at session end from the test-host process only (controller does not double-emit).PlatformCommandLineProviderflips--list-testsarity fromZerotoZeroOrOne, validates the argument againstjson/text(case-insensitive), and exposesIsListTestsJsonOutput(...).Tests
Microsoft.Testing.Platform.UnitTests(full, net462 / net8 / net9)MSTest.Acceptance.IntegrationTests.TestDiscoveryTests(incl. new JSON tests × 2 TFMs)MSTest.Acceptance.IntegrationTests.HelpInfoTestsMicrosoft.Testing.Platform.Acceptance.IntegrationTests.HelpInfoTests*Microsoft.Testing.Platform.Acceptance.IntegrationTests.ExecutionTestsMicrosoft.Testing.Platform.Acceptance.IntegrationTests.TrxTests.\build.cmd -pack -c Debug(warnings-as-errors)Notes
PlatformResources.resxupdated; all 13 XLFs regenerated viadotnet msbuild … /t:UpdateXlf.docs/Changelog-Platform.mdis curated at release time, and v2.2.3 was just cut without anUnreleasedsection.The change went through two rounds of internal expert review before opening. Highlights addressed in review:
Ctrl+Cno longer corrupts the JSON document (cancellation message goes to stderr).PropertyBag's reverse-insertion-order traversal is reversed back to insertion order.propertiesis an array (not object) so duplicate keys are preserved.type.namespaceomitted for the global namespace.--list-tests textaccepted as explicit alias of the default for forward compatibility.