Tests are sent back as in-progress and passed

[Youssef1313]

Test updates are batched by count and time, and if test is in progress and finishes in the same batch we send the same test case info twice in the same batch.

Test updates can simply discard in-progress update for any test case for which they already have a result.

Repro: Just run any test.

 {
 "jsonrpc": "2.0",
 "method": "testing/testUpdates/tests",
 "params": {
 "runId": "d336342f-705a-4171-9b44-80f58d2957e3",
 "changes": [
 {
 "node": {
 "uid": "Microsoft.NET.TestAnywhere.XUnit.UnitTest1.TestWithCategory",
 "display-name": "Microsoft.NET.TestAnywhere.XUnit.UnitTest1.TestWithCategory",
 "traits": [
 {
 "A": "UnitTest"
 }
 ],
 "vstest.original-executor-uri": "executor://xunit/VsTestRunner2/net",
 "vstest.TestCase.Id": "94afc2e0-ebf9-cbb1-3115-7751ede4fefc",
 "vstest.TestCase.FullyQualifiedName": "Microsoft.NET.TestAnywhere.XUnit.UnitTest1.TestWithCategory",
 "node-type": "action",
 "execution-state": "in-progress"
 },
 "parent": null
 },
 {
 "node": {
 "uid": "Microsoft.NET.TestAnywhere.XUnit.UnitTest1.TestWithCategory",
 "display-name": "Microsoft.NET.TestAnywhere.XUnit.UnitTest1.TestWithCategory",
 "time.start-utc": "2023-10-31T15:06:14.7410985+00:00",
 "time.stop-utc": "2023-10-31T15:06:14.7410985+00:00",
 "time.duration-ms": 4,
 "traits": [
 {
 "A": "UnitTest"
 }
 ],
 "vstest.original-executor-uri": "executor://xunit/VsTestRunner2/net",
 "vstest.TestCase.Id": "94afc2e0-ebf9-cbb1-3115-7751ede4fefc",
 "vstest.TestCase.FullyQualifiedName": "Microsoft.NET.TestAnywhere.XUnit.UnitTest1.TestWithCategory",
 "node-type": "action",
 "execution-state": "passed"
 },
 "parent": null
 }
 ]
 }
}

Originally posted by Jakub Jareš (@nohwnd) in https://github.com/microsoft/testanywhere/issues/1509

[Evangelink]

I don't think you will work on this one soon but when you do plan it, please start by collecting the perf impact of the current solution. It feels to me a rather complex batching for a relatively small benefit.

[nohwnd]

the goal of this ask is to simply not send the "execution-state": "in-progress" status when there is a final "execution-state": in the same batch. That does not seem very complicated, and saves 1 object serialization, and 1 update on the receiving side.

[Evangelink]

Looking at the current implementation in TestNodeStateChangeAggregator (TestNodeStateChangeAggregator.cs), the fix @nohwnd describes is straightforward. BuildAggregatedChange() currently does:

public TestNodeStateChangedEventArgs BuildAggregatedChange()
 => new(RunId, [.. _stateChanges]);

The deduplication would filter out any in-progress update whose UID already has a terminal state in the same batch:

public TestNodeStateChangedEventArgs BuildAggregatedChange()
{
 // Collect UIDs that have a terminal state in this batch.
 HashSet<string>? terminalUids = null;
 foreach (TestNodeUpdateMessage msg in _stateChanges)
 {
 if (msg.TestNode.Properties.SingleOrDefault<TestNodeStateProperty>() is { } state
 && state is not InProgressTestNodeStateProperty)
 {
 (terminalUids ??= []).Add(msg.TestNode.Uid.Value);
 }
 }

 if (terminalUids is null)
 return new(RunId, [.. _stateChanges]);

 // Suppress redundant in-progress updates for tests that finished in the same batch.
 TestNodeUpdateMessage[] filtered = [..
 _stateChanges.Where(m =>
 m.TestNode.Properties.SingleOrDefault<TestNodeStateProperty>() is not InProgressTestNodeStateProperty
 || !terminalUids.Contains(m.TestNode.Uid.Value))];
 return new(RunId, filtered);
}

Analytical measurements (no in-batch dedup today):

Scenario	In-progress updates suppressed	Bytes saved per batch
100 fast tests (all complete in 1 batch)	100	~50 KB
1 000 fast tests	1 000	~500 KB
Slow tests (batch spans multiple flushes)	0	0 (correct)

Each in-progress TestNodeUpdateMessage JSON node is roughly 500 bytes (uid, display-name, traits, vstest properties, execution-state: in-progress). For a fast suite of 1 000 tests, this is ~500 KB of JSON serialization and RPC transfer eliminated per run — plus one fewer state update that IDEs (VS, VS Code) need to process per test.

The fix only applies when both states land in the same batch, so slow tests that straddle two flush cycles are unaffected. The fix is opt-in-safe: the HashSet path is only entered when terminalUids is non-null (i.e., at least one test finished in this batch).

The complexity @Evangelink mentioned is proportional to batch size, which is already bounded by the existing flush policy. Happy to implement this if the team is ready to move forward.

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Efficiency Improver workflow. · 2K AIC · ⌖ 38 AIC · [◷]( · ◷)

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/efficiency-improver.md@main