[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path - #9507

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec
Jun 29, 2026
Merged

[perf-improver] perf: avoid List(TestResult) allocation in RunTestMethodAsync non-data-driven fast path#9507
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Goal and Rationale

For the vast majority of test executions — non-data-driven tests — RunTestMethodAsync was paying three unnecessary costs on every call:

  1. Two attribute scans: TryExecuteDataSourceBasedTestsAsync scans all cached method attributes for DataSourceAttribute; if that returns false, TryExecuteFoldedDataDrivenTestsAsync scans them again for ITestDataSource implementors.
  2. List<TestResult> allocation: a List<TestResult> is created even though only ExecuteTestAsync will be called (which already returns TestResult[]).
  3. Spread allocation: return [.. results] converts the list back to an array, allocating a second short-lived TestResult[].

In a typical suite with 1,000 non-data-driven tests, this adds ~80 KB of short-lived allocations and ~2,000 extra attribute-scan iterations per run.

Approach

New IsDataDrivenTest() private method: one pass over GetCustomAttributesCached that checks attribute is DataSourceAttribute or UTF.ITestDataSource — combining both scans into one.

Fast path (new): if _test.DataType != DynamicDataType.ITestDataSourceandIsDataDrivenTest() returns false, the method:

  • Calls ExecuteTestAsync directly
  • Returns its TestResult[] without allocating a List<TestResult>
  • Performs one attribute-cache scan instead of two

Slow path (unchanged): all data-driven tests continue through the existing List<TestResult> path.

GetAggregateOutcome signature: widened from List<TestResult> to IReadOnlyList<TestResult> so both the fast-path TestResult[] and slow-path List<TestResult> can call it (arrays and lists both implement IReadOnlyList<T>).

Performance Evidence

Per non-data-driven test, the change eliminates approximately:

  • 1 List<TestResult> object (~56 bytes: 24-byte header + internal capacity-4 backing array)
  • 1 TestResult[] spread allocation (~24 bytes, length 1)
  • 1 redundant attribute-cache enumeration (second TryExecute... scan avoided)

For a 1,000-test non-data-driven suite: ~80 KB fewer short-lived allocations and ~2,000 fewer attribute-scan iterations per run.

No local SDK is available in this CI agent; build/test verification is delegated to CI.

Trade-offs

  • Very slight code duplication: the "set display name → execute → set outcome" block appears in both the fast path and the slow path's else branch. The else branch is an edge case (test has DataSourceAttribute but multiple such attributes cause TryExecuteDataSourceBasedTestsAsync to skip it), so it is never exercised on common paths.
  • No behavioral changes: the fast path is only reached when both DataType != ITestDataSource and IsDataDrivenTest() is false, ensuring data-driven tests are unaffected.

Test Status

Delegated to CI. Existing unit tests in TestMethodRunnerTests.cs exercise both the new fast path (non-data-driven) and the unchanged slow path (data-driven) code paths.

Reproducibility

# Build and run adapter unit tests
./build.sh -test

🤖 Automated content by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Perf Improver workflow. · 1.7K AIC · ⌖ 26.2 AIC · ⊞ 57.7K · [◷]( · )

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/perf-improver.md@main

…a-driven fast path
For non-data-driven tests (the common case), RunTestMethodAsync previously:
- Allocated a List<TestResult> upfront
- Scanned all cached method attributes twice (TryExecuteDataSourceBasedTestsAsync
then TryExecuteFoldedDataDrivenTestsAsync) before reaching the execute call
- Spread the list back into a TestResult[] via [.. results]
The new fast path:
- Performs a single combined attribute scan (IsDataDrivenTest)
- Skips List allocation entirely, returning the TestResult[] from ExecuteTestAsync directly
- Saves ~3 heap allocations per non-data-driven test execution
The existing slow path (with List<TestResult>) is preserved for data-driven tests.
GetAggregateOutcome parameter is widened to IReadOnlyList<TestResult> so both paths
can call it (arrays and lists both implement IReadOnlyList<T>).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CopilotAI review requested due to automatic review settings June 29, 2026 15:07
@EvangelinkAmaury Levé (Evangelink) added area/performance Runtime / build performance / efficiency. type/automation Created or maintained by an agentic workflow. labels Jun 29, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR optimizes MSTest adapter execution for the common non-data-driven case by avoiding redundant attribute scans and short-lived allocations in TestMethodRunner.RunTestMethodAsync.

Changes:

  • Added a private IsDataDrivenTest() helper to detect DataSourceAttribute / ITestDataSource in a single cached-attribute pass.
  • Introduced a non-data-driven fast path that calls ExecuteTestAsync directly and returns its TestResult[] without allocating/spreading a List<TestResult>.
  • Widened GetAggregateOutcome from List<TestResult> to IReadOnlyList<TestResult> to support both arrays and lists.
Show a summary per file
FileDescription
src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.csAdds a non-data-driven fast path and consolidates attribute scanning to reduce allocations and iterations.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Low

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Dismissed
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review June 29, 2026 17:13
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

🤖 Automated review by GitHub Copilot. Posted via a maintainer's GitHub token, so it appears under their account — the account owner did not write or approve this content personally. Generated by the Expert Code Review workflow. To request a follow-up action, reply by tagging @copilot directly.

Review Summary

Reviewed src/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs against all 22 applicable dimensions. The optimization is well-reasoned and algorithmically sound for the common case, but there is one correctness concern worth addressing before merge.


Verdict Table

#DimensionStatusNotes
1Algorithmic Correctness⚠️SEE INLINEFast path bypasses results.Count == 0 guard (detail below)
2Threading & Concurrency✅ LGTMNo new shared-state access; all reads are already synchronized by callers
3Security & IPC Contract Safety✅ LGTMNo security-relevant code touched
4Public API & Binary Compatibility✅ LGTMAll changes are private; IReadOnlyList<T> widening is private-only
5Performance & Allocations✅ LGTMCorrectly eliminates List<TestResult> + spread allocation on the hot path
6Cross-TFM Compatibility✅ LGTMNo TFM-conditional APIs introduced
7Resource & IDisposable Management✅ LGTMNo disposables affected
8Defensive Coding at Boundaries⚠️ Same as #1See inline comment on line 134
9Localization & Resources✅ LGTMNo new user-facing strings added
10Test Isolation✅ LGTMNo static mutation introduced
11Assertion Quality✅ LGTMTests use AwesomeAssertions (project convention)
12Flakiness Patterns✅ LGTMNo timing/ordering sensitivity introduced
13Test Completeness & Coverage✅ LGTMExisting non-data-driven tests exercise the fast path; data-driven tests cover the slow path
14Data-Driven Test CoverageN/ANo new data-driven tests
15Code Structure & Simplification✅ LGTMDuplication acknowledged in PR; else branch is an edge case only
16Naming & Conventions✅ LGTMIsDataDrivenTest() name is clear and consistent with codebase style
17Documentation Accuracy✅ LGTMXML doc and inline comments are accurate
18Analyzer & Code Fix QualityN/ANo src/Analyzers/ changes
19IPC Wire CompatibilityN/ANo serialization code changed
20Build Infrastructure & Dependencies✅ LGTMNo build changes
21Scope & PR Discipline✅ LGTMFocused, single-file, single-concern change
22PowerShell Scripting HygieneN/ANo .ps1 files changed

The One Correctness Issue (inline comment, line 134)

DynamicDataType has only two values (None, ITestDataSource), and IsDataDrivenTest() correctly mirrors the exact attribute checks performed by TryExecuteDataSourceBasedTestsAsync (looks for DataSourceAttribute) and TryExecuteFoldedDataDrivenTestsAsync (looks for UTF.ITestDataSource). The fast-path gate is logically equivalent to the old else branch and the algorithm is correct.

The single gap: the slow path has a defensive guard at lines 190-199 that ensures RunTestMethodAsync never returns [] — because ExecuteAsync's finally block unconditionally dereferences result![0], an empty return would throw IndexOutOfRangeException rather than surfacing a clean error. The fast path omits this guard.

For all practical purposes (standard TestMethodAttribute always returns ≥ 1 result) this is benign. But a custom executor returning TestResult[0]{} would now crash on the fast path where it previously received a graceful error result. The inline comment explains the fix options.

No other issues found across the remaining 20 applicable dimensions.

Comment threadsrc/Adapter/MSTestAdapter.PlatformServices/Execution/TestMethodRunner.cs Outdated
@EvangelinkAmaury Levé (Evangelink) added the state/needs-review Awaiting review from the team. label Jun 29, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 9cbe2da into mainJun 29, 2026
67 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the perf-assist/skip-list-alloc-non-data-driven-8e762aaad2e45aec branch June 29, 2026 19:42
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/performanceRuntime / build performance / efficiency.state/needs-reviewAwaiting review from the team.type/automationCreated or maintained by an agentic workflow.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Evangelink@0101