Add combinatorial test data support

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.

[Andrew Arnott (AArnott)]

Ports the current exhaustive combinatorial test-data functionality from AArnott/Xunit.Combinatorial into MSTest.TestFramework, allowing MSTest users to generate Cartesian products from inferred or explicitly supplied parameter values.

• Adds value, range, random, member, and class parameter providers.
• Supports exact and wildcard test-case exclusions, exhaustive generation, permutations, and fluent data construction.
• Intentionally omits the pairwise attribute and algorithm.
• Adds focused API coverage and executable MSTest examples that exercise discovery and execution through the real test runner.
• Records that Andrew Arnott contributed this implementation under the repository's MIT License.

Related issue: N/A

[Copilot]

Pull request overview

Adds exhaustive combinatorial test-data generation to MSTest, including providers, exclusions, generators, and runner-level examples.

Changes:

• Adds inferred, explicit, range, random, member, and class data providers.
• Adds Cartesian products, permutations, exclusions, and fluent construction.
• Adds API baselines, tests, examples, and third-party attribution.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
THIRD-PARTY-NOTICES.TXT	Records implementation attribution.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs	Tests value, range, and random providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialMemberDataAttributeTests.cs	Tests member and class providers.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialGenerationTests.cs	Tests generators and fluent builder.
test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialDataAttributeTests.cs	Tests inference, exclusions, and data rows.
test/UnitTests/MSTest.SelfRealExamples.UnitTests/CombinatorialDataTests.cs	Exercises features through MSTest.
src/TestFramework/TestFramework/PublicAPI/PublicAPI.Unshipped.txt	Tracks new public APIs.
src/TestFramework/TestFramework/InternalAPI/InternalAPI.Unshipped.txt	Tracks new internal APIs.
src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs	Resolves parameter candidate values.
src/TestFramework/TestFramework/Attributes/DataSource/ICombinatorialValuesProvider.cs	Defines the provider contract.
src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs	Implements exact and wildcard exclusions.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialValuesAttribute.cs	Supplies explicit values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs	Adds fluent data construction.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs	Generates combinations and permutations.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs	Generates integer ranges.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs	Generates unique random values.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs	Reads values from static members.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs	Integrates generation with MSTest.
src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs	Reads values from source classes.
src/TestFramework/TestFramework/Attributes/DataSource/AnyDataValue.cs	Defines the exclusion wildcard sentinel.

Suppressed comments (2)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:62

• Computing to - from in int overflows for valid cross-domain ranges. For example, (int.MinValue, int.MaxValue, 1) calculates a count of zero and returns no values. Widen the distance and step arithmetic before calculating the array size and elements.

 int count = ((to - from) / step) + 1;
Values = new object[count];
for (int i = 0; i < count; i++)
{
Values[i] = from + (i * step);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:103

• These loops rely on unsigned addition/subtraction crossing the bound, but wraparound can make them repeat indefinitely. For example, (uint.MaxValue - 1, uint.MaxValue, 2u) wraps immediately to zero and the ascending condition remains true. Stop based on the remaining distance before performing the next step.

 for (uint i = from; i <= to; i += step)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

🔍 Build Failure Analysis

Summary — The build fails to compile on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance) with identical CS0104 ambiguous-reference errors in test/IntegrationTests/MSTest.Acceptance.IntegrationTests.

Root cause: CombinatorialData/ICombinatorialValuesProvider name collision with the Combinatorial.MSTest NuGet package

This PR adds a new, built-in implementation of combinatorial test data support directly to MSTest (src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs, ICombinatorialValuesProvider.cs, etc., all in namespace Microsoft.VisualStudio.TestTools.UnitTesting). However, test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj still has:

<PackageReferenceInclude="Combinatorial.MSTest" />

and several acceptance-test files under that project still do using Combinatorial.MSTest;, which also exposes types named CombinatorialDataAttribute and ICombinatorialValuesProvider. Because these test files also implicitly see Microsoft.VisualStudio.TestTools.UnitTesting (via MSTest usings), the compiler now finds two same-named types in scope and reports CS0104.

Affected files / errors (identical across all 5 legs)

• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/InconclusiveTests.cs:31 — CS0104: 'CombinatorialData' is ambiguous between 'Combinatorial.MSTest.CombinatorialDataAttribute' and 'Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute'
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/DotnetTestCliTests.cs:17 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/TestFilterProviderRegistrationTests.cs:58 — same
• test/IntegrationTests/MSTest.Acceptance.IntegrationTests/RunnerTests.cs:20,47,81,111 — same
• test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/Helpers/AcceptanceTestBase.cs:394,409 — CS0104: 'ICombinatorialValuesProvider' is ambiguous between 'Combinatorial.MSTest.ICombinatorialValuesProvider' and 'Microsoft.VisualStudio.TestTools.UnitTesting.ICombinatorialValuesProvider'

Proposed fix

None of the above files are touched by this PR's diff, so no inline suggestion can be attached to them, and this run's push_to_pull_request_branch output is not available, so no automated fix commit can be appended — this needs a manual follow-up commit. Two viable approaches:

1. Preferred, given the PR's intent (replacing Combinatorial.MSTest with a native implementation): remove the <PackageReference Include="Combinatorial.MSTest" /> from MSTest.Acceptance.IntegrationTests.csproj, drop the using Combinatorial.MSTest; lines from InconclusiveTests.cs, DotnetTestCliTests.cs, TestFilterProviderRegistrationTests.cs, RunnerTests.cs, and AcceptanceTestBase.cs, and confirm the new Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute/ICombinatorialValuesProvider types provide equivalent behavior for these acceptance tests (e.g. [MetadataModeValues] implementing ICombinatorialValuesProvider).
2. Alternatively, if the two implementations are meant to coexist for now, disambiguate with fully-qualified type names (Combinatorial.MSTest.CombinatorialDataAttribute / Microsoft.VisualStudio.TestTools.UnitTesting.CombinatorialDataAttribute) or a using alias at each call site — more mechanical but leaves duplicate functionality in place.

Note on verification: the GitHub MCP server returned an integrity-policy filter when reading PR #10896's metadata directly, so I could not re-confirm the current head.sha/merge_commit_sha against this run's values before posting. This comment cites file paths/line numbers only (no diff-line inline suggestions), so it isn't affected by a stale diff mapping, but please confirm the PR hasn't moved since this analysis was generated.

---

Build overview

• Build outcome: failure on all 5 legs (Linux Release, macOS Release, Windows Release, Windows Debug, Windows application-model acceptance)
• Failing project: test/IntegrationTests/MSTest.Acceptance.IntegrationTests/MSTest.Acceptance.IntegrationTests.csproj
• Failing target: CoreCompile (Csc task)
• Every leg reports the identical set of CS0104 errors — deterministic, not a flake.

All MSBuild errors (8 distinct, ×5 legs)

Code	Project	File:Line	Message
CS0104	MSTest.Acceptance.IntegrationTests	InconclusiveTests.cs:31	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:409	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	AcceptanceTestBase.cs:394	ambiguous ICombinatorialValuesProvider
CS0104	MSTest.Acceptance.IntegrationTests	DotnetTestCliTests.cs:17	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	TestFilterProviderRegistrationTests.cs:58	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:20	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:47	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:81	ambiguous CombinatorialData
CS0104	MSTest.Acceptance.IntegrationTests	RunnerTests.cs:111	ambiguous CombinatorialData

---

_{🤖 Generated by the Build Failure Analysis workflow using binlog-mcp · commit b7fb099}

🤖 Automated content by GitHub Copilot. Generated by the Build Failure Analysis workflow. · auto · 92.3 AIC · ⌖ 2.05 AIC · ⊞ 13.3K · [◷]( · ◷)

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Suppressed comments (5)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:11

• All registered Where predicates receive the same mutable row array, so an earlier predicate can change the values observed by later predicates. Make this parameter read-only (for example, ReadOnlySpan<object?>, as in the source API) so constraints cannot interfere with one another.

public delegate bool CombinatorialTheoryDataPredicate(object?[] values);

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:58

• Enumeration occurs inside the constructor-error catch. If a lazy source throws from GetEnumerator/MoveNext, the exception is reported as “Failed to create an instance” with constructor advice even though activation succeeded. Restrict the catch to Activator.CreateInstance and enumerate afterward.

 return values.Cast<object[]>().SelectMany(row => row).ToArray();
}
catch (Exception ex)

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:124

• This selection is reflection-order-dependent when multiple public static overloads accept the supplied arguments. For example, both GetValues(object) and GetValues(string) match a string argument, so either overload can be invoked. Apply deterministic overload resolution (prefer the most specific/exact match) or reject ambiguous matches.

 .FirstOrDefault(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:149

• As with property lookup, this stops before validating visibility and staticness. A derived instance field with the same name hides a valid public static field on a base class and makes the documented source unresolvable. Continue the base-type search until an eligible field is found.

 fieldInfo = reflectionType.GetRuntimeField(MemberName);
if (fieldInfo is not null)
{
break;
}

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:185

• A null argument is valid for Nullable<T>, but this branch rejects every value type. For example, arguments "key", null cannot bind to a member method GetValues(string key, int? selector). Treat nullable value types like reference types here.

 else if (parameters[i].ParameterType.IsValueType)
{
return false;
}

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:17

• Different provider types can legally be applied to the same parameter (for example, both CombinatorialValuesAttribute and CombinatorialRangeAttribute), because AllowMultiple = false only prevents duplicates of one attribute type. SingleOrDefault() then aborts discovery with the generic “Sequence contains more than one element” message. Detect this case explicitly and report which parameter has conflicting providers so users can correct the declaration.

 ICombinatorialValuesProvider? valuesSource = parameter.GetCustomAttributes()
.OfType<ICombinatorialValuesProvider>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:48

• The MemberType preservation annotation does not flow through this merged local into the unannotated Get*Accessor(Type, ...) parameters, and those helpers call DeclaredProperties, DeclaredMethods, and DeclaredFields while also walking unannotated BaseType values. Since this project enables AOT analysis, these reflection calls can produce IL2070 warnings, and trimmed member sources are not guaranteed to survive. Preserve the annotation through the helper parameters and handle the DeclaringType fallback/base traversal using the rooted pattern in DynamicDataOperations.

This issue also appears on line 123 of the same file.

 Type? type = MemberType ?? parameter.Member.DeclaringType;

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:127

• Exclude open generic methods from the compatible set. A parameterless GetValues<T>() currently passes this filter, but MethodInfo.Invoke cannot invoke it without closing its type parameters, so data discovery fails with a late-bound reflection exception; it can also prevent lookup from continuing to a valid inherited source. This matches the existing generic-method guard in DynamicDataOperations.cs:116.

 .Where(method =>
method.Name == MemberName
&& method.IsPublic
&& method.IsStatic
&& ParameterTypesCompatible(method.GetParameters(), Arguments))

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• Class-data values are materialized once and the same object instances are reused in every Cartesian row. For a mutable class-data value combined with a bool, both generated tests receive the same instance, so one test can affect (or race with) the other; the member-data path already refreshes values per test case to avoid this. Retain the source type/arguments and recreate its values when materializing each generated row, with a mutable-value regression test.

 => _values = GetValues(valuesSourceType, arguments);

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:32

• This reflection scan runs once per generated row and parameter. GetCustomAttributes() constructs every parameter attribute each time, so a 100-value range crossed with another dimension repeatedly rebuilds and allocates that 100-value range; class-data attributes even recreate and enumerate their source only to be discarded. Capture the initially resolved provider (or whether it is member data) while collecting candidate values, then reuse that metadata when materializing rows.

 CombinatorialMemberDataAttribute? memberData = parameter.GetCustomAttributes()
.OfType<CombinatorialMemberDataAttribute>()
.SingleOrDefault();

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/UnitTests/TestFramework.UnitTests/Attributes/CombinatorialValueAttributeTests.cs:77

• All seeded cases in this suite use 42 and only compare two instances configured identically, so an implementation that ignores Seed and always uses new Random(42) would still pass. Add a second configured seed and assert that it produces a different sequence to cover the public Seed setting itself.

 Seed = 42,

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (4)

Previously missed (4) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTheoryDataBuilder.cs:17

• The implicit parameterless constructor is part of the newly tracked public API, but it cannot carry XML documentation. Declare it explicitly so the complete public surface follows the repository's documented-API convention.

public sealed class CombinatorialTheoryDataBuilder

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:33

• This new reflected member-source path carries explicit trimming annotations and suppressions, but the executable coverage only runs managed builds. The existing NativeAOT acceptance asset exercises DynamicData (MSTest.Acceptance.IntegrationTests/NativeAotTests.cs:91-101), so add a comparable combinatorial case with an explicit MemberType; otherwise a linker regression that removes the source member will not be detected.

 [DynamicallyAccessedMembers(DynamicDataOperations.RequiredMemberTypes)]
public Type? MemberType { get; set; }

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialDataAttribute.cs:13

• This class exposes an implicit public constructor (tracked in PublicAPI.Unshipped.txt), but that constructor has no XML documentation. Public data-source attributes such as DataRowAttribute declare and document their parameterless constructor explicitly; please do the same for this new public API.

public class CombinatorialDataAttribute : Attribute, ITestDataSource

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialClassDataAttribute.cs:23

• The PublicConstructors contract is intended to make reflective class activation survive trimming, but no new test publishes and runs a NativeAOT asset using CombinatorialClassDataAttribute. Add such a case (preferably with constructor arguments) so constructor and enumeration preservation are verified rather than only compiled.

 public CombinatorialClassDataAttribute(
[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicConstructors)] Type valuesSourceType,
params object[]? arguments)

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 2 comments.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Internal/CombinatorialValuesUtilities.cs:23

• This diagnostic is hard-coded in English, so it bypasses TestFramework's localization path. The existing data-source diagnostics use FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170) backed by Resources/FrameworkMessages.resx; move this and the other new diagnostics in this file into that resource and regenerate the XLF files.

 $"Parameter '{parameter.Name}' on '{parameter.Member.Name}' has multiple combinatorial value providers: {string.Join(", ", valueSources.Select(provider => provider.GetType().Name))}. Apply exactly one attribute that implements {nameof(ICombinatorialValuesProvider)}.",

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:40

• This new user-facing validation message bypasses TestFramework's localized FrameworkMessages resources. Please add resource entries for the diagnostics in this file and regenerate the FrameworkMessages XLF files, consistent with the existing data-source errors in DynamicDataOperations.cs:151-170.

 throw new ArgumentException(
$"The number of arguments in {nameof(ExcludeTestCaseAttribute)} must match the number of test method parameters.",
nameof(testMethod));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialTestCaseGenerator.cs:115

• This public API's validation message is hard-coded in English, bypassing TestFramework's FrameworkMessages localization convention. Add it to Resources/FrameworkMessages.resx, consume the generated resource property here, and regenerate the XLF files as is done for existing data-source diagnostics in DynamicDataOperations.cs:151-170.

 throw new ArgumentOutOfRangeException(nameof(dimensionSizes), dimensions[i], "Dimension sizes cannot be negative.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRangeAttribute.cs:26

• The range validation diagnostics are newly hard-coded in English, so localized MSTest installations cannot translate them. Use entries in Resources/FrameworkMessages.resx and regenerate the XLF files, following the established data-source pattern in DynamicDataOperations.cs:151-170.

 if ((long)from + count - 1 > int.MaxValue)
{
throw new ArgumentOutOfRangeException(nameof(count), "The range exceeds the maximum integer value.");

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialRandomDataAttribute.cs:57

• These random-data configuration errors are hard-coded in English even though TestFramework routes public data-source diagnostics through FrameworkMessages resources (for example, DynamicDataOperations.cs:151-170). Move all new diagnostics in this method to Resources/FrameworkMessages.resx and regenerate the XLF files.

 if (Count < 1)
{
throw new InvalidOperationException(string.Format(CultureInfo.CurrentCulture, "{0} must be positive.", nameof(Count)));

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:63

• The member-provider diagnostics are hard-coded in English, unlike the existing TestFramework data-source errors backed by Resources/FrameworkMessages.resx (DynamicDataOperations.cs:151-170). Move the new messages in this file to FrameworkMessages and regenerate the XLF files.

 throw new ArgumentException(
$"Could not find public static member (property, field, or method) named '{MemberName}' on {type.FullName}{parameterText}.");

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Resources/FrameworkMessages.resx:551

• The new diagnostic is missing the article “the”; as emitted, it reads “Expected to have same array length as …”. Change it to “Expected to have the same array length as …” and regenerate the XLF files from this resource.

 <data name="CombinatorialArrayLengthMismatch" xml:space="preserve">
<value>Expected to have same array length as {0}.</value>
<comment>{0} is the name of the array being compared.</comment>

src/TestFramework/TestFramework/Attributes/DataSource/CombinatorialMemberDataAttribute.cs:293

• This rejects an IEnumerable<T> member for a T? test parameter because non-null nullable values are boxed as T, making typeof(T?).IsAssignableFrom(typeof(T)) false. The same file already handles this representation for method arguments; apply that nullable-underlying compatibility here too so valid member data is not rejected.

 if (!parameterInfo.ParameterType.GetTypeInfo().IsAssignableFrom(enumeratedType))

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 2 comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/TestFramework/TestFramework/Attributes/DataSource/ExcludeTestCaseAttribute.cs:18

• [ExcludeTestCase(null)] binds the single null attribute argument as a null params array, so this constructor throws instead of excluding the null-valued case. Handle this like DataRowAttribute/CombinatorialValuesAttribute: accept a nullable params array, interpret null as [null], and update the public API baseline.

 public ExcludeTestCaseAttribute(params object?[] arguments)
=> Arguments = arguments ?? throw new ArgumentNullException(nameof(arguments));

[Copilot]

Pull request overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated no new comments.