Summary
The only way to currently customize the .trx filename when using MTP with the Microsoft.Testing.Extensions.TrxReport extension is by using the much more general purpose TestingPlatformCommandLineArgumentsMSBuild property.
I believe forcing the use of this single property makes for a more convoluted configuration experience and creates maintenance/flexibility issues for consumers.
A dedicated, more specific property should be provided that only concerns itself with the format of the trx file and nothing else.
Background and Motivation
We integrated .trx-based test reports in our GitHub workflows this week and it was a pain to get things to how we wanted. The current defaults for the file name are, in my opinion, awful, so we wanted to change them to something that would be more readable to the outside.
We used the dorny/test-reporter GitHub action to parse the .trx files from each of our projects and generate the test report. The tool relies on the filename to produce the various sections in its report (which by itself makes sense).
We made it work by configuring the TestingPlatformCommandLineArguments property like this:
<TestingPlatformCommandLineArguments>--report-trx --report-trx-filename $(MSBuildProjectName).trx</TestingPlatformCommandLineArguments>
Which worked, but with some caveats.
The main issue is that this basically forces us to now always output a trx report, even when we don't want to. Because the way these properties work (as cmdline arguments) it is not possible to only specify --report-trx-filename, otherwise dotnet test will crash with an error saying that --report-trx-filename must be used in conjunction with --report-trx.
From a CLI tool's perspective, this is very reasonable, but because the CLI argument list is the only way to customize those values, it now becomes impossible to just change the format of the trx filenames without having to introduce the extra parameters and without forcing the entire command to always run.
Rather, what we really wanted, was to be able to customize just the format of the filename without touching anything else, and we wanted it to apply only when actually requesting a trx report, which we were trying to setup only in our GitHub workflows, not locally.
Yes, it is possible to add a MSBuildCondition to that property with something like:
<TestingPlatformCommandLineArgumentsCondition="'$(GITHUB_ACTIONS)' == 'true'">--report-trx --report-trx-filename $(MSBuildProjectName).trx</TestingPlatformCommandLineArguments>
But at the end of the day, we decided against that as it was a bit too obscure. We opted for the less-than-ideal approach of "always generate the trx whenever running dotnet test" even though our need was only to do it on server builds.
The fact that it is all cmdline-based also prevents us from attempting something like this, which would be a much cleaner way of handling this (pseudo-code):
<TestingPlatformCommandLineArgumentsCondition="'$(cmdline_args).Include('--report-trx')">--report-trx-filename $(MSBuildProjectName).trx</TestingPlatformCommandLineArguments>Because msbuild does not have access to the passed-in cmdline arguments to dotnet test.
We think this would be significantly simpler if there was a dedicated MSBuild property that only concerned itself with the default format of the trx filenames, so that we could set only that, like:
<TrxFileNameFormat>$(MSBuildProjectName)</TrxFileNameFormat>
And then, whenever --report-trx was used, it would just honor this value automatically, so we could be more explicit on the build server using:
dotnet test -- --report-trx
And avoid the unnecessary process from just happening locally for everyone.
Proposed Feature
I propose that a dedicated MSBuild property be created to customize just the format of the filename for .trx files created by Microsoft.Testing.Extensions.TrxReport.
Suggested property name: TrxFileNameFormat
The commandline value, if specified, would still have precedence over this variable, since it is the more direct/explicit setting.
Alternative Designs
I don't see any other approaches that would be as clean as a dedicated MSBuild property for this. This approach allows using the full breadth of options that MSBuild provides including project-specific properties (such as the one we used, $(MSBuildProjectName)) to customize the format in a generic fashion especially in conjunction with a localized Directory.Build.props file.
Such customization is not possible directly at the commandline when we are running dotnet test because that is usually fired in a much broader context (we are calling it with no solution/project arguments so it grabs the default master solution of the entire repo).
At the same time, the existing TestingPlatformCommandLineArguments also doesn't allow us to cleanly override individual aspects of the generation such as the trx filename without forcing us into a whole other set of decisions.
I think this is preferred to other options such as the one described here:
Although I could see both working in conjunction as well as they feel like independent enhancements.
Summary
The only way to currently customize the
.trxfilename when using MTP with theMicrosoft.Testing.Extensions.TrxReportextension is by using the much more general purposeTestingPlatformCommandLineArgumentsMSBuildproperty.I believe forcing the use of this single property makes for a more convoluted configuration experience and creates maintenance/flexibility issues for consumers.
A dedicated, more specific property should be provided that only concerns itself with the format of the
trxfile and nothing else.Background and Motivation
We integrated
.trx-based test reports in our GitHub workflows this week and it was a pain to get things to how we wanted. The current defaults for the file name are, in my opinion, awful, so we wanted to change them to something that would be more readable to the outside.We used the dorny/test-reporter GitHub action to parse the
.trxfiles from each of our projects and generate the test report. The tool relies on the filename to produce the various sections in its report (which by itself makes sense).We made it work by configuring the
TestingPlatformCommandLineArgumentsproperty like this:Which worked, but with some caveats.
The main issue is that this basically forces us to now always output a
trxreport, even when we don't want to. Because the way these properties work (as cmdline arguments) it is not possible to only specify--report-trx-filename, otherwisedotnet testwill crash with an error saying that--report-trx-filenamemust be used in conjunction with--report-trx.From a CLI tool's perspective, this is very reasonable, but because the CLI argument list is the only way to customize those values, it now becomes impossible to just change the format of the
trxfilenames without having to introduce the extra parameters and without forcing the entire command to always run.Rather, what we really wanted, was to be able to customize just the format of the filename without touching anything else, and we wanted it to apply only when actually requesting a trx report, which we were trying to setup only in our GitHub workflows, not locally.
Yes, it is possible to add a
MSBuildConditionto that property with something like:But at the end of the day, we decided against that as it was a bit too obscure. We opted for the less-than-ideal approach of "always generate the trx whenever running
dotnet test" even though our need was only to do it on server builds.The fact that it is all cmdline-based also prevents us from attempting something like this, which would be a much cleaner way of handling this (pseudo-code):
Because msbuild does not have access to the passed-in cmdline arguments to
dotnet test.We think this would be significantly simpler if there was a dedicated
MSBuildproperty that only concerned itself with the default format of thetrxfilenames, so that we could set only that, like:And then, whenever
--report-trxwas used, it would just honor this value automatically, so we could be more explicit on the build server using:dotnet test -- --report-trxAnd avoid the unnecessary process from just happening locally for everyone.
Proposed Feature
I propose that a dedicated
MSBuildproperty be created to customize just the format of the filename for.trxfiles created byMicrosoft.Testing.Extensions.TrxReport.Suggested property name:
TrxFileNameFormatThe commandline value, if specified, would still have precedence over this variable, since it is the more direct/explicit setting.
Alternative Designs
I don't see any other approaches that would be as clean as a dedicated
MSBuildproperty for this. This approach allows using the full breadth of options thatMSBuildprovides including project-specific properties (such as the one we used,$(MSBuildProjectName)) to customize the format in a generic fashion especially in conjunction with a localizedDirectory.Build.propsfile.Such customization is not possible directly at the commandline when we are running
dotnet testbecause that is usually fired in a much broader context (we are calling it with no solution/project arguments so it grabs the default master solution of the entire repo).At the same time, the existing
TestingPlatformCommandLineArgumentsalso doesn't allow us to cleanly override individual aspects of the generation such as thetrxfilename without forcing us into a whole other set of decisions.I think this is preferred to other options such as the one described here:
--report-trx-filename#5364Although I could see both working in conjunction as well as they feel like independent enhancements.