--report-trx with --timeout does not write trx in case of timeout?

[nohwnd]

Discussed in #5142

^{Originally posted by japj February 26, 2025}
I am using MSTest.Sdk 3.8.2 and was trying to combine —report-trx and —timeout when I noticed that the trx file is not being written in case of a timeout expiration.

in my situation, there were several test executed (succesfully), but the test session was canceled and the test run was aborted due to the timeout.

is this expected behavior or a bug by any chance?

[Youssef1313]

NOTE: It's not uncommon in CI scenarios that failing CI is done per TRX. We should make sure that the TRX we will output will fail at least Azure's PublishTestResult, even if all tests there were passing.

[Evangelink]

It seems to be an expected behavior to me. The timeout is global to the platform so if it expires then you stop everything including post actions. There could be a discussion to decide whether it would be good to add a new timeout for only the test execution (if needed).

[Youssef1313]

I'm pushing to 3.10/1.8 for now.

[Youssef1313]

TRX today can either be run in-process as a testhost extension, or out-of-proc as testhost controller extension. Today we take this decision based on whether --crash-dump is used.

If we are running under test host controller, we should be able to write trx in case of timeout.

But we need to make sure that if all tests have passed before the timeout, AzDO will be able to consider the test result as failing. For me, I would say this is going to be a hard ship blocker if AzDO doesn't correctly read TRX.

We could also have --trx-mode and allow in-proc and out-of-proc as values, so user can force out-of-proc mode if they always want to have a valid TRX.
After that, we then need to decide how we want --timeout to behave in test host controller. Should we consider this timeout to be a "testhost-only" timeout that we skip in test host controller to always allow out of process extensions to finalize their work?

[nohwnd]

Personally I don't care that much about the exact definition of --timeout at that point my tests are already hanging for some unreasonable time that is much longer than normal execution of the tests, so it should be at least the test time, but I don't care if it takes 2 minutes more to complete, when the added value of this is giving me as much info about the hang as I can get.

At minimum when I go to azdo tests dashboard I should see that something aborted that was not supposed to abort.

For CI there is also a global timeout that kill the pipeline, on which I would rely when I need to completely stop the action. At least in GH actions there is per step and per job timeout. I think it is valid to assume other CIs have similar protections.

https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idstepstimeout-minutes

For the trx mode let's offer as much functionality as we can in the in-proc mode, which is there to be used as often as possible because it is more efficient.

How much would it cost us to just output a generic "this xxx.dll crashed" trx with one fake test, at the start, and overwrite it with the actual trx at the end?

Do we also write the results progressively? (I don't think so?)

how hard would it be to allocate some space in the trx for the closing tags so it is always valid, even if we crash and burn after first test?

<trx>
 <test>
 <result>My.Dll tests crashed, see the output of the run for more information</result>
 
... insert more info into this buffer until we reach the end of the file


1 MB empty space buffer that we fill












 </test>
</trx>

And if we succeed we shrink the file.

Or is this too naive ? :)