Package:Microsoft.Testing.Extensions.CodeCoverage 18.8.0 (also reproduced on 18.9.0)
Repo:https://github.com/microsoft/codecoverage
Runtime: .NET 8 (8.0.28) and .NET 10 hosts, Linux x64 (Ubuntu container), xunit.v3 3.2.2 on Microsoft.Testing.Platform
Invocation:dotnet test --project <proj> -c Release --coverage --coverage-output-format cobertura
Summary
Static instrumentation prefixes every basic block of every instrumented assembly with an unconditional raw byte store through a static pointer:
ldsfld uint8* Tracker::Begin
ldc.i4 N
addldc.i4.1stind.i1
which the JIT compiles to a single mov byte ptr [reg+idx], 1.
Begin is obtained as
Begin=(byte*)_view.SafeMemoryMappedViewHandle.DangerousGetHandle();
with no DangerousAddRef, over a small file-backed mapping (MemoryMappedFile.CreateFromFile("/tmp/CodeCoverage.<sessionGuid>.<moduleGuid>", FileMode.Open, null, <bufferSize>, MemoryMappedFileAccess.ReadWrite)).
Because the store is unguarded and executes on every basic block, any condition that makes that mapped page unbacked turns the next executed instrumented method into a process-fatal AccessViolationException. The reported stack is whichever user method happened to run at that instant, so the failure is systematically misattributed to innocent user code — including code that provably cannot fault, such as a two-field constructor.
Impact
- A test host dies with
Fatal error. System.AccessViolationException / SIGABRT (exit 134). - The blamed frame is arbitrary and varies run to run, so users chase phantom concurrency and memory-safety bugs in their own code. We spent a substantial investigation before establishing that the faulting instruction was the injected probe rather than our code.
- On CI this reds unrelated pipelines.
Reproducer (self-contained, no proprietary code)
Reproduced deterministically, 3/3, on a synthetic solution containing only ordinary safe C# (no unsafe, no pointers, no P/Invoke, no interop):
- Create a solution with a class library
LibA and an MTP test project referencing it (xunit.v3 + Microsoft.Testing.Extensions.CodeCoverage, global.json with "test": { "runner": "Microsoft.Testing.Platform" }). - Run
dotnet test --project Tests -c Release --coverage --coverage-output-format cobertura once and terminate it while the rewritten assemblies are on disk (the run leaves LibA.dll rewritten plus LibA.dll.orig). This step just makes the instrumented artifact available for direct execution. - Read the buffer path baked into the rewritten library:
strings -el LibA.dll | grep '^/tmp/CodeCoverage\.' - Create that file at the buffer size and confirm the probes are live:
dd if=/dev/zero of=$BUF bs=1 count=<bufferSize> then run the test host directly
(dotnet exec Tests.dll) and observe non-zero bytes appear in $BUF. - Run the test host again and, while it is executing, truncate the buffer:
truncate -s 0 $BUF.
Result (3/3):
Fatal error. System.AccessViolationException: Attempted to read or write protected memory.
at BigRepro.LibA.Widget4.Touch(Int32, Int32, Int32, Int64)
at BigRepro.Tests.Suite34.Calc10Works(Int32)
Fatal error. System.AccessViolationException: ...
at BigRepro.LibA.Widget12..ctor(Double, Double, Int64, Boolean)
at BigRepro.Tests.Suite12+<>c.<RejectsBadConfig>b__1_0()
Fatal error. System.AccessViolationException: ...
at BigRepro.LibA.Widget6.Evict(Int32, Int32, Int32)
at BigRepro.Tests.Suite46+<ConcurrentTouchVsEvict>d__0.MoveNext()
Note the second one: the faulting "user code" is a constructor that only validates arguments and assigns fields.
Specificity control: unlinking and recreating the buffer file (new inode) does not crash the host — the existing mapping keeps the old inode alive. Only losing backing for the mapped page faults. So this is specifically about page backing, not about the file being touched.
Evidence from a naturally-occurring crash (not induced)
We independently captured core dumps of the same failure occurring spontaneously in CI-shaped runs (~0.5-1% of invocations). In those dumps:
- SOS
clrstack -f shows a FaultingExceptionFrame directly above a trivial property getter. - Disassembly at the faulting offset is the probe store, before any user IL executes:
513e: 48 8b 3d cb 4d aa ff mov rdi, [rip-0x55b235] ; Tracker::Begin
5145: b8 c9 00 00 00 mov eax, 0xc9 ; probe index 201
514a: 48 98 cdqe
514c: c6 04 07 01 mov byte ptr [rdi+rax], 1 ; <-- faulting instruction
5150: ... ; user code starts only here
Begin = 0x7824061b5000 (non-null), buffer size 0x99b (2459), probe index 201 (in range), and the core's NT_FILE maps /tmp/CodeCoverage.<session>.<module> at exactly 0x7824061b5000. So: valid pointer, in-range index, mapping present — the page was simply not backed at the instant of the store.- The managed heap verifies clean (
verifyheap: 0 errors in one dump; the only "errors" in
others are InvalidMethodTable at a thread's alloc_limit, i.e. allocation-boundary artifacts). So this is not heap corruption by user code. - The process then FailFasts; dumps carry
ExecutionEngineExceptionHResult 0x80131506.
We also observed, with a 5 ms poller during a normal (non-crashing) run, that buffer files are rewritten in place while mapped (size regressions such as 2459 -> 2458 on 2 of 158 buffer files), which suggests these files are not stable for the lifetime of the mapping.
Plausible real-world causes of lost page backing include truncation/rewrite of the buffer file by another participant, and filesystem exhaustion (a dirty page of a file-backed mapping that cannot be allocated yields SIGBUS; CoreCLR's PAL reports both SIGBUS and SIGSEGV as
EXCEPTION_ACCESS_VIOLATION). Our failures cluster on a small, disk-pressured CI runner.
Suggested remedies
- Hold a reference for the lifetime of the pointer — use
DangerousAddRef/DangerousRelease (or keep and use the SafeMemoryMappedViewHandle directly) so the mapping cannot be released while probes may still execute. - Do not use a shared, world-writable, guessable path.
/tmp/CodeCoverage.<guid>.<guid> is writable by any process on the machine; a truncation by anything else is fatal to unrelated test hosts. Consider an unlinked/anonymous mapping, memfd_create, or a private directory. - Do not let instrumentation faults kill the host. A coverage probe failing should at worst lose coverage data, never terminate the process — and never surface as an
AccessViolationException attributed to user code. Even a one-time validity check with a graceful disable would prevent the misattribution. - If the buffer must be file-backed, preallocate and fsync it, and handle SIGBUS.
Why this is easy to misdiagnose (please consider the diagnostics angle)
Because the injected store is attributed to the enclosing user method, the crash looks exactly like a memory-safety bug in the user's own code. In our case it repeatedly pointed at a lock-protected ConcurrentDictionary wrapper and at trivial constructors. Anything that makes the probe's provenance visible — a marker frame, a distinguishable exception, or a documented signature — would save considerable investigation time.
Package:
Microsoft.Testing.Extensions.CodeCoverage18.8.0 (also reproduced on 18.9.0)Repo:https://github.com/microsoft/codecoverage
Runtime: .NET 8 (8.0.28) and .NET 10 hosts, Linux x64 (Ubuntu container), xunit.v3 3.2.2 on Microsoft.Testing.Platform
Invocation:
dotnet test --project <proj> -c Release --coverage --coverage-output-format coberturaSummary
Static instrumentation prefixes every basic block of every instrumented assembly with an unconditional raw byte store through a static pointer:
which the JIT compiles to a single
mov byte ptr [reg+idx], 1.Beginis obtained aswith no
DangerousAddRef, over a small file-backed mapping (MemoryMappedFile.CreateFromFile("/tmp/CodeCoverage.<sessionGuid>.<moduleGuid>", FileMode.Open, null, <bufferSize>, MemoryMappedFileAccess.ReadWrite)).Because the store is unguarded and executes on every basic block, any condition that makes that mapped page unbacked turns the next executed instrumented method into a process-fatal
AccessViolationException. The reported stack is whichever user method happened to run at that instant, so the failure is systematically misattributed to innocent user code — including code that provably cannot fault, such as a two-field constructor.Impact
Fatal error. System.AccessViolationException/ SIGABRT (exit 134).Reproducer (self-contained, no proprietary code)
Reproduced deterministically, 3/3, on a synthetic solution containing only ordinary safe C# (no
unsafe, no pointers, no P/Invoke, no interop):LibAand an MTP test project referencing it (xunit.v3 +Microsoft.Testing.Extensions.CodeCoverage,global.jsonwith"test": { "runner": "Microsoft.Testing.Platform" }).dotnet test --project Tests -c Release --coverage --coverage-output-format coberturaonce and terminate it while the rewritten assemblies are on disk (the run leavesLibA.dllrewritten plusLibA.dll.orig). This step just makes the instrumented artifact available for direct execution.strings -el LibA.dll | grep '^/tmp/CodeCoverage\.'dd if=/dev/zero of=$BUF bs=1 count=<bufferSize>then run the test host directly(
dotnet exec Tests.dll) and observe non-zero bytes appear in$BUF.truncate -s 0 $BUF.Result (3/3):
Note the second one: the faulting "user code" is a constructor that only validates arguments and assigns fields.
Specificity control: unlinking and recreating the buffer file (new inode) does not crash the host — the existing mapping keeps the old inode alive. Only losing backing for the mapped page faults. So this is specifically about page backing, not about the file being touched.
Evidence from a naturally-occurring crash (not induced)
We independently captured core dumps of the same failure occurring spontaneously in CI-shaped runs (~0.5-1% of invocations). In those dumps:
clrstack -fshows aFaultingExceptionFramedirectly above a trivial property getter.Begin = 0x7824061b5000(non-null), buffer size0x99b(2459), probe index 201 (in range), and the core'sNT_FILEmaps/tmp/CodeCoverage.<session>.<module>at exactly0x7824061b5000. So: valid pointer, in-range index, mapping present — the page was simply not backed at the instant of the store.verifyheap: 0 errors in one dump; the only "errors" inothers are
InvalidMethodTableat a thread'salloc_limit, i.e. allocation-boundary artifacts). So this is not heap corruption by user code.ExecutionEngineExceptionHResult 0x80131506.We also observed, with a 5 ms poller during a normal (non-crashing) run, that buffer files are rewritten in place while mapped (size regressions such as
2459 -> 2458on 2 of 158 buffer files), which suggests these files are not stable for the lifetime of the mapping.Plausible real-world causes of lost page backing include truncation/rewrite of the buffer file by another participant, and filesystem exhaustion (a dirty page of a file-backed mapping that cannot be allocated yields SIGBUS; CoreCLR's PAL reports both SIGBUS and SIGSEGV as
EXCEPTION_ACCESS_VIOLATION). Our failures cluster on a small, disk-pressured CI runner.Suggested remedies
DangerousAddRef/DangerousRelease(or keep and use theSafeMemoryMappedViewHandledirectly) so the mapping cannot be released while probes may still execute./tmp/CodeCoverage.<guid>.<guid>is writable by any process on the machine; a truncation by anything else is fatal to unrelated test hosts. Consider an unlinked/anonymous mapping,memfd_create, or a private directory.AccessViolationExceptionattributed to user code. Even a one-time validity check with a graceful disable would prevent the misattribution.Why this is easy to misdiagnose (please consider the diagnostics angle)
Because the injected store is attributed to the enclosing user method, the crash looks exactly like a memory-safety bug in the user's own code. In our case it repeatedly pointed at a lock-protected
ConcurrentDictionarywrapper and at trivial constructors. Anything that makes the probe's provenance visible — a marker frame, a distinguishable exception, or a documented signature — would save considerable investigation time.