Assert failure: Consistency check failed: AV in clr at this callstack

[VincentBu]

Failed in: runtime-coreclr gcstress-extra 20260620.1

Failed tests:

coreclr windows x64 Checked gcstress0xc_jitminopts_heapverify1 @ Windows.10.Amd64.Open
- baseservices/exceptions/OutOfMemoryException/OutOfMemoryException/OutOfMemoryException.cmd

Error message:

Return code: 1
Raw output file: C:hwACA30992wA6220922uploadsOutOfMemoryExceptionOutOfMemoryExceptionoutput.txt
Raw output:
BEGIN EXECUTION
 "C:hwACA30992pcorerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false" -p "System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true" OutOfMemoryException.dll 
Testing OOM with small allocations...
Subprocess exit code: -1073740286
Subprocess stderr: 
Assert failure(PID 13368 [0x00003438], Thread: 13384 [0x3448]): Consistency check failed: AV in clr at this callstack:
------
CORECLR! WKS::gc_heap::init_heap_segment + 0x44 (0x00007ffe`1675adf4)
CORECLR! WKS::gc_heap::make_heap_segment + 0xBF (0x00007ffe`1675b7af)
CORECLR! WKS::gc_heap::get_free_region + 0x1C7 (0x00007ffe`1675aa07)
CORECLR! WKS::gc_heap::get_new_region + 0x14 (0x00007ffe`16752394)
CORECLR! WKS::gc_heap::soh_try_fit + 0x6F8 (0x00007ffe`1674f428)
CORECLR! WKS::gc_heap::allocate_soh + 0x16E (0x00007ffe`1674bd9e)
CORECLR! WKS::gc_heap::allocate_more_space + 0x1D0 (0x00007ffe`1674bc20)
CORECLR! WKS::GCHeap::StressHeap + 0x337 (0x00007ffe`16748c47)
CORECLR! MethodDesc::DoPrestub + 0x584 (0x00007ffe`16a396c4)
CORECLR! PreStubWorker + 0x667 (0x00007ffe`16a450b7)
CORECLR! ThePreStub + 0x55 (0x00007ffe`16ede8a5)
<no module>! <no symbol> + 0x0 (0x00007ffd`b7492fd5)
<no module>! <no symbol> + 0x0 (0x000002b9`820afc20)
SYSTEM.PRIVATE.CORELIB! <no symbol> + 0x0 (0x00007ffe`13dc0000)
<no module>! <no symbol> + 0x0 (0x00000000`0180a000)
-----
.AV on tid=0x3448 (13384), cxr=000000C958988C00, exr=000000C9589890F0

FAILED: false

CORECLR! CHECK::Trigger + 0x23D (0x00007ffe`1658344d)
CORECLR! CLRVectoredExceptionHandlerPhase3 + 0x514 (0x00007ffe`168ccb34)
CORECLR! CLRVectoredExceptionHandlerPhase2 + 0xBB (0x00007ffe`168cc36b)
CORECLR! CLRVectoredExceptionHandler + 0x834 (0x00007ffe`168cc284)
CORECLR! CLRVectoredExceptionHandlerShim + 0x1B7 (0x00007ffe`168ccda7)
NTDLL! RtlInitializeCriticalSection + 0x190 (0x00007ffe`412b7120)
NTDLL! RtlImageNtHeaderEx + 0x17B (0x00007ffe`41284a2b)
NTDLL! KiUserExceptionDispatcher + 0x3A (0x00007ffe`412fa04a)
CORECLR! WKS::gc_heap::init_heap_segment + 0x44 (0x00007ffe`1675adf4)
CORECLR! WKS::gc_heap::make_heap_segment + 0xBF (0x00007ffe`1675b7af)
 File: D:�_work1ssrccoreclr�mexcep.cpp:6139
 Image: C:hwACA30992pcorerun.exe


Expected OOM diagnostic token not found in subprocess stderr.
Expected: 100
Actual: -1
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:
Set up CORE_ROOT and run.
> C:hwACA30992wA6220922e�aseservicesexceptions�aseservices-exceptions../OutOfMemoryException/OutOfMemoryException/OutOfMemoryException.cmd

Stack trace:

 at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/arcade/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 135
 at Xunit.Assert.True(Boolean condition, String userMessage) in /_/src/arcade/src/Microsoft.DotNet.XUnitAssert/src/BooleanAsserts.cs:line 117
 at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String assemblyPath, String testPathPrefix)
 at Program.<<Main>$>g__TestExecutor137|0_138(StreamWriter tempLogSw, StreamWriter statsCsvSw, <>c__DisplayClass0_0&)

Kusto query result:

Queued	Pipeline	Pipeline Configuration	OS	Arch	Test	Outcome	Properties	Build Reason
2026-06-13 19:10:25.416000+00:00	runtime-coreclr outerloop	Checked	azurelinux.3.amd64.open.rt	x64	baseservices/exceptions/OutOfMemoryException/OutOfMemoryException/OutOfMemoryException.cmd	Failed	{"AzurePipelinesTestRunId":"40446120","DefinitionName":"runtime-coreclr outerloop","System.JobId":"958a2a9a-ade5-50b3-3ce6-ea69efbcce9c","architecture":"x64","System.JobDisplayName":"coreclr Pri1 Runtime Tests Run linux_musl x64 checked","BuildId":"1462917","CollectionUri":"https://dev.azure.com/dnceng-public/","System.StageName":"Build","System.JobAttempt":"1","System.PhaseDisplayName":"coreclr Pri1 Runtime Tests Run linux_musl x64 checked","BuildNumber":"20260613.2","DefinitionId":"108","System.StageDisplayName":"Build","System.JobName":"__default","configuration":"Checked","System.StageAttempt":"1","Project":"public","operatingSystem":"(Alpine.323.Amd64.Open)AzureLinux.3.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.23-helix-amd64@sha256:daedc4ed84012104bed0fbcfe8dc92191253bcc5d38490ce5e999427f98853a0","System.PhaseName":"run_test_p1__linux_musl_x64_checked","Reason":"Schedule","System.PhaseAttempt":"1"}	Schedule
2026-06-09 19:17:54.040000+00:00	runtime-coreclr outerloop	Checked	azurelinux.3.arm64.open	arm64	baseservices/exceptions/OutOfMemoryException/OutOfMemoryException/OutOfMemoryException.cmd	Failed	{"configuration":"Checked","System.StageDisplayName":"Build","System.PhaseAttempt":"1","System.StageAttempt":"1","operatingSystem":"(Alpine.323.Arm64.Open)AzureLinux.3.Arm64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.23-helix-arm64v8@sha256:33d25e02f1ffcdbb2a9912ca04feea1f7248686326b4d123d80f09e38650a2fe","System.PhaseName":"run_test_p1__linux_musl_arm64_checked","BuildId":"1456375","System.StageName":"Build","DefinitionId":"108","System.JobName":"__default","CollectionUri":"https://dev.azure.com/dnceng-public/","BuildNumber":"20260609.6","Reason":"Schedule","AzurePipelinesTestRunId":"40300402","Project":"public","DefinitionName":"runtime-coreclr outerloop","System.PhaseDisplayName":"coreclr Pri1 Runtime Tests Run linux_musl arm64 checked","architecture":"arm64","System.JobDisplayName":"coreclr Pri1 Runtime Tests Run linux_musl arm64 checked","System.JobId":"7c87b8fc-cd86-5445-5ca8-4814cf8d81d6","System.JobAttempt":"1"}	Schedule

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @anicka-net, @dotnet/gc
See info in area-owners.md if you want to be subscribed.

[mangod9]

Root cause: AV in gc_heap::init_heap_segment under GCStress + GCHeapHardLimit (OOM)

Summary

This is a latent GC bug in the lazy bookkeeping-commit logic, newly surfaced by the
baseservices/exceptions/OutOfMemoryException test (added in #125311, 2026-06-08; first
failures 2026-06-09). Under a GCStress run, the last page of the seg_mapping_table is
reserved but never committed. When a tight GCHeapHardLimit lets the heap fill to the very
top of its reserved range, the GC allocates a region whose heap_segment metadata lives in
that uncommitted page; init_heap_segment writes the metadata and takes an AV, which the
checked-build vectored handler reports as "Consistency check failed: AV in clr".

The defect is platform-independent (seen on Windows x64, linux-musl x64, linux-arm64) and is
not a recent regression in the GC — the buggy logic is pre-existing; the new test is just
the first to drive a hard-limited heap to genuine OOM.

CORECLR! WKS::gc_heap::init_heap_segment + 0x44 <-- mov qword ptr [rdi+40h], rax (write to seg metadata)
CORECLR! WKS::gc_heap::make_heap_segment + 0xBF
CORECLR! WKS::gc_heap::get_free_region + 0x1C7
CORECLR! WKS::gc_heap::get_new_region + 0x14
CORECLR! WKS::gc_heap::soh_try_fit + 0x6F8
CORECLR! WKS::gc_heap::allocate_soh + 0x16E
CORECLR! WKS::gc_heap::allocate_more_space + 0x1D0
CORECLR! WKS::GCHeap::StressHeap + 0x337
CORECLR! MethodDesc::DoPrestub + 0x584

In init_heap_segment, the write to [seg+0x28] (flags) succeeds but [seg+0x40] faults — a
page-boundary signature: the region's heap_segment entry straddles the committed/uncommitted
boundary of the seg_mapping_table.

Why the new test exposes it (the causal chain)

In region builds, each region's heap_segment is stored inside the seg_mapping_table
(struct seg_mapping { heap_segment region_info; }), which is committed lazily by
on_used_changed -> inplace_commit_card_table. The commit layout is computed in
get_card_table_commit_layout, and seg_mapping is the last element committed there.

The page-alignment that protects the end of seg_mapping comes entirely from the next
bookkeeping element, mark_array, whose size is non-zero only when concurrent GC is usable:

// get_card_table_element_sizes
#ifdef BACKGROUND_GC
 if (gc_can_use_concurrent)
 sizes[mark_array_element] = size_mark_array_of (start, end); // else stays 0
#endif

// get_card_table_element_layout -- the page-align is gated on the element being non-empty:
if ((element != total_bookkeeping_elements) && (sizes[element] != 0))
 layout[element] = ALIGN_UP(layout[element], alignment[element]); // mark_array align = OS_PAGE_SIZE

// get_card_table_commit_layout -- last committed element (seg_mapping) is capped:
commit_end = align_on_page(required_end); // round UP (good)
commit_end = min (commit_end, align_lower_page(bookkeeping_start + layout[i + 1])); // cap, round DOWN

GCStress turns concurrent GC off automatically (the issue is even labeled GCStress):

// eeconfig.cpp -- when GCStress is on and gcConcurrent was not explicitly forced:
// "let's turn off concurrent GC since it makes objects move less"
iGCconcurrent = g_IGCconcurrent = 0;

→ gc_can_use_concurrent = false (init.cpp) → sizes[mark_array] == 0.

With mark_array empty, the ALIGN_UP is skipped, so
layout[mark_array] == layout[seg_mapping] + sizes[seg_mapping] — i.e. the seg_mapping
table ends mid-page. The commit cap align_lower_page(layout[mark_array]) then rounds
down below the table's true end, leaving its final partial page uncommitted (it is
still reserved, because the reservation size layout[total_bookkeeping_elements] is rounded
up to page granularity — hence an AV rather than a different fault).

Full chain:

DOTNET_GCStress=0xC (gcConcurrent not forced)
 -> concurrent GC disabled (eeconfig.cpp)
 -> gc_can_use_concurrent = false (init.cpp)
 -> sizes[mark_array_element] = 0 (get_card_table_element_sizes)
 -> layout[mark_array] NOT page-aligned (ALIGN_UP skipped in get_card_table_element_layout)
 -> seg_mapping commit capped at align_lower_page(layout[mark_array]) (get_card_table_commit_layout)
 -> final page of seg_mapping_table reserved but never committed
 + DOTNET_GCHeapHardLimit=0x2000000 (32 MB) + allocate-to-OOM
 -> a top-of-range region's heap_segment entry lands in that uncommitted page
 -> init_heap_segment writes it -> AV

The test reliably provides the two missing ingredients: a tiny hard limit (so the heap can
actually fill to the top of the reserved bookkeeping range) and relentless pinned/large
allocations to OOM. Normal apps escape this because concurrent GC is usually on (which adds the
page padding) and heaps rarely fill the reserved bookkeeping to the last page.

Proposed fix (candidate)

Restore the invariant that the commit logic relies on — the element following seg_mapping
(the last in-line-committed element) must always start on a page boundary, even when it is
empty — so that seg_mapping's tail page is always covered:

// get_card_table_element_layout
 for (int element = brick_table_element; element <= total_bookkeeping_elements; element++)
 {
 layout[element] = layout[element - 1] + sizes[element - 1];
- if ((element != total_bookkeeping_elements) && (sizes[element] != 0))
+ if (element != total_bookkeeping_elements)
 layout[element] = ALIGN_UP(layout[element], alignment[element]);
 }

Rationale / safety:

• This changes behavior only for empty elements with a large (page) alignment requirement —
  i.e. mark_array when concurrent GC is off. Empty elements with small natural alignment
  (card_bundle, software_write_watch, ...) gain at most a few bytes of padding.
• When concurrent GC is on, sizes[mark_array] != 0 already, so the alignment was already
  applied — no behavior change, no regression.
• It costs at most one extra (now-committed) page of bookkeeping and matches the documented
  intent already stated for mark_array ("make sure pages for mark array never overlap with
  pages in the seg mapping table").

Alternative (commit-side) fix, if changing the reserved layout is undesirable: in
get_card_table_commit_layout, do not truncate the last committed element below its own
page-rounded end — e.g. for i == seg_mapping_table_element, use align_on_page(required_end)
rather than capping with align_lower_page(layout[i + 1]), since nothing else commits those
tail pages when mark_array is empty.

Notes on diagnosis / repro

• The crashing process set DOTNET_DbgEnableMiniDump=0 (the test does this for its OOM
  subprocess), so no crash dump for the AV exists — the only dumps in the work item belong
  to unrelated tests (collecteddelegate, unhandledmissingdependency). This analysis is from
  the symbolicated console stack + disassembly of the matching checked build
  (commit e329cd6b) + source.
• To capture a confirming dump, re-run with DOTNET_DbgEnableMiniDump=1 and inspect
  bookkeeping_covered_committed vs the faulting seg address; the boundary should fall inside
  the final seg_mapping_table page.

Build analyzed: runtime-coreclr gcstress-extra 20260620.1 (buildId 1474066), windows x64
Checked, commit e329cd6b56bc7b78dcf561568eacab4838e565df, config
gcstress0xc_jitminopts_heapverify1.

[AndyAyersMS]

Test introduced by #125311 @eduardo-vp