Enable in-proc crash reporting on MacOS + Linux

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991

[mdh1418]

Enable the in-proc crash reporter on Linux and macOS using the existing crash-report settings.

Reporter selection

• DOTNET_DbgEnableMiniDump=1 continues to select createdump.
  • DOTNET_EnableCrashReport=1 adds createdump’s JSON report.
  • DOTNET_EnableCrashReportOnly=1 generates only createdump’s JSON report.
• When DOTNET_DbgEnableMiniDump is disabled, either crash-report setting enables the in-proc reporter.
• Mobile platforms retain their existing in-proc behavior.

This preserves existing desktop and CI configurations while avoiding an additional reporter-selection environment variable.

Fixes: #119945

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Enables the in-proc crash reporter to be built and selected on Linux/macOS, shifting automatic reporter selection into PAL startup while keeping createdump as the default desktop path and preserving mobile in-proc behavior.

Changes:

• Enable FEATURE_INPROC_CRASHREPORT for Linux and macOS in CoreCLR CMake feature selection.
• Add PAL startup selection state (CrashReportMode) and expose it to the VM via PAL_InProcCrashReporterEnabled().
• Broaden Apple-specific in-proc reporter codepaths from mobile-only to all Apple targets (TARGET_APPLE).

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/coreclr/vm/crashreportstackwalker.h	Updates comments to reflect startup-policy-driven crash reporter configuration.
src/coreclr/vm/ceemain.cpp	Gates automatic crash reporter startup (CrashReportConfigure) on PAL-selected in-proc mode.
src/coreclr/pal/src/thread/process.cpp	Adds startup selection state for in-proc vs createdump and changes fatal-signal dispatch behavior.
src/coreclr/pal/src/include/pal/process.h	Updates PAL header documentation for crash reporter selection/initialization timing.
src/coreclr/pal/inc/pal.h	Introduces PAL_InProcCrashReporterEnabled() and updates callback registration docs.
src/coreclr/debug/crashreport/inproccrashreporter.cpp	Expands Apple-specific implementations from iOS/tvOS/MacCatalyst to all Apple targets.
src/coreclr/clrfeatures.cmake	Enables in-proc crash reporting feature compilation for Linux and macOS targets.

Comments suppressed due to low confidence (1)

src/coreclr/pal/src/thread/process.cpp:1868

• When automatic in-proc reporting is selected, the fatal-signal path returns early if the callback hasn't been registered yet. This can silently suppress crash dumps/reports (e.g., if CrashReportConfigure didn't run due to missing EnableCrashReport*, or if in-proc initialization fails and never installs the callback). It should fall back to the createdump path when the callback is null, like the previous behavior did.

 if (g_inProcCrashReporterEnabled)
{
PINPROCCRASHREPORT_CALLBACK callback = g_inProcCrashReportCallback;
if (callback == nullptr)
{

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

[mdh1418]

cc: @rolfbjarne

[rolfbjarne]

cc: @rolfbjarne

This is great! While not a direct fix for #119945, it would serve the same purpose (allowing crash reports for sandboxed apps).

[lateralusX]

LGTM!

[jkotas]

Enable in-proc crash reporting on desktop

Using "desktop" in this PR title is confusing.

"Desktop" is typically WinForms + WPF combo (look for .NET Desktop Runtime under https://dotnet.microsoft.com/en-us/download/dotnet/8.0). Or an acronym for .NET Framework (documented in the glossary https://github.com/dotnet/runtime/blob/989dae08a9b17e8c3bd2a9ce18da656a6589a923/docs/project/glossary.md#net-framework ).

[mdh1418]

/ba-g "The failure is #131517"

[am11]

/ba-g "The failure is #131517"

@mdh1418 GCC failure was related to this PR:

 FAILED: vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o /usr/bin/g++-16 -DBUILDENV_CHECKED=1 -DDEBUG -DDEBUGGING_SUPPORTED -DDISABLE_CONTRACTS -DFEATURE_CACHED_INTERFACE_DISPATCH -DFEATURE_CODE_VERSIONING -DFEATURE_COLD_R2R_CODE -DFEATURE_COLLECTIBLE_TYPES -DFEATURE_COMWRAPPERS -DFEATURE_CORECLR -DFEATURE_CORECLR_FLUSH_INSTRUCTION_CACHE_TO_PROTECT_STUB_READS -DFEATURE_DBGIPC_TRANSPORT_DI -DFEATURE_DBGIPC_TRANSPORT_VM -DFEATURE_DEFAULT_INTERFACES -DFEATURE_DYNAMIC_CODE_COMPILED -DFEATURE_EVENTSOURCE_XPLAT -DFEATURE_EVENT_TRACE -DFEATURE_HIJACK -DFEATURE_INPROC_CRASHREPORT -DFEATURE_INTERPRETER -DFEATURE_JAVAMARSHAL -DFEATURE_MANUALLY_MANAGED_CARD_BUNDLES -DFEATURE_METADATA_UPDATER -DFEATURE_MULTICOREJIT -DFEATURE_MULTITHREADING -DFEATURE_PAL_ANSI -DFEATURE_PERFMAP -DFEATURE_PERFTRACING -DFEATURE_PGO -DFEATURE_PROFAPI_ATTACH_DETACH -DFEATURE_READYTORUN -DFEATURE_REJIT -DFEATURE_REMAP_FUNCTION -DFEATURE_REMOTE_PROC_MEM -DFEATURE_STANDALONE_GC -DFEATURE_SVR_GC -DFEATURE_SYMDIFF -DFEATURE_TIERED_COMPILATION -DFEATURE_USE_ASM_GC_WRITE_BARRIERS -DFEATURE_USE_SOFTWARE_WRITE_WATCH_FOR_GC_HEAP -DFEATURE_VIRTUAL_STUB_DISPATCH -DGC_DESCRIPTOR -DHOST_64BIT -DHOST_AMD64 -DHOST_UNIX -DPROFILING_SUPPORTED -DTARGET_64BIT -DTARGET_AMD64 -DTARGET_LINUX -DTARGET_UNIX -DUNICODE -DUNIX_AMD64_ABI -DUNIX_AMD64_ABI_ITF -DURTBLDENV_FRIENDLY=Checked -DWRITE_BARRIER_CHECK -D_DBG -D_DEBUG -D_FILE_OFFSET_BITS=64 -D_SECURE_SCL=0 -D_TIME_BITS=64 -D_UNICODE -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks -I/__w/1/s/src/coreclr/vm/wks -I/__w/1/s/src/coreclr/vm -I/__w/1/s/src/native -I/__w/1/s/src/native/inc -I/__w/1/s/src/coreclr/pal/prebuilt/inc -I/__w/1/s/artifacts/obj -I/__w/1/s/src/coreclr/pal/inc -I/__w/1/s/src/coreclr/pal/inc/rt -I/__w/1/s/src/coreclr/pal/src/safecrt -I/__w/1/s/src/coreclr/inc -I/__w/1/s/src/coreclr/debug/inc -I/__w/1/s/src/coreclr/debug/inc/amd64 -I/__w/1/s/src/coreclr/debug/inc/dump -I/__w/1/s/src/coreclr/md/inc -I/__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/inc -I/__w/1/s/src/coreclr/hosts/inc -I/__w/1/s/src/coreclr/interpreter/inc -I/__w/1/s/src/coreclr/minipal -I/__w/1/s/src/coreclr/nativeresources -I/__w/1/s/src/coreclr/vm/amd64 -I/__w/1/s/src/coreclr/vm/../interop/inc -I/__w/1/s/src/coreclr/debug/crashreport -I/__w/1/s/src/coreclr/runtime -I/__w/1/s/src/native/libs/System.IO.Compression.Native -I/__w/1/s/src/native/libs/Common -I/__w/1/s/src/coreclr/vm/eventing/eventpipe -std=gnu++17 -fPIC -O2 -g -Wall -fno-omit-frame-pointer -fno-strict-overflow -fno-strict-aliasing -fstack-protector-strong -Werror -Wno-unused-variable -Wno-unused-value -Wno-unused-function -Wno-tautological-compare -Wno-unknown-pragmas -Wimplicit-fallthrough -Wvla -Wno-invalid-offsetof -Wno-unused-but-set-variable -ffp-contract=off -fno-rtti -Wno-uninitialized -Wno-strict-aliasing -Wno-array-bounds -Wno-misleading-indentation -Wno-stringop-overflow -Wno-restrict -Wno-stringop-truncation -Wno-class-memaccess -faligned-new -fsigned-char -fvisibility=hidden -ffunction-sections -fdata-sections -mcx16 -Wno-conversion-null -Wno-pointer-arith -Winvalid-pch -include /__w/1/s/artifacts/obj/coreclr/linux.x64.Checked/vm/wks/CMakeFiles/cee_wks_core.dir/cmake_pch.hxx -MD -MT vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -MF vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o.d -o vm/wks/CMakeFiles/cee_wks_core.dir/__/excep.cpp.o -c /__w/1/s/src/coreclr/vm/excep.cpp
/__w/1/s/src/coreclr/vm/excep.cpp: In function ‘void CrashDumpAndTerminateProcess(UINT)’:
/__w/1/s/src/coreclr/vm/excep.cpp:3490:18: error: comparison of integer expressions of different signedness: ‘UINT’ {aka ‘unsigned int’} and ‘HRESULT’ {aka ‘int’} [-Werror=sign-compare]
3490 | if (exitCode == COR_E_STACKOVERFLOW)
| ^

[EgorBo]

Any idea why this PR could cause a massive regression ^ (624) cc @mdh1418@jkotas@lateralusX

[EgorBo]

AI's theory: Adding crash-report objects shifts libcoreclr.so; JIT code sits near the ±128 MiB BL limit, so common helper calls may start requiring jump stubs. That adds several ns broadly—matching the 600+ unrelated regressions. Not crash-reporter runtime overhead.

[EgorBo]

Minimal repro with perf artifacts: EgorBot/Benchmarks#465

[EgorBo]

Follow-up on my earlier comment — I dug into the EgorBot perf artifacts and the jump-stub theory was wrong. The real cause is TLS.

TL;DR

libcoreclr.so loses its static TLS allocation with this PR, so every TLS access in the runtime — including the allocation fast path — now goes through glibc's _dl_tlsdesc_dynamic instead of _dl_tlsdesc_return. That's a flat few-ns tax on every allocation and every managed↔native transition, which is exactly the shape of the 600+ "unrelated" regressions.

Evidence

1. 9.1% of samples land in ld-linux-aarch64.so.1 in the AFTER build, and 0% in BEFORE.

symbol	before (00fc09ea~1)	after (00fc09ea)
ld-linux-aarch64.so.1 +0x128fc	—	3.52%
ld-linux-aarch64.so.1 +0x128e4	—	3.02%
ld-linux-aarch64.so.1 +0x128e8	—	2.53%

All three have RhpNewFast as their caller in the collapsed stacks.

2. Those offsets are _dl_tlsdesc_dynamic.

I pulled libc6_2.39-*_arm64.deb and disassembled ld-linux-aarch64.so.1:

0x128a0: nop ; _dl_tlsdesc_return <-- STATIC resolver
0x128a4: ldr x0, [x0, #8]
0x128a8: ret
...
0x128c8: nop ; _dl_tlsdesc_dynamic <-- DYNAMIC resolver
0x128cc: stp x1, x2, [sp, #-32]!
0x128d0: stp x3, x4, [sp, #16]
0x128d4: mrs x4, tpidr_el0
0x128d8: ldr x1, [x0, #8] ; td
0x128dc: ldr x0, [x4] ; dtv
0x128e0: ldr x3, [x1, #16] ; td->gen_count
0x128e4: ldr x2, [x0] ; dtv[0].counter <<< HOT
0x128e8: cmp x3, x2 <<< HOT
0x128ec: b.hi <slow>
0x128f0: ldp x2, x3, [x1] ; ti_module, ti_offset
0x128f4: add x0, x0, x2, lsl #4
0x128f8: ldr x0, [x0] ; dtv[ti_module].pointer.val
0x128fc: cmn x0, #1 ; == TLS_DTV_UNALLOCATED? <<< HOT
0x12900: b.eq <slow>
0x12904: sub x3, x3, x4
0x12908: add x0, x0, x3
0x1290c: ldp x3, x4, [sp, #16]
0x12910: ldp x1, x2, [sp], #32
0x12914: ret

Byte-for-byte glibc/sysdeps/aarch64/dl-tlsdesc.S. The hot addresses are its fast path.

3. It's visible directly in the allocation helper. From the .asm artifacts, RhpNewFast / RhNewString (numbers are local period %):

 before after
1.31 adrp x0, :tlsdesc:... 0.78 adrp x0, :tlsdesc:...
2.59 ldr x3, [x0, #912] 2.95 ldr x3, [x0, #976]
1.42 add x0, x0, #0x390 2.40 add x0, x0, #0x3d0
3.44 → blr x3 2.49 → blr x3 <-- TLSDESC call
1.42 mrs x3, tpidr_el0 5.70 mrs x3, tpidr_el0
1.58 add x3, x3, x0 1.55 add x3, x3, x0
28.75 ldr x12, [x3, #8] 3.95 ldr x12, [x3, #8]
12.53 ldr x13, [x3] 43.39 ldr x13, [x3]

The blr x3 resolves to a 2-instruction leaf before and a ~17-instruction call with four dependent loads after. And note ldr x13, [x3] (the alloc context load) going 12.53% → 43.39% — with dynamic TLS the block is a separate malloc'd chunk instead of sitting next to tpidr_el0, so it starts missing cache.

4. perf stat agrees. Both runs are 6.00s wall / ~20.3G cycles. Normalizing by throughput (17.15/20.82 = 0.824):

• instructions/op +14%
• cache-references/op +32%
• IPC 4.11 → 3.87

5. The JIT'd managed code is identical. I diffed both .asm artifacts with addresses masked — zero structural difference. No jump stubs, no codegen change, no inlining change.

Why this PR triggers it

corerundlopens libcoreclr.so. glibc will only place a dlopen'd module's TLS in the static TLS block if it fits the optional static TLS surplus — glibc.rtld.optional_static_tls, default 512 bytes (_dl_try_allocate_static_tls(map, optional=true) in elf/dl-reloc.c).

libcoreclr sits right on that cliff: t_ThreadStatics (ThreadLocalData, ~96B) + t_runtime_thread_locals (ee_alloc_context) + ~20 other thread_locals. Flipping FEATURE_INPROC_CRASHREPORT on for Linux/OSX adds 8 new .cpp files to the libcoreclr link; that appears to have tipped the PT_TLS segment (size and/or alignment) past the budget. Once glibc gives up on static TLS for the module, every TLSDESC reloc in it resolves to _dl_tlsdesc_dynamic, permanently.

Note that none of the new files declare a thread_local themselves — this is a whole-module property, which is why the regression looks completely unrelated to the feature.

Secondary effect: IsValidTLSResolver() in src/coreclr/vm/threadstatics.cpp explicitly pattern-matches _dl_tlsdesc_return and disables the JIT's inline thread-static access when it sees the dynamic resolver. This benchmark doesn't use thread statics so it isn't visible here, but thread-static-heavy benchmarks should regress harder.

How to confirm

GLIBC_TUNABLES=glibc.rtld.optional_static_tls=4096 ./corerun <bench>

If the regression disappears, that's it. Also worth diffing between the two builds:

readelf -lW libcoreclr.so | grep -A1 'TLS'# compare MemSiz / Align

I'd expect AFTER to cross 512 bytes (or bump alignment enough that the rounded-up consumption does).

Possible fixes

• Shrink libcoreclr's TLS footprint so it stays under the surplus (fragile — we're one thread_local away from this happening again).
• Have the host reserve static TLS / raise the tunable before loading libcoreclr.
• Longer term, make this non-silent: a startup check that detects the dynamic resolver and warns, so a several-percent-across-the-board regression doesn't ship as a mystery again.

[lateralusX]

In-Proc Crash Reporter uses minipal_get_current_thread_id that could inline and have a static thread local tid. If it inlines into different compile units, then we would duplicate that TLS variable in every compile unit that uses minipal_get_current_thread_id. For the In-Proc Crash Reporter we could use minipal_get_current_thread_id_no_cache that won't cache anything in TLS, but maybe we should make sure we only have one TLS variable for minipal_get_current_thread_id, putting it into its own compilation unit inside minipal and extern declare it in the header.

[lateralusX]

#131991