[ci-scan] Known Build Error: Android x86 Mono APP_CRASH in libmonosgen-2.0.so after Android API level bump

[simonrozsival]

Note

This issue was drafted with GitHub Copilot.

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1418805&view=logs&j=f580a640-930a-547e-3e08-155527737d6c&t=462974e4-73a9-512f-970b-7525db7a3891&s=ff05ad62-bb9a-53b6-ce9f-72f329a63e7c
Build error leg or test failing: net11.0-android-Release-x86-Mono_Release-Ubuntu.2204.Amd64.Android.29.Open - Microsoft.Extensions.Primitives.Tests / System.Runtime.Tests

Error Message

{
 "ErrorMessage": ["<<XHARNESS_RESULT_START>>","\"exitCodeName\": \"APP_CRASH\"","\"platform\": \"android\"","\"helixWorkItemId\": \"Microsoft.Extensions.Primitives.Tests\"","workitems/Microsoft.Extensions.Primitives.Tests/console","\"architecture\": \"x86\""],
 "BuildRetry": false,
 "ExcludeConsoleLog": false
}

Notes

This tracks recurring Android x86 Mono APP_CRASH failures in runtime-extra-platforms.

Observed failing work items:

• Microsoft.Extensions.Primitives.Tests
• System.Runtime.Tests

Representative signature:

--------- beginning of crash
04-25 03:59:25.518 5978 6003 F libc : Fatal signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0 in tid 6003 (.NET TP Worker), pid 5978 (rimitives.Tests)
04-25 03:59:25.536 1893 1919 E memtrack: Couldn't load memtrack module
04-25 03:59:25.536 1893 1919 W android.os.Debug: failed to get memory consumption info: -1
04-25 03:59:25.673 6008 6008 I crash_dump32: obtaining output fd from tombstoned, type: kDebuggerdTombstone
04-25 03:59:25.674 1782 1782 I /system/bin/tombstoned: received crash request for pid 6003
04-25 03:59:25.675 6008 6008 I crash_dump32: performing dump of process 5978 (target tid = 6003)
04-25 03:59:25.693 6008 6008 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
04-25 03:59:25.693 6008 6008 F DEBUG : Build fingerprint: 'Android/sdk_phone_x86/generic_x86:10/QSR1.210820.001/7663313:userdebug/test-keys'
04-25 03:59:25.693 6008 6008 F DEBUG : Revision: '0'
04-25 03:59:25.693 6008 6008 F DEBUG : ABI: 'x86'
04-25 03:59:25.694 6008 6008 F DEBUG : Timestamp: 2026-04-25 03:59:25+0000
04-25 03:59:25.694 6008 6008 F DEBUG : pid: 5978, tid: 6003, name: .NET TP Worker >>> net.dot.Microsoft.Extensions.Primitives.Tests <<<
04-25 03:59:25.694 6008 6008 F DEBUG : uid: 10123
04-25 03:59:25.694 6008 6008 F DEBUG : signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0
04-25 03:59:25.694 6008 6008 F DEBUG : eax 00000000 ebx 7b40e5ac ecx 00000010 edx 799b7604
04-25 03:59:25.694 6008 6008 F DEBUG : edi 00000000 esi 78b876fc
04-25 03:59:25.694 6008 6008 F DEBUG : ebp 8eda41f8 esp 78b8767c eip 7b2db927
04-25 03:59:25.765 6008 6008 F DEBUG : 
04-25 03:59:25.765 6008 6008 F DEBUG : backtrace:
04-25 03:59:25.766 6008 6008 F DEBUG : #00 pc 00248927 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.766 6008 6008 F DEBUG : #01 pc 002748d5 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.766 6008 6008 F DEBUG : #02 pc 002747fc /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #03 pc 002860d9 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #04 pc 00287132 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #05 pc 00286cef /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #06 pc 00286d5a /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (mono_class_setup_vtable+26) (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #07 pc 002d9b46 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (mono_class_vtable_checked+454) (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #08 pc 002e184b /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #09 pc 0028e4d8 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (mono_exception_from_token+104) (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #10 pc 00203c54 /data/app/net.dot.Microsoft.Extensions.Primitives.Tests-L1SoeqUitlghwTnk-YwB4g==/lib/x86/libmonosgen-2.0.so (BuildId: ba9fe95f091776b9dfb98426935559db42ca45ca)
04-25 03:59:25.767 6008 6008 F DEBUG : #11 pc 00000fae <anonymous:988d3000>

The crash is present on main and was first observed in the scanned daily range between:

• Last clean checked build: 1394339, commit c835d0fc4f8a
• First failing checked build: 1396313, commit 51cfc7a608e6

Recent main builds and PR build 1418805 show the same failure mode. This does not appear related to PR #124173.

Local build plus Helix execution points to #126838, "Bump Android minimum API level from 21 to 24", as the likely regression cause or trigger:

Experiment	Work item	Helix job	Result
Baseline upstream/main commit 553d88792317e028eff9703301d84fab46ba095d	Microsoft.Extensions.Primitives.Tests	3eb04ceb-819a-4cce-9f86-0913f4944dda	Failed with the same APP_CRASH / libmonosgen-2.0.so signature
Revert #125835	Microsoft.Extensions.Primitives.Tests	452b3530-79bc-47bf-90ff-b168e260413b	Still failed with the same signature
Revert #126838	Microsoft.Extensions.Primitives.Tests	8056938a-ead1-462e-8d2e-f0750bddee52	Passed
Revert #126838	System.Runtime.Tests	ee3751f3-cdcc-4bb6-8027-64710103b9e9	Passed

The #126838-reverted APKs were built locally and submitted to Ubuntu.2204.Amd64.Android.29.Open, matching the failing queue. The first #126838-revert rebuild hit a stale native CMake configuration issue after changing API level from 24 back to 21; removing the stale Android x86 native obj outputs fixed the local build:

rm -rf artifacts/obj/mono/android.x86.Release artifacts/obj/native/net11.0-android-Release-x86 artifacts/bin/mono/android.x86.Release artifacts/bin/native/net11.0-android-Release-x86
./build.sh mono+libs -os android -arch x86 -c Release

No exact existing GitHub issue was found for the combination of Android x86 Mono, these work items, XHarness APP_CRASH exit 80, Fatal signal 11 (SIGSEGV), code 128 (SI_KERNEL), and libmonosgen-2.0.so.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1396313
Error message validated: [Fatal signal 11 \(SIGSEGV\), code 128 \(SI_KERNEL\), fault addr 0x0[^\n]*\n(?:[^\n]*\n){0,80}[^\n]*/data/app/net\.dot\.(?:Microsoft\.Extensions\.Primitives|System\.Runtime)\.Tests-[^\n]*/lib/x86/libmonosgen-2\.0\.so]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 5/14/2026 11:32:39 AM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1396313
Error message validated: [signal 11 (SIGSEGV), code 128 (SI_KERNEL), fault addr 0x0 /lib/x86/libmonosgen-2.0.so >>> net.dot.Microsoft.Extensions.Primitives.Tests <<<]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 5/14/2026 11:51:18 AM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Known Issue Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
 "ErrorMessage": "",
 "ErrorPattern": "",
 "BuildRetry": false,
 "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Known Issue Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
 "ErrorMessage": "",
 "ErrorPattern": "",
 "BuildRetry": false,
 "ExcludeConsoleLog": false
}

[dotnet-policy-service]

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival
See info in area-owners.md if you want to be subscribed.

[simonrozsival]

Note

This comment was drafted with GitHub Copilot.

Additional symbolication for the Microsoft.Extensions.Primitives.Tests tombstone from build 1396313.

Caveat: I could not find the exact CI libmonosgen-2.0.so.dbg for BuildId ba9fe95f091776b9dfb98426935559db42ca45ca in the public build artifacts; those contain log/binlog bundles only. I rebuilt the exact source commit 51cfc7a608e60e2e1d5b8e65c9f0ffe8a322fe1f locally for Android x86 Release. That produced BuildId c7abdd93d87b658c3f41354c05b67c3c9126215d, so direct addr2line was not valid.

The exported frames in the tombstone let us derive a constant local adjustment of +0x2dc0:

crash pc 0x00286d5a = mono_class_setup_vtable+26
local symbol mono_class_setup_vtable = 0x00289b00

crash pc 0x002d9b46 = mono_class_vtable_checked+454
local symbol mono_class_vtable_checked = 0x002dc740

crash pc 0x0028e4d8 = mono_exception_from_token+104
local symbol mono_exception_from_token = 0x00291230

Using local_pc = crash_pc + 0x2dc0, the native stack resolves to:

#00 monoeg_g_hash_table_new
 src/mono/mono/eglib/ghashtable.c:114

#01 collect_implemented_interfaces_aux
 src/mono/mono/metadata/class.c:1908

#02 mono_class_get_implemented_interfaces
 src/mono/mono/metadata/class.c:1934

#03 mono_class_setup_interface_offsets_internal
 src/mono/mono/metadata/class-setup-vtable.c:220

#04 mono_class_setup_vtable_general
 src/mono/mono/metadata/class-setup-vtable.c:1723

#05 mono_class_setup_vtable_full
 src/mono/mono/metadata/class-setup-vtable.c:968

#06 mono_class_setup_vtable
 src/mono/mono/metadata/class-setup-vtable.c:896

#07 mono_class_create_runtime_vtable
 src/mono/mono/metadata/object.c:2153

#08 mono_object_new_handle
 src/mono/mono/metadata/object.c:5369

#09 mono_exception_from_token
 src/mono/mono/metadata/exception.c:133

#10 mono_x86_throw_corlib_exception
 src/mono/mono/mini/exceptions-x86.c:506

#11 <anonymous JIT frame>

The interesting path appears to be: the x86 throw helper tries to create a corlib exception, mono_exception_from_token() allocates the exception object, that forces vtable creation, and the crash occurs while collecting implemented interfaces during interface-offset setup.

[simonrozsival]

Note

This update was generated with GitHub Copilot based on local builds and Helix runs.

Additional narrowing from local Android x86 builds + Helix:

• On the first failing source commit (51cfc7a608e60e2e1d5b8e65c9f0ffe8a322fe1f), changing only the four Android API-level defaults from 24 back to 21 made Microsoft.Extensions.Primitives.Tests pass on Helix.
  • Work item: Microsoft.Extensions.Primitives.Tests
  • Helix job: 57f34cf7-8ec6-4f2e-b8c6-fe5066107b62
  • Result: exit code 0, Tests run: 514 Passed: 514 Inconclusive: 0 Failed: 0 Ignored: 0 Skipped: 0
  • Changed knobs only: AndroidApiLevelMin, native ANDROID_API_LEVEL, AndroidLibraryMinApiLevel, and ApkBuilder.DefaultMinApiLevel.

This makes the Android API-level/native-build configuration a much stronger suspect than just a coincidental PR-level correlation. It still does not prove that the API 21 -> 24 bump is the semantic root cause; it proves that the native build produced by those knobs controls at least the Microsoft.Extensions.Primitives.Tests repro on the first failing source.

Native comparison between the passing API21-only build and an API24 inverse build (#126838 reverted source with only those four knobs set back to 24):

Build	libmonosgen-2.0.so BuildId	stripped size	notable CMake/import difference
API21-only	6fb3d8634935d487d490bffdc9c574c18dc1aabf	3,270,948	ANDROID_PLATFORM=android-21, HAVE_GNU_STRERROR_R empty
API24 inverse	78c69ef3a01b52926a1d5ab1438c513671eafb0c	3,671,028	ANDROID_PLATFORM=android-24, HAVE_GNU_STRERROR_R=1

The API24 build also uniquely imports fortified/Bionic symbols observed in the local comparison (__fread_chk, __fwrite_chk, __gnu_strerror_r, __register_atfork, __write_chk, stderr, stdout), while the API21 build uniquely had __sF, strerror_r, and write in the comparison.

I also built and packaged the inverse APK successfully (#126838 reverted source + API24 knobs; package confirmed ANDROID_PLATFORM=android-24 and apksigner --min-sdk-version 24), but the ad hoc Helix submissions for that inverse payload created jobs that stayed before work-item materialization (details API showed 0 waiting/running/finished work items and the wait endpoint kept returning 503 RetryAfter). So the inverse execution result is not yet known.

[simonrozsival]

Note

This comment was generated with help from GitHub Copilot.

I found a more concrete root-cause lead: the API 21 -> 24 bump changes Android x86 Mono native compilation by removing -mstackrealign.

Evidence:

• NDK r27c explicitly adds -mstackrealign only for x86 API levels < 24:
  • build/cmake/flags.cmake: if(CMAKE_ANDROID_ARCH_ABI STREQUAL x86 AND CMAKE_SYSTEM_VERSION LESS 24) ... -mstackrealign
  • build/cmake/android-legacy.toolchain.cmake: same condition, with comment: x86 devices have stack alignment issues.
• The actual Mono compile commands match that:
  • API21 passing build: --target=i686-none-linux-android21 ... -mstackrealign ... ghashtable.c
  • API24 failing build: --target=i686-none-linux-android24 ... with no -mstackrealign
• The g_hash_table_new codegen changes accordingly:
  • API21 has defensive stack realignment at function entry:
    • push %ebp; mov %esp,%ebp; ... and $-0x10,%esp
  • API24 does not realign:
    • push %ebx; push %esi; sub $0x34,%esp
  • API24 still emits aligned SSE stack accesses (movdqa ... 0x20(%esp), movdqa ... 0x10(%esp)). If the stack entering from Mono x86/JIT exception handling is not 16-byte aligned, this is a plausible direct SIGSEGV mechanism.
• I built a diagnostic API24 Android x86 Mono with -mstackrealign restored for i686.*android. The g_hash_table_new prologue returned to the API21-style realigning form while keeping the API24 target/libc surface:
  • BuildId: 8a2234a1214c34318dbe5de8f4599450cc0ede14
  • Prologue: push %ebp; mov %esp,%ebp; ... and $-0x10,%esp
  • API24 imports still include __register_atfork, __*_chk, stdout/stderr, and __gnu_strerror_r, so this isolates stack realignment from the other API24 deltas.

I packaged the diagnostic Microsoft.Extensions.Primitives.Tests.apk, but Helix is still not materializing ad hoc Android x86 work items: job d72757e4-191c-41de-bf2a-52f67b2b285a was accepted but details show all work item counts at zero and the workitems list is empty. So I do not yet have the pass/fail confirmation from device execution.

Current interpretation: the most likely cause is not Microsoft.Extensions.Primitives or HAVE_GNU_STRERROR_R; it is that Android x86 API24 builds no longer get -mstackrealign, while Mono x86 can enter native runtime helpers from JIT/exception paths with stack alignment that is unsafe for API24 Clang-generated aligned SSE stack operations.