[BUG] Segmentation Fault in ipc_comp_free during IPC3 Fuzz Testing

[tmleman]

Describe the bug
A segmentation fault occurs in the ipc_comp_free function during IPC3 fuzz testing. The issue is caused by a read memory access to a null pointer. This was detected using AddressSanitizer.

To Reproduce

1. Run fuzz testing with the provided corpus.
2. Observe the segmentation fault in the logs.

Reproduction Rate
The issue occurs consistently during fuzz testing.

Expected behavior
The fuzz testing should complete without causing a segmentation fault.

Impact
This issue is a showstopper as it prevents the completion of fuzz testing and affects the stability of the IPC3 configuration.

Environment

1. Branch name and commit hash.
   • SOF: 7650d0a
2. Name of the platform(s) on which the bug is observed.
   • Platform: native_sim

Screenshots or console output

================== Job 1 exited with exit code 1 ============ INFO: Running with entropic power schedule (0xFF, 100). INFO: 11411 files found in ./ipc3_fuzz_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 11411 min: 1b max: 510b total: 1108557b rss: 49Mb
AddressSanitizer:DEADLYSIGNAL
=================================================================
==305587==ERROR: AddressSanitizer: SEGV on unknown address 0x00000000 (pc 0x08186fb0 bp 0xe63fae88 sp 0xe63fae50 T7)
==305587==The signal is caused by a READ memory access.
==305587==Hint: address points to the zero page.
 #0 0x8186fb0 in ipc_comp_free /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c
 #1 0x81783d2 in ipc_glb_tplg_free /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc3/handler.c:1394:8
 #2 0x8177637 in ipc_cmd /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc3/handler.c:1657:9
 #3 0x818b879 in ipc_platform_do_cmd /home/tmleman/work/repos/thesofproject/sof/src/platform/posix/ipc.c:160:2
 #4 0x8185b8c in ipc_do_cmd /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-common.c:361:9
 #5 0x81a934b in task_run /home/tmleman/work/repos/thesofproject/sof/zephyr/include/rtos/task.h:94:9
 #6 0x81a8ebe in edf_work_handler /home/tmleman/work/repos/thesofproject/sof/zephyr/edf_schedule.c:32:16
 #7 0x8260cbf in work_queue_main /home/tmleman/work/repos/thesofproject/zephyr/kernel/work.c:684:3
 #8 0x815aa92 in z_thread_entry /home/tmleman/work/repos/thesofproject/zephyr/lib/os/thread_entry.c:48:2

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c in ipc_comp_free
Thread T7 created by T3 here:
 #0 0x810602c in __interceptor_pthread_create (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x810602c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #1 0x82716e2 in nct_new_thread /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nct.c:476:2
 #2 0x816e397 in posix_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/posix_core_nsi.c:48:9
 #3 0x816ddaf in arch_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/thread.c:55:30
 #4 0x8262e7c in z_setup_new_thread /home/tmleman/work/repos/thesofproject/zephyr/kernel/thread.c:563:2
 #5 0x8263293 in z_impl_k_thread_create /home/tmleman/work/repos/thesofproject/zephyr/kernel/thread.c:658:2
 #6 0x8260777 in k_thread_create /home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/include/generated/zephyr/syscalls/kernel.h:85:9
 #7 0x8260777 in k_work_queue_start /home/tmleman/work/repos/thesofproject/zephyr/kernel/work.c:752:8
 #8 0x81a8c67 in scheduler_init_edf /home/tmleman/work/repos/thesofproject/sof/zephyr/edf_schedule.c:107:2
 #9 0x818d748 in platform_init /home/tmleman/work/repos/thesofproject/sof/src/platform/posix/posix.c:77:2
 #10 0x8177103 in primary_core_init /home/tmleman/work/repos/thesofproject/sof/src/init/init.c:314:6
 #11 0x825cea9 in z_sys_init_run_level /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:371:13
 #12 0x825cea9 in bg_thread_main /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:519:2
 #13 0x815aa92 in z_thread_entry /home/tmleman/work/repos/thesofproject/zephyr/lib/os/thread_entry.c:48:2

Thread T3 created by T2 here:
 #0 0x810602c in __interceptor_pthread_create (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x810602c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #1 0x82716e2 in nct_new_thread /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nct.c:476:2
 #2 0x816e397 in posix_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/posix_core_nsi.c:48:9
 #3 0x816ddaf in arch_new_thread /home/tmleman/work/repos/thesofproject/zephyr/arch/posix/core/thread.c:55:30
 #4 0x8262e7c in z_setup_new_thread /home/tmleman/work/repos/thesofproject/zephyr/kernel/thread.c:563:2
 #5 0x825cd01 in prepare_multithreading /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:672:14
 #6 0x825c76d in z_cstart /home/tmleman/work/repos/thesofproject/zephyr/kernel/init.c:795:24
 #7 0x8128e79 in __asan::AsanThread::ThreadStart(unsigned long long) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8128e79) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)

Thread T2 created by T0 here:
 #0 0x810602c in __interceptor_pthread_create (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x810602c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #1 0x82703ce in nce_boot_cpu /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nce.c:209:2
 #2 0x817526a in posix_boot_cpu /home/tmleman/work/repos/thesofproject/zephyr/soc/native/inf_clock/soc.c:126:2
 #3 0x827e336 in nsif_cpu0_boot /home/tmleman/work/repos/thesofproject/zephyr/boards/native/native_sim/nsi_if.c:24:2
 #4 0x8273942 in nsif_cpun_boot /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nsi_cpun_if.c:16:1
 #5 0x827319b in nsi_cpu_auto_boot /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/nsi_cpu_ctrl.c:38:4
 #6 0x826fb46 in nsi_init /home/tmleman/work/repos/thesofproject/zephyr/scripts/native_simulator/common/src/main.c:87:2
 #7 0x827e4cc in LLVMFuzzerTestOneInput /home/tmleman/work/repos/thesofproject/sof/src/platform/posix/fuzz.c:30:3
 #8 0x8083da3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned int) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8083da3) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #9 0x8084d3d in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile> >&) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8084d3d) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #10 0x8085261 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile> >&) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x8085261) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #11 0x807325c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned int)) (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x807325c) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #12 0x809cc67 in main (/home/tmleman/work/repos/thesofproject/build-fuzz/zephyr/zephyr.exe+0x809cc67) (BuildId: 75cebb6bc8f6d4675a31c691ba1356e4cb016476)
 #13 0xf0c21518 (/lib/i386-linux-gnu/libc.so.6+0x21518) (BuildId: 559bcd138c06a98975421c680491f5666816aef2)

==305587==ABORTING
MS: 1 ChangeBit-; base unit: 3242021c62b0183ca5022cf0394a277dd444e5ee
0x4,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0x0,0xff,0xd9,0x0,0x3,0x30,0x6,0xf6,0x0,0x2c,0x8,0x60,0x8,0x0,0x0,0x0,0x2d,0x2d,0x3c,0x0,0x10,0x30,0x0,0x0,0x29,0x87,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x6,0x0,0x0,0x8,0x0,0xfc,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0xff,0xfe,0x0,0x0,0x3,0x30,0x3,0xf6,0x8,0x60,0x0,0x0,0x0,0x23,0x1,0x0,0x0,0x0,0x60,0x0,0xf6,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x41,0x0,0x4,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0x0,0xff,0xd9,0x0,0x3,0x30,0x6,0x0,0x0,0x1,0x30,0x60,0x8,0x0,0x0,0x0,0x2d,0x2d,0x3a,0x0,0x10,0x30,0x0,0x0,0x29,0x87,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x1,0xf6,0x8,0x0,0xfc,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x17,0x0,0x0,0xff,0xfe,0x0,0x0,0x2,0x30,0x3,0x2f,0x8,0x60,0x0,0x0,0x0,0x23,0x1,0x0,0x0,0x0,0x60,0x0,0xf6,0x2,0x0,0x0,0x0,0x87,0x0,0x0,0x1,0x30,0xff,0xff,0x3,0x0,0x41,0x0,0x0,0x3,0x0,0x0,0x11,0x0, \004\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\027\000\000\000\377\331\000\0030\006\366\000,\010`\010\000\000\000--<\000\0200\000\000)\207\000\000\0010\377\377\003\000\006\000\000\010\000\374\000\000\000\000\000\000\000\027\000\000\377\376\000\000\0030\003\366\010`\000\000\000#\001\000\000\000`\000\366\000\000\0010\377\377\003\000A\000\004\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\027\000\000\000\377\331\000\0030\006\000\000\0010`\010\000\000\000--:\000\0200\000\000)\207\000\000\0010\377\377\003\000\001\366\010\000\374\000\000\000\000\000\000\000\027\000\000\377\376\000\000\0020\003/\010`\000\000\000#\001\000\000\000`\000\366\002\000\000\000\207\000\000\0010\377\377\003\000A\000\000\003\000\000\021\000
artifact_prefix='./'; Test unit written to ./crash-1a8c95ba333361f3d7fd7465442d2d8eeab398df
Base64: BAAAAAAAAAAAAAAAAAAAABcAAAD/2QADMAb2ACwIYAgAAAAtLTwAEDAAACmHAAABMP//AwAGAAAIAPwAAAAAAAAAFwAA//4AAAMwA/YIYAAAACMBAAAAYAD2AAABMP//AwBBAAQAAAAAAAAAAAAAAAAAAAAXAAAA/9kAAzAGAAABMGAIAAAALS06ABAwAAAphwAAATD//wMAAfYIAPwAAAAAAAAAFwAA//4AAAIwAy8IYAAAACMBAAAAYAD2AgAAAIcAAAEw//8DAEEAAAMAABEA

$ hexdump crash-1a8c95ba333361f3d7fd7465442d2d8eeab398df
0000000 0004 0000 0000 0000 0000 0000 0000 0000
0000010 0017 0000 d9ff 0300 0630 00f6 082c 0860
0000020 0000 2d00 3c2d 1000 0030 2900 0087 0100
0000030 ff30 03ff 0600 0000 0008 00fc 0000 0000
0000040 0000 0017 ff00 00fe 0300 0330 08f6 0060
0000050 0000 0123 0000 6000 f600 0000 3001 ffff
0000060 0003 0041 0004 0000 0000 0000 0000 0000
0000070 0000 0000 0017 0000 d9ff 0300 0630 0000
0000080 3001 0860 0000 2d00 3a2d 1000 0030 2900
0000090 0087 0100 ff30 03ff 0100 08f6 fc00 0000
00000a0 0000 0000 1700 0000 feff 0000 3002 2f03
00000b0 6008 0000 2300 0001 0000 0060 02f6 0000
00000c0 8700 0000 3001 ffff 0003 0041 0300 0000
00000d0 0011
00000d2

[lgirdwood]

@dbaluta @bhiregoudar @andyross fyi as impact with your configs

[andyross]

Is this a regression? Odd that it's suddenly popping up.

[andyross]

Also: can you attach that corpus file as a binary (or at least something like a base64 string that I can programmatically convert instead of writing software to do the endianness flipping on all the shorts)? And how was the fuzzer built, and with what config?

[tmleman]

Is this a regression?

The issue might have been hidden due to the fuzzer used in CI having limited code coverage. I am not certain about the coverage achieved by SOF under OSS-Fuzz, but it is possible that the current fuzzer configuration in CI is not exercising all code paths effectively.

can you attach that corpus file

Here is the corpus file encoded as a base64 string:

BAAAAAAAAAAAAAAAAAAAABcAAAD/2QADMAb2ACwIYAgAAAAtLTwAEDAAACmHAAABMP//AwAGAAAIAPwAAAAAAAAAFwAA//4AAAMwA/YIYAAAACMBAAAAYAD2AAABMP//AwBBAAQAAAAAAAAAAAAAAAAAAAAXAAAA/9kAAzAGAAABMGAIAAAALS06ABAwAAAphwAAATD//wMAAfYIAPwAAAAAAAAAFwAA//4AAAIwAy8IYAAAACMBAAAAYAD2AgAAAIcAAAEw//8DAEEAAAMAABEA

And how was the fuzzer built, and with what config?

The fuzzer was built using the script provided in the SOF repository (sof/scripts/fuzz.sh).

System and Tool Versions:

• Operating System: Ubuntu 22.04
• Kernel Version: 6.5.0-25-generic
• CMake Version: 3.22.1
• Python Version: 3.10.12 (used within a virtual environment)
• West Version: 1.2.0
• Board: native_sim
• Toolchain: LLVM (Clang/ld)
• DTC Version: 1.6.1
• Clang Version: 14.0.0
• Clang++ Version: 14.0.0
• GNU LD Version: 2.38
• SOF Commit: 7650d0a

[kv2019i]

@tmleman @andyross @thesofproject/maintainers any input on correct priority for this bug? Should this block v2.12 release?

[tmleman]

any input on correct priority for this bug? Should this block v2.12 release?

I am not sure how to prioritize this bug. However, I believe it is not a blocker for the v2.12 release.

[lgirdwood]

any input on correct priority for this bug? Should this block v2.12 release?

I am not sure how to prioritize this bug. However, I believe it is not a blocker for the v2.12 release.

Will need to fix for v2.12 as this has not root cause and could also impact other use cases and IPC4.

[tmleman]

@kv2019i @andyross I have identified the same problem using a fuzzer with UndefinedBehaviorSanitizer:

INFO: Running with entropic power schedule (0xFF, 100).
INFO: 29455 files found in ./ipc3_corpus
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 29455 min: 1b max: 510b total: 2367995b rss: 32Mb
/home/tmleman/work/repos/thesofproject/sof/zephyr/include/rtos/string.h:43:50: runtime error: addition of unsigned offset to 0x088fbb24 overflowed to 0x088fbb23
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/zephyr/include/rtos/string.h:43:50 in
/home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2: runtime error: member access within null pointer of type 'struct list_item'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2 in
/home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2: runtime error: load of null pointer of type 'struct list_item *'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2 in
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==432370==ERROR: UndefinedBehaviorSanitizer: SEGV on unknown address 0x00000000 (pc 0x080e38a0 bp 0xeb9fcfd8 sp 0xeb9fcfb0 T432404)
==432370==The signal is caused by a READ memory access.
==432370==Hint: address points to the zero page.
 #0 0x80e38a0 in ipc_comp_free /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c

UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c in ipc_comp_free
==432370==ABORTING
MS: 2 InsertRepeatedBytes-InsertByte-; base unit: b4f43ebcf03f40281d674b9e826e0d9be4f3ed4f
0x0,0x0,0x0,0x26,0x0,0xff,0x60,0x87,0x0,0x0,0x2,0x30,0x0,0x53,0x53,0x0,0xff,0x0,0x53,0x0,0x0,0x20,0x30,0x53,0x53,0x5b,0x53,0x53,0x51,0x0,0x0,0x1,0x50,0x53,0x53,0x53,0x0,0x0,0x1,0x30,0x0,0x0,0x0,0x2,0xff,0xff,0xff,0xff,0x2,0xff,0x19,0xff,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x24,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,
\000\000\000&\000\377`\207\000\000\0020\000SS\000\377\000S\000\000 0SS[SSQ\000\000\001PSSS\000\000\0010\000\000\000\002\377\377\377\377\002\377\031\377\000\000\000\000\000\000\000\000\000$\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000
artifact_prefix='./findings/ipc3_2025-01-08_undefined/'; Test unit written to ./findings/ipc3_2025-01-08_undefined/crash-6eb29353a782085538e848200ca661280437a13b
Base64: AAAAJgD/YIcAAAIwAFNTAP8AUwAAIDBTU1tTU1EAAAFQU1NTAAABMAAAAAL/////Av8Z/wAAAAAAAAAAACQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=

The root cause is likely an uninitialized list of sources and sinks inside the IPC component device. In case of an issue within string.h, I have submitted a separate issue: #9768

[kv2019i]

Hmm, not sure how we end up here. I've been trying to repro this with the provided corpus but no luck so far. It would seem commit 403c33f should catch an uninitialized bsink/source list, but this was merged already last year, so can't be this. ipc_get_comp_by_id() should be safe as well as the comp_list is initialized in ipc_init() which is part of posix platform init.

The newer log seems to point to &icd->cd->bsource_list
/home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2: runtime error: member access within null pointer of type 'struct list_item'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /home/tmleman/work/repos/thesofproject/sof/src/ipc/ipc-helper.c:308:2 in

... but yeah, there's a NULL check just before this.