app: zephyr: add CONFIG_ASSERT debug overlay

[kv2019i]

Enable CONFIG_ASSERT=y in SOF debug overlay.

Also add a commented out list of additional useful Zephyr debugging
tools to the overlay. These incur a higher runtime cost, so are left
disabled by default. These are all debug tools that have been tested to
work with SOF and found useful.

Dependencies for merging (SOF):

• dai-zephyr: check max block count supported by DMA #6823
• dai-zephyr: handle NULL case when calling dai_config_get() #6897
• dai-zephyr: handle dai_config_get() errors properly #6899
• mixin-mixout: handle congestion corner case without assert #6900
• Convert the mixin module to use the module adapter #6836 (OR module_adapter: fix buffer consumption logic in simple_copy mode #6909 )
• dai-zephyr: fixes to dma_stop() usage #7110 (needed for Zephyr baseline update)
• Zephyr baseline update (to bring in all fixes in below fix) -> [DNM] west.yml: upgrade Zephyr to 0c0d73721ed #7125
• rootcause assert hit in mixin_mixout/mixin_mixout.c:368 (https://sof-ci.01.org/sofpr/PR6530/build4058/devicetest/index.html)

Dependencies for merging (Zephyr):

• drivers: dai: intel: dmic: don't use assert for error handling zephyrproject-rtos/zephyr#53394
• intel_adsp: ace: power: fix build error with asserts enabled zephyrproject-rtos/zephyr#53395
• mtl: bugfix: tlb driver should allow remapping all unused pages, including one at UNUSED_L2_START_ALIGNED address zephyrproject-rtos/zephyr#54278
• to merge PR54278, solution is needed for regression observed in [UPDATE ZEPHYR] zephyr: dai: Check error code before xrun report #6965 that is now blocking Zephyr updates, fix in dai: fix xrun handling #7044
• dma: dw: allow to stop in DW_DMA_SUSPENDED state zephyrproject-rtos/zephyr#54900

[kv2019i]

FYI @nashif@teburd -- CONFIG_SPIN_LOCK_TIME_LIMIT is pretty cool. I had to set the default threshold quite high due to the busywaiting we do in zephyr/drivers/dai/intel/ssp/ (a known limitation it seems), but it was cool to see this picked up by SPIN_LOGK_TIME_LIMIT.

[teburd]

FYI @nashif@teburd -- CONFIG_SPIN_LOCK_TIME_LIMIT is pretty cool. I had to set the default threshold quite high due to the busywaiting we do in zephyr/drivers/dai/intel/ssp/ (a known limitation it seems), but it was cool to see this picked up by SPIN_LOGK_TIME_LIMIT.

Glad its working for you! Let me know if there's any ideas for improvement around this, happy to add them.

[teburd]

Not that my +1 counts here, but yes, this makes sense for debugging

[lgirdwood]

Not that my +1 counts here, but yes, this makes sense for debugging

You count :)

[btian1]

for HEAP_VALIDATE, sof does not call it directly, no sure zephyr will call it or not.
My feeling is that this does not like stack, low rate to have problems for heap.
I mean this maybe treated as option.

Stack overflow is must.

[kv2019i]

@btian1 This has found issues in the past and it triggers an assert if corruption is hit.

[lgirdwood]

@kv2019i seeing red on CI will rerun.

[lgirdwood]

SOFCI TEST

[kv2019i]

SOFCI TEST

[kv2019i]

Let me try once more. I couldn't make sense out of the failure. The tests failed because non-existing FW file, suggesting a build failure, but this overlay is not supposed to be used in CI runs. @keqiaozhang@greg-intel any insights? Let's not merge until this is clarified.

[kv2019i]

Moving back to draft. Our CI uses CONFIG_DEBUG to turn on features like the IPC flood test. So we really can't put feature like these that do add overhead across the board, to the builds.

[lgirdwood]

@kv2019i some failing tests around pause resume, wonder if timing has changed here ?

[kv2019i]

@wszypelt merge/build fails to "Unhandled Exception: System.BadImageFormatException" in IPC4 smoketest. Is this a failure in test runner, or actual failing case? I can't seem to find any errors related to test execution. @aborisovich you did a recent check with asserts enabled, right?

As for SOF test with Linux driver, I can't believe we hit an assert again:
https://sof-ci.01.org/sofpr/PR6530/build4058/devicetest/index.html

https://sof-ci.01.org/sofpr/PR6530/build4058/devicetest/index.html?model=ADLP_RVP_SDW_IPC4ZPH&testcase=multiple-pause-resume-5

ASSERTION FAIL [free_frames >= pending_frames] @ /srv/home/jenkins/workspace/sof_generic_build/sof/src/audio/mixin_mixout/mixin_mixout.c:368
[ 6.343360] <err> os: ** FATAL EXCEPTION
[ 6.343370] <err> os: ** CPU 0 EXCCAUSE 63 (zephyr exception)
[ 6.343378] <err> os: ** PC 0xbe050f5a VADDR (nil)

This is not the same as previously fixed in #6900. This has been passing many times in last few weeks, so either this is a very recent regression (in other code), or has a low occurence rate.

So back to draft status :(

[aborisovich]

@wszypelt merge/build fails to "Unhandled Exception: System.BadImageFormatException" in IPC4 smoketest. Is this a failure in test runner, or actual failing case? I can't seem to find any errors related to test execution. @aborisovich you did a recent check with asserts enabled, right?

As for SOF test with Linux driver, I can't believe we hit an assert again: https://sof-ci.01.org/sofpr/PR6530/build4058/devicetest/index.html

https://sof-ci.01.org/sofpr/PR6530/build4058/devicetest/index.html?model=ADLP_RVP_SDW_IPC4ZPH&testcase=multiple-pause-resume-5

ASSERTION FAIL [free_frames >= pending_frames] @ /srv/home/jenkins/workspace/sof_generic_build/sof/src/audio/mixin_mixout/mixin_mixout.c:368
[ 6.343360] <err> os: ** FATAL EXCEPTION
[ 6.343370] <err> os: ** CPU 0 EXCCAUSE 63 (zephyr exception)
[ 6.343378] <err> os: ** PC 0xbe050f5a VADDR (nil)

This is not the same as previously fixed in #6900. This has been passing many times in last few weeks, so either this is a very recent regression (in other code), or has a low occurence rate.

So back to draft status :(

Looks like a test runner issue.
I've executed full scope tests for MTL on 05.02.2023 so it was a while ago, maybe something new came up?

[kv2019i]

Another update:

• rebased as locally the recently merged mixin-mixout: fix 2 corner cases #7131 seems to help to avoid the assert on L368 of mixout_process()
• let's see if CI agrees with local results

[kv2019i]

@aborisovich Is this ok to you, you still have "changes requested" on this PR?

[kv2019i]

After rebase, tests are now good so we are finally ready to go.

[aborisovich]

Good to go

[teburd]

🥳

[marc-hb]

After a relatively painful git bisect I found that this has the unfortunate effect of making even the stripped ELF file non-reproducible :-(

gunzip -c ../build-sof-staging/sof-info/tgl/stripped-zephyr.elf.gz | strings now shows some absolute paths. It also shows a lot of relative paths so maybe there is hope?

cc: @lyakh , @andyross

[marc-hb]

I just remembered I fixed this with -fmacro-prefix-map in Zephyr 4 years ago :

• cmake: zephyr_cc_option(-fmacro-prefix-map=${ZEPHYR_BASE}=.) zephyrproject-rtos/zephyr#15376

[marc-hb]

After a relatively painful git bisect I found that this has the unfortunate effect of making even the stripped ELF file non-reproducible :-(

Good news and bad news.

Good news: mystery solved. This affects only old toolchains like the 2017 and gcc v4 based xt-xcc currently used for TGL.

https://reproducible-builds.org/docs/build-path/

-fmacro-prefix-map=OLD=NEW is similar to -fdebug-prefix-map, but addresses unreproducibility due to the use of FILE macros in assert calls for example. (available since GCC 8 and Clang 10)

Bad news: this will never be fixed for TGL, see issue #7114 why.