Up2 with nocodec topology: DSP panic with 6 SSPs

[plbossart]

With the usual 6 SSP+1DMIC topology the DSP panics. When the SSP5 is commented out from the topology file things work fine.
I suspect an off-by-one error. Filing as a kernel bug for now but this is more of a firmware issue I suspect.

Differences in the topology file to make things work:

diff --git a/topology/sof-apl-nocodec.m4 b/topology/sof-apl-nocodec.m4
index 2a99560..711589f 100644
--- a/topology/sof-apl-nocodec.m4
+++ b/topology/sof-apl-nocodec.m4
@@ -100,17 +100,17 @@ PIPELINE_PCM_ADD(sof/pipe-volume-capture.m4,
 	10, 4, 2, s16le,
 	48, 1000, 0, 0)
 
-# Low Latency playback pipeline 11 on PCM 5 using max 2 channels of s16le.
-# Schedule 48 frames per 1000us deadline on core 0 with priority 0
-PIPELINE_PCM_ADD(sof/pipe-volume-playback.m4,
-	11, 5, 2, s16le,
-	48, 1000, 0, 0)
-
-# Low Latency capture pipeline 12 on PCM 5 using max 2 channels of s16le.
-# Schedule 48 frames per 1000us deadline on core 0 with priority 0
-PIPELINE_PCM_ADD(sof/pipe-volume-capture.m4,
-	12, 5, 2, s16le,
-	48, 1000, 0, 0)
+# # Low Latency playback pipeline 11 on PCM 5 using max 2 channels of s16le.
+# # Schedule 48 frames per 1000us deadline on core 0 with priority 0
+# PIPELINE_PCM_ADD(sof/pipe-volume-playback.m4,
+# 	11, 5, 2, s16le,
+# 	48, 1000, 0, 0)
+
+# # Low Latency capture pipeline 12 on PCM 5 using max 2 channels of s16le.
+# # Schedule 48 frames per 1000us deadline on core 0 with priority 0
+# PIPELINE_PCM_ADD(sof/pipe-volume-capture.m4,
+# 	12, 5, 2, s16le,
+# 	48, 1000, 0, 0)
 
 # Passthrough capture pipeline 13 on PCM 6 using max 2 channels.
 # Schedule 48 frames per 1000us deadline on core 0 with priority 0
@@ -197,26 +197,26 @@ DAI_ADD(sof/pipe-dai-capture.m4,
 	PIPELINE_SINK_10, 2, s16le,
 	48, 1000, 0, 0)
 
-# playback DAI is SSP5 using 2 periods
-# Buffers use s16le format, with 48 frame per 1000us on core 0 with priority 0
-DAI_ADD(sof/pipe-dai-playback.m4,
-	11, SSP, 5, NoCodec-5,
-	PIPELINE_SOURCE_11, 2, s16le,
-	48, 1000, 0, 0)
-
-# capture DAI is SSP5 using 2 periods
-# Buffers use s16le format, with 48 frame per 1000us on core 0 with priority 0
-DAI_ADD(sof/pipe-dai-capture.m4,
-	12, SSP, 5, NoCodec-5,
-	PIPELINE_SINK_12, 2, s16le,
-	48, 1000, 0, 0)
-
-# capture DAI is DMIC 0 using 2 periods
-# Buffers use s32le format, with 48 frame per 1000us on core 0 with priority 0
-DAI_ADD(sof/pipe-dai-capture.m4,
-	13, DMIC, 0, NoCodec-6,
-	PIPELINE_SINK_13, 2, s32le,
-	48, 1000, 0, 0)
+# # playback DAI is SSP5 using 2 periods
+# # Buffers use s16le format, with 48 frame per 1000us on core 0 with priority 0
+# DAI_ADD(sof/pipe-dai-playback.m4,
+# 	11, SSP, 5, NoCodec-5,
+# 	PIPELINE_SOURCE_11, 2, s16le,
+# 	48, 1000, 0, 0)
+
+# # capture DAI is SSP5 using 2 periods
+# # Buffers use s16le format, with 48 frame per 1000us on core 0 with priority 0
+# DAI_ADD(sof/pipe-dai-capture.m4,
+# 	12, SSP, 5, NoCodec-5,
+# 	PIPELINE_SINK_12, 2, s16le,
+# 	48, 1000, 0, 0)
+
+# # capture DAI is DMIC 0 using 2 periods
+# # Buffers use s32le format, with 48 frame per 1000us on core 0 with priority 0
+# DAI_ADD(sof/pipe-dai-capture.m4,
+# 	13, DMIC, 0, NoCodec-6,
+# 	PIPELINE_SINK_13, 2, s32le,
+# 	48, 1000, 0, 0)
 
 dnl PCM_DUPLEX_ADD(name, pcm_id, playback, capture)
 PCM_DUPLEX_ADD(Port0, 0, PIPELINE_PCM_1, PIPELINE_PCM_2)
@@ -224,7 +224,7 @@ PCM_DUPLEX_ADD(Port1, 1, PIPELINE_PCM_3, PIPELINE_PCM_4)
 PCM_DUPLEX_ADD(Port2, 2, PIPELINE_PCM_5, PIPELINE_PCM_6)
 PCM_DUPLEX_ADD(Port3, 3, PIPELINE_PCM_7, PIPELINE_PCM_8)
 PCM_DUPLEX_ADD(Port4, 4, PIPELINE_PCM_9, PIPELINE_PCM_10)
-PCM_DUPLEX_ADD(Port5, 5, PIPELINE_PCM_11, PIPELINE_PCM_12)
+#PCM_DUPLEX_ADD(Port5, 5, PIPELINE_PCM_11, PIPELINE_PCM_12)
 dnl PCM_CAPTURE_ADD(name, pipeline, capture)
 PCM_CAPTURE_ADD(DMIC01, 6, PIPELINE_PCM_13)
 
@@ -270,12 +270,12 @@ DAI_CONFIG(SSP, 4, 4, NoCodec-4,
 		 SSP_TDM(2, 16, 3, 3),
 		 SSP_CONFIG_DATA(SSP, 4, 16)))
 
-DAI_CONFIG(SSP, 5, 5, NoCodec-5,
-	 SSP_CONFIG(I2S, SSP_CLOCK(mclk, 24576000, codec_mclk_in),
-		 SSP_CLOCK(bclk, 1536000, codec_slave),
-		 SSP_CLOCK(fsync, 48000, codec_slave),
-		 SSP_TDM(2, 16, 3, 3),
-		 SSP_CONFIG_DATA(SSP, 5, 16)))
+# DAI_CONFIG(SSP, 5, 5, NoCodec-5,
+# 	 SSP_CONFIG(I2S, SSP_CLOCK(mclk, 24576000, codec_mclk_in),
+# 		 SSP_CLOCK(bclk, 1536000, codec_slave),
+# 		 SSP_CLOCK(fsync, 48000, codec_slave),
+# 		 SSP_TDM(2, 16, 3, 3),
+# 		 SSP_CONFIG_DATA(SSP, 5, 16)))
 
 DAI_CONFIG(DMIC, 0, 6, NoCodec-6,
 	 dnl DMIC_CONFIG(driver_version, clk_min, clk_mac, duty_min, duty_max,

[plbossart]

dmesg log excerpt

[ 4.747067] sof-audio sof-audio: dai SSP5.IN: type 1 index 5
[ 4.747068] sof-audio sof-audio: config: periods snk 2 src 0 fmt 0
[ 4.747081] sof-audio sof-audio: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[ 4.747202] sof-audio sof-audio: ipc tx succeeded: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[ 4.747211] sof-audio sof-audio: tplg: connected widget SSP5.IN -> DAI link NoCodec-5
[ 4.747217] sof-audio sof-audio: tplg: ready widget id 71 pipe 12 type 32 name : PIPELINE.12.SSP5.IN stream SSP5.IN
[ 4.747220] sof-audio sof-audio: tplg: pipeline id 12 comp 71 scheduling comp id 70
[ 4.747223] sof-audio sof-audio: pipeline PIPELINE.12.SSP5.IN: deadline 1000 pri 0 mips 5000 core 0 frames 48
[ 4.747235] sof-audio sof-audio: ipc tx: 0x30100000: GLB_TPLG_MSG: PIPE_NEW
[ 4.747297] sof-audio sof-audio: error : DSP panic!
[ 4.747636] sof-audio sof-audio: panic: dsp_oops_offset 788480 offset 788480
[ 4.747641] sof-audio sof-audio: status: fw entered - code 00000005
[ 4.748034] sof-audio sof-audio: error: runtime exception
[ 4.748352] sof-audio sof-audio: error: trace point 00004000
[ 4.748690] sof-audio sof-audio: error: panic happen at :0
[ 4.749009] sof-audio sof-audio: error: DSP Firmware Oops

Full log here:
dmesg_ssp5_fail.log

[keyonjie]

@plbossart I think @libinyang is working on a similar issue.

@libinyang can you help comment?

[libinyang]

@plbossart @keyonjie
I think this should be the similar issue of thesofproject/sof#539

I'm working on #539 in sof repo. And I have PR for SOF: thesofproject/sof#541 to fix such issue.

The issue is caused by initializing memory heap incorrectly. This will cause some memory related potential issues.

[plbossart]

@libinyang yes the problem goes away with your fix

However @ranj063 I now have a DMIC IPC issue that appears with your DMIC alloc fix in the firmware, can you look at the topology for the nocodec case? Looks like we fix GLK but broke the rest...

dmesg log with Libin's PR541+sof-apl-nocodec.m4 topology:

 5.288015] sof-audio sof-audio: tplg: ready widget id 74 pipe 13 type 28 name : DMIC0.IN stream NoCodec-6
[ 5.288018] sof-audio sof-audio: dai DMIC0.IN: type 2 index 0
[ 5.288020] sof-audio sof-audio: config: periods snk 2 src 0 fmt 2
[ 5.288033] sof-audio sof-audio: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[ 5.288151] sof-audio sof-audio: ipc tx succeeded: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[ 5.288156] sof-audio sof-audio: tplg: connected widget DMIC0.IN -> DAI link NoCodec-6
[ 5.288162] sof-audio sof-audio: tplg: ready widget id 75 pipe 13 type 32 name : PIPELINE.13.DMIC0.IN stream DMIC0.IN
[ 5.288164] sof-audio sof-audio: tplg: pipeline id 13 comp 75 scheduling comp id 74
[ 5.288167] sof-audio sof-audio: pipeline PIPELINE.13.DMIC0.IN: deadline 1000 pri 0 mips 5000 core 0 frames 48
[ 5.288179] sof-audio sof-audio: ipc tx: 0x30100000: GLB_TPLG_MSG: PIPE_NEW
[ 5.288297] sof-audio sof-audio: error: ipc error for 0x30100000 size 0x14
[ 5.288728] sof-audio sof-audio: error: DSP failed to add widget id 0 type 32 name : PIPELINE.13.DMIC0.IN stream DMIC0.IN reply -12
[ 5.289425] sof-audio sof-audio: ASoC: failed to load widget PIPELINE.13.DMIC0.IN
[ 5.289875] sof-audio sof-audio: error: tplg component load failed -12
[ 5.290282] sof-audio sof-audio: ipc rx: 0x90020000: GLB_TRACE_MSG
[ 5.290297] sof-audio sof-audio: ipc rx done: 0x90020000: GLB_TRACE_MSG
[ 5.299733] sof-audio sof-audio: error: failed to load DSP topology -22
[ 5.300172] sof-audio sof-audio: ASoC: failed to probe component -22
[ 5.300644] sof-nocodec sof-nocodec: ASoC: failed to instantiate card -22
[ 5.301135] sof-nocodec: probe of sof-nocodec failed with error -22

if I take out the DMIC from the topology things are just fine. How many engineers does it take to fix memory allocation....

[libinyang]

@plbossart @ranj063
The later bug can be fixed by:
thesofproject/sof#549

However, 549 must be applied after 541 because if the memory heap initialization issue is not fixed, the system is unstable for memory mapping changing, and many potential issues may occur.

[mengdonglin]

This is a duplicate of thesofproject/sof#539, and Libin has a PR to fix it

[libinyang]

@stevyan
This bug should be fixed with the latest SOF with PR thesofproject/sof#541 and thesofproject/sof#549. Could you please verify?

@plbossart

[markyang]

Summary:
This issue cannot be reproduced on UP2 any more.

Test env:
sof master: 5838f71
sof tool: 2cd668c
kernel sof-dev: 9b81e16
tplg: sof-apl-nocodec.tplg-2cd668c