ipc: move delayed IPC sending to the primary core by lyakh · Pull Request #9764 · thesofproject/sof

lyakh · 2025-01-06T11:50:57Z

Low level IPC processing should be confined to the primary core. Move delayed IPC sending to a dedicated work queue with core 0 affinity.

Fixes: #8165

lgirdwood

@marcinszkudlinski pls review.

lgirdwood · 2025-01-06T13:38:31Z


+#ifdef __ZEPHYR__
+static struct k_work_q ipc_send_wq;
+static K_THREAD_STACK_DEFINE(ipc_send_wq_stack, 2048);


CONFIG_IPC_SEND_STACK_SIZE - lets make this a tunable.

@lgirdwood is it really worth it? This only occurs at one place. Maybe just put it in a header together with an additional one for
sof/zephyr/edf_schedule.c
Line 17 in c7b7c91
staticK_THREAD_STACK_DEFINE(edf_workq_stack, 8192);

Kconfig is cheap - we should have this option for all system threads.

@lgirdwood thinking a bit more - maybe we can have a single SOF work queue, run on core 0, serving both these (and potentially other) needs? EDF is currently used by IPC reception, KPB, mtrace logging, we could move the work queue out of it to make it global and then also use it for IPC sending?

We can look at this as a secondary phase, yes this will save memory but we need to look at what will/wont work here.

@lgirdwood if that approach is acceptable, I'd rather do that than adding significant infrastructure to only remove it again soon. I'll push that version for a test.

lgirdwood · 2025-01-06T13:45:07Z

+	k_work_queue_start(&ipc_send_wq,
+ ipc_send_wq_stack,
+ K_THREAD_STACK_SIZEOF(ipc_send_wq_stack),
+ 1, NULL);
+
+	k_thread_suspend(thread);
+
+	k_thread_cpu_mask_clear(thread);
+	k_thread_cpu_mask_enable(thread, PLATFORM_PRIMARY_CORE_ID);
+	k_thread_name_set(thread, "ipc_send_wq");
+
+	k_thread_resume(thread);


@lyakh probably worth a comment describing the aim here since its more than a couple of lines of code.
@peter-mitsis @andyross is there a convenience API for the above "pin work to a single core" ?

a slight improvement would be to use k_thread_cpu_pin()

lyakh · 2025-01-10T13:15:14Z

Interesting, the first attempt to run both IPC processing and sending on the same work queue resulted in apparent races, the new attempt produced IO errors on LNL https://sof-ci.01.org/sofpr/PR9764/build10135/devicetest/index.html?model=LNLM_SDW_AIOC&testcase=multiple-pause-resume-50 which don't look relevant but I cannot find any evidence of them appearing before. The QB failure is OTOH unrelated

Instead of hard-coding EDF scheduler work queue thread stack size, add a Kconfig option for it. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Low level IPC processing should be confined to the primary core. Move delayed IPC sending to a dedicated work queue with core 0 affinity. Fixes: thesofproject#8165 Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

lyakh · 2025-01-14T10:06:01Z

jenkins seems to be hanging, try to relaunch

lyakh · 2025-01-14T10:06:08Z

SOFCI TEST

lgirdwood · 2025-01-14T14:05:06Z

jenkins seems to be hanging, try to relaunch

Seeing its completed, but results not uploaded yet... lets check later.

lyakh · 2025-01-15T08:32:13Z

LNL test came back empty https://sof-ci.01.org/sofpr/PR9764/build10171/devicetest/index.html . The other two were clean. Need to re-run yet again.

lyakh · 2025-01-15T08:32:30Z

SOFCI TEST

lyakh · 2025-01-16T11:14:50Z

SOFCI TEST

lyakh · 2025-01-17T14:18:51Z

SOFCI TEST

lyakh · 2025-01-20T07:54:59Z

CI: only unrelated failures in https://sof-ci.01.org/sofpr/PR9764/build10250/devicetest/index.html , https://sof-ci.01.org/sofpr/PR9764/build10248/devicetest/index.html and https://sof-ci.01.org/sofpr/PR9764/build10249/devicetest/index.html

abonislawski · 2025-01-20T08:06:08Z

@wszypelt please run multicore tests with this PR (or just full scope to compare easily)

wszypelt · 2025-02-03T13:29:52Z

@lyakh @abonislawski after carefully checking all the differences, I think PR is suitable for merging

kv2019i · 2025-02-04T13:11:36Z

FYI @marcinszkudlinski @abonislawski I'll proceed with merge end-of-day today if no objections...

lgirdwood reviewed Jan 6, 2025
View reviewed changes

lyakh marked this pull request as ready for review January 6, 2025 14:53

lyakh requested review from bardliao, marcinszkudlinski and pblaszko as code owners January 6, 2025 14:53

marcinszkudlinski requested a review from abonislawski January 7, 2025 09:03

lyakh force-pushed the ipc branch from c7b7c91 to c24c7e2Compare January 9, 2025 15:11

lyakh requested review from dbaluta, iuliana-prodan and kv2019i as code owners January 9, 2025 15:11

kv2019i approved these changes Jan 9, 2025
View reviewed changes

lyakh force-pushed the ipc branch from c24c7e2 to 0bf9779Compare January 10, 2025 10:56

lyakh added the DNM Do Not Merge tag label Jan 10, 2025

scheduler: make EDF stack size configurable
decc67e
Instead of hard-coding EDF scheduler work queue thread stack size, add a Kconfig option for it. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

lyakh force-pushed the ipc branch from 0bf9779 to 549927cCompare January 10, 2025 15:03

lyakh force-pushed the ipc branch from 549927c to 9428f95Compare January 13, 2025 11:30

lyakh removed the DNM Do Not Merge tag label Jan 21, 2025

wszypelt approved these changes Feb 4, 2025
View reviewed changes

marcinszkudlinski approved these changes Feb 4, 2025
View reviewed changes

kv2019i merged commit 85f1dbe into thesofproject:mainFeb 4, 2025

lyakh deleted the ipc branch February 5, 2025 07:03

Uh oh!

Conversation

lyakh commented Jan 6, 2025

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyakh commented Jan 10, 2025

Uh oh!

lyakh commented Jan 14, 2025

Uh oh!

lyakh commented Jan 14, 2025

Uh oh!

lgirdwood commented Jan 14, 2025

Uh oh!

lyakh commented Jan 15, 2025

Uh oh!

lyakh commented Jan 15, 2025

Uh oh!

lyakh commented Jan 16, 2025

Uh oh!

lyakh commented Jan 17, 2025

Uh oh!

lyakh commented Jan 20, 2025

Uh oh!

abonislawski commented Jan 20, 2025

Uh oh!

wszypelt commented Feb 3, 2025

Uh oh!

kv2019i commented Feb 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants