topology2: add dummy wov pipeline to nocodec by libinyang · Pull Request #6782 · thesofproject/sof

libinyang · 2022-12-13T06:56:44Z

Add dummy WoV support to nocodec topology on MTL platform.

The topology of dummy WoV is:
DMIC ------> KPB ------------------------------> host_copier
| ^
| |
|----> micsel ----> dummy wov ----> virtual_widget

The connection between virtual_widget and host_copier is a virtual connection, and the virtual widget doesn't exist in the FW.

PS: there seems to be an issue that DMIC0 and DMIC1 can't co-work in the topology2. If I enabled both DMICs, the nhlt blob will complain doesn't support the audio format which is used by dummy WoV. After this issue is fixed, the cavs-nocodec-wov.conf is not necessary and it can reuse cavs-nocodec.conf.

libinyang · 2022-12-13T10:29:45Z

Thank @juimonen 's reminder, @singalsu had a PR #6668 which can fix the dmic issue. I will refine this PR to reuse cavs-nocodec.conf.

libinyang · 2022-12-13T11:53:55Z

Patches are updated to reuse cavs-nocodec based on 8938c31

lgirdwood

LGTM, but need inputs from others here.

libinyang · 2022-12-19T06:18:52Z

V3 updated: remove enabling WoV on sof-mtl-nocodec. This is because WoV is waiting for Chao's patch on hw_params updating.

libinyang · 2022-12-19T07:35:21Z

v4 update:

add the blob's struct description of ipc4 header

ranj063 · 2022-12-19T17:50:41Z

how is this determined?

Copied from CoE team.

@libinyang when you give someone an answer like this, you should know what the next question is going to be right? Please go back to them and find out how this was derived and add a comment. Thanks!

cps = cpc * sampling frequency // sample group in ms * frame size, which can be found in the platform config file. So you can calculate the cpc value.
ps: I won't add the cpc calculation description in the patch

sorry what is cps?

sorry what is cps?
I think CPS is "Cycles Per Second" after grep the sof fw src code.
Cited more from CoE team
"frame_size - number of milliseconds that one frame consists of.
cpc - number of cycles per chunk, chunk is a data processed during one frame. "
And cps value can be found in the file platform config file in the rimage.

marc-hb · 2022-12-19T22:16:15Z

sof-test has a https://github.com/thesofproject/sof-test/blob/main/test-case/check-keyword-detection.sh test that we could never use so far (see thesofproject/sof-test#859 and links from there).

Will this test be finally usable? If not, how is this PR being tested?

Disclaimer: I have NOT looked at this PR.

libinyang · 2022-12-20T03:51:03Z

sof-test has a https://github.com/thesofproject/sof-test/blob/main/test-case/check-keyword-detection.sh test that we could never use so far (see thesofproject/sof-test#859 and links from there).
Will this test be finally usable? If not, how is this PR being tested?
Disclaimer: I have NOT looked at this PR.

@marc-hb I didn't really enable WoV in CMakefile. We still need @aiChaoSONG PR of updating the hw_params. Without @aiChaoSONG ' patch, there will be some failure in test WoV. After Chao's patch is merged, I will enable WoV in CMakefile.

libinyang · 2022-12-20T14:52:42Z

v9 update:

use wov-detect as a general wov detection pipeline
use conf file as the input of wov kcontrol data
some other small changes as @ranj063 suggested.

ranj063 · 2022-12-20T16:53:46Z

sorry what is cps?

ranj063 · 2022-12-20T20:01:18Z

Can you please remind me what would happen if we dont set the cpc?

Can you please remind me what would happen if we dont set the cpc?
This is a good question! I tried not to set cpc before when enabling wov SOF fw. It seemed to work even without cpc setting. Maybe we need CoE expert help to answer this question. @iganakov

And maybe we can enhance the fw to set the cpc in FW itself and don't need tplg setting it.

was also wondering if the cpc is needed in every widget, we should probably add it to base class etc. I guess it is originally used to count the pipeline cycles so it has "room" to run, not sure what is the use in practice.

if we dont need cpc for wov, is it needed for kpb and micsel?

ranj063 · 2022-12-21T03:16:38Z

Capture 18? Can we call this "WoV Capture" instead?

DMIC1 is a general usage, it can be used for other purposes.

libinyang · 2022-12-21T04:10:22Z

v10 update:

refine payload_with_output_fmt token based on thesofproject/linux@831a267
update kpb cpc
some other tiny changes based on @ranj063 's comments.

ranj063 · 2022-12-21T04:13:23Z

Passthrough Capture 18 makes it even worse. Please call it something more indicative of what it is. DMIC1 capture?

This is defined in dmic-default.conf, cavs-nocodec.conf will re-define it as 'Capture 18'. I will change it to 'DMIC1 WOV capture" in cavs-nocodec.conf

libinyang · 2022-12-21T07:19:28Z

v11 update:
Some tiny changes based on @ranj063 's comments: remove the 'stream_name' define; rename the DMIC1_PCM_CAPS; refine the commit message.

The token payload_with_output_fmt is process module specific, so set it as a process module token. Signed-off-by: Chao Song <chao.song@linux.intel.com> Signed-off-by: Libin Yang <libin.yang@intel.com>

Add the audio channel selection component support. In case 1 output channel is selected, the component provides the selected channel on output. In case 2 or 4 channels are selected on output, the component works in a passthrough mode. Signed-off-by: Libin Yang <libin.yang@intel.com>

Add the kpb component support. kpb component is used to transfer the data to the proper pipeline based on the WOV component events. Signed-off-by: Libin Yang <libin.yang@intel.com>

Add the wov module support. The keyword of the wov is the sound of clap. Signed-off-by: Libin Yang <libin.yang@intel.com>

ranj063 · 2022-12-21T16:38:29Z

ranj063 · 2022-12-21T16:41:55Z

@libinyang even though this works. This is incorrect. What you need is a new pipeline for kpb-capture that has the DAI copier and the KPB connected

@libinyang even though this works. This is incorrect. What you need is a new pipeline for kpb-capture that has the DAI copier and the KPB connected
Why do we need a different pipeline for kbp and DAI? The dmic-wov is only used for wov and I prefer it not to be re-used by other scenarios.

ranj063 · 2022-12-21T16:42:23Z

if we dont need cpc for wov, is it needed for kpb and micsel?

wov-detect pipeline contains a micsel component, a wov component and a virtual component. micsel component accepts 2 channels audio data and output 1 channel audio data to the wov component. wov's keyword is a clap. The virtual component is only used by the driver, and the FW will not know that there is a virtual component in the pipeline. The virtual component is used to connect the pipeline to a host copier pipeline as per drivers requirement. Signed-off-by: Libin Yang <libin.yang@intel.com>

dai-kpb-be pipeline contains a copier DAI component and a kpb component. Signed-off-by: Libin Yang <libin.yang@intel.com>

The dmic-wov.conf file implements the WOV feature. The connection is DMIC ------> KPB ------------------------------> host_copier | ^ | | |----> micsel ----> wov ------> virtual_widget The DMIC and KPB is on DAI ppl. The host_copier is on Host ppl. The micsel, wov and virtual is on wov ppl. The connection between host_copier and virtual_widget is a virtual connection. The virtual_widget doesn't exist in the FW. Signed-off-by: Libin Yang <libin.yang@intel.com>

Add the definitions of wov in cavs-nocodec.conf. Signed-off-by: Libin Yang <libin.yang@intel.com>

libinyang · 2022-12-22T02:06:25Z

v12 update:

add a dai-kpb-be pipeline instead of using passthrough-be.

ranj063

Looks good @libinyang. The only open I have is the cpc. I think we need to do a broader cleanup of cpc's for all topologies. Lets plan for that next

libinyang requested review from aborisovich, dbaluta, lbetlej, lgirdwood, marc-hb, mmaka1 and plbossart as code owners December 13, 2022 06:56

libinyang mentioned this pull request Dec 13, 2022
[BUG][Regression] bind KPB and copier failed #6784
Closed

libinyang force-pushed the dummy_wov_tplg_submit branch from b065b0a to ef4fa5eCompare December 13, 2022 11:52

libinyang force-pushed the dummy_wov_tplg_submit branch from ef4fa5e to 58dcbaaCompare December 13, 2022 12:26

libinyang mentioned this pull request Dec 13, 2022
[BUG] triggering multiple pipeline start failed for WoV tplg #6793
Closed

libinyang requested review from bardliao, juimonen and ranj063 December 14, 2022 09:24

libinyang changed the title ~~Dummy wov tplg submit~~topology2: dummy wov pipeline supportDec 14, 2022

mengdonglin changed the title ~~topology2: dummy wov pipeline support~~topology2: add dummy wov pipeline to nocodecDec 14, 2022

lgirdwood reviewed Dec 14, 2022
View reviewed changes

Comment threadtools/topology/topology2/include/pipelines/cavs/passthrough-wov.conf Outdated
Comment threadtools/topology/topology2/platform/intel/dmic-generic.conf Outdated
Comment threadtools/topology/topology2/sof-ace-tplg/CMakeLists.txt Outdated

libinyang force-pushed the dummy_wov_tplg_submit branch from 58dcbaa to e7f0613Compare December 15, 2022 06:36

singalsu reviewed Dec 15, 2022
View reviewed changes

libinyang force-pushed the dummy_wov_tplg_submit branch 2 times, most recently from ce6edf8 to fc3cf85Compare December 19, 2022 05:31

bardliao reviewed Dec 19, 2022
View reviewed changes

Comment threadtools/topology/topology2/include/pipelines/cavs/passthrough-wov.conf Outdated

libinyang force-pushed the dummy_wov_tplg_submit branch from fc3cf85 to b5705ceCompare December 19, 2022 07:34

ranj063 reviewed Dec 19, 2022
View reviewed changes

ranj063 reviewed Dec 20, 2022
View reviewed changes

libinyang force-pushed the dummy_wov_tplg_submit branch from 0d75f94 to cd2f990Compare December 21, 2022 03:00

ranj063 reviewed Dec 21, 2022
View reviewed changes

Comment threadtools/topology/topology2/platform/intel/dmic-default.conf Outdated

ranj063 reviewed Dec 21, 2022
View reviewed changes

Comment threadtools/topology/topology2/cavs-nocodec.conf Outdated

libinyang force-pushed the dummy_wov_tplg_submit branch 2 times, most recently from 094a87d to d2b8cdbCompare December 21, 2022 04:07

ranj063 reviewed Dec 21, 2022
View reviewed changes

Comment threadtools/topology/topology2/platform/intel/dmic-wov.conf Outdated

ranj063 reviewed Dec 21, 2022
View reviewed changes

Comment threadtools/topology/topology2/platform/intel/dmic-wov.conf Outdated

libinyang force-pushed the dummy_wov_tplg_submit branch from d2b8cdb to fe5c4f4Compare December 21, 2022 07:17

libinyang added 4 commits December 21, 2022 15:21

topology2: move payload_with_output_fmt into process token set
08dee14
The token payload_with_output_fmt is process module specific, so set it as a process module token. Signed-off-by: Chao Song <chao.song@linux.intel.com> Signed-off-by: Libin Yang <libin.yang@intel.com>

topology2: add kpb support
d774ae9
Add the kpb component support. kpb component is used to transfer the data to the proper pipeline based on the WOV component events. Signed-off-by: Libin Yang <libin.yang@intel.com>

topology2: add wov support
32dc55a
Add the wov module support. The keyword of the wov is the sound of clap. Signed-off-by: Libin Yang <libin.yang@intel.com>

libinyang force-pushed the dummy_wov_tplg_submit branch from fe5c4f4 to 5dc2a8fCompare December 21, 2022 07:22

ranj063 reviewed Dec 21, 2022
View reviewed changes

libinyang added 4 commits December 22, 2022 09:52

topology2: add dai-kpb-be pipeline
020a5fe
dai-kpb-be pipeline contains a copier DAI component and a kpb component. Signed-off-by: Libin Yang <libin.yang@intel.com>

topology2: cavs-nocodec add wov support
a25f4f6
Add the definitions of wov in cavs-nocodec.conf. Signed-off-by: Libin Yang <libin.yang@intel.com>

libinyang force-pushed the dummy_wov_tplg_submit branch from 5dc2a8f to a25f4f6Compare December 22, 2022 02:05

ranj063 approved these changes Dec 22, 2022
View reviewed changes

bardliao approved these changes Dec 22, 2022
View reviewed changes

lgirdwood approved these changes Dec 22, 2022
View reviewed changes

lgirdwood merged commit 1447444 into thesofproject:mainDec 22, 2022

Uh oh!

Conversation

libinyang commented Dec 13, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

libinyang commented Dec 13, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

libinyang commented Dec 13, 2022

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

libinyang commented Dec 19, 2022

Uh oh!

libinyang commented Dec 19, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marc-hb commented Dec 19, 2022

Uh oh!

libinyang commented Dec 20, 2022

Uh oh!

libinyang commented Dec 20, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ranj063Dec 20, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

libinyang commented Dec 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

libinyang commented Dec 13, 2022•
edited
Loading

libinyang commented Dec 13, 2022•
edited
Loading

ranj063Dec 20, 2022•
edited
Loading

libinyang commented Dec 21, 2022•
edited
Loading