Uh oh!
There was an error while loading. Please reload this page.
audio: tensorflow: keyword detect - #11092
Draft
singalsu wants to merge 12 commits into
Draft
Conversation
Add a page-level VMA allocator so LLEXT modules can be placed on aligned pages of the module heap and enable SLID-based symbol linking in the module manager Kconfig. Extend the llext link helper so it bypasses the linker step when the toolchain already produced a relocatable output, forwards Clang --target= and --ld-path= flags to the underlying linker so cross-target invocations succeed, emits a merged section-layout script for the LLEXT link, and strips debug info from the final object. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Wire the TFLM sources into the SOF build both for statically linked firmware and for LLEXT relocatable modules. LLEXT builds cooperate with the llext link helper and pick up the Zephyr SDK GCC HiFi3 toolchain by default; the static path unblocks non-LLEXT firmware where TFLM has to live in the base image. Also make the xt-clang path portable: only add --target and --ld-path when the linker itself sets one, so Zephyr's clang wrapper is not double-configured. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Introduce a TensorFlow Lite Micro-based keyword-detection component that runs an int8-quantized micro-speech model on the mel features from the MFCC component. The key-phrase buffer (KPB) sits ahead of MFCC in the pipeline; on a positive keyword the TFLM component posts a KPB trigger notification so the host can drain the buffered audio. Also includes tensor byte-sizing fixes for the int8 input path, per-hop mel feature requantization that strips the MFCC header and avoids float math on the DSP, and stream-shutdown event logging so the host sees a clean end-of-capture on trigger. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
The on-device TFLM wake-word path (KPB -> SRC -> MFCC -> tflmcly) needs a 40-bin/20 ms-hop mel spectrogram to match TFLM micro_speech's front-end shape (FEATURE_SIZE=40, FEATURE_STRIDE_MS=20, FEATURE_DURATION_MS=30). Two variants are required: mel40.conf for the plain audio-buffer path used for off-target sanity checking and mel40_compress.conf for the compress PCM output actually consumed by the wake-word inference component. Add both export blocks to setup_mfcc.m so the blobs stay in lock-step under any future front-end tuning change, and ship the initial generated mel40.conf and mel40_compress.conf so the Wake-on-Voice topology graphs that consume them have matching blobs to reference. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add topology graphs and IPC4 ABI manifest bits for the Wake-on-Voice pipeline on HDA and SoundWire jack inputs, plus a bench variant for off-target training and validation. Register the KPB IPC4 UUID, attach the mel40 MFCC binary config blob as a real config, and give the ARL target the HDA_MIC_MFCC_PARAMS it was missing. Wire the dual-path KPB routing so the always-on inference branch and the host capture branch share the same buffer. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
The two stft_process source files pulled in libc's <assert.h>, which under a newlib-based Zephyr build (e.g. MTL with COMMON_LIBC_MALLOC_ARENA_SIZE set) expands assert() to a __assert_no_args() call whose implementation is not linked into the SOF firmware image. Enabling COMP_STFT_PROCESS on such a build therefore fails at link time with an "undefined reference to __assert_no_args" error. Every other SOF audio component that uses assert() includes <rtos/panic.h> instead, which maps assert() to Zephyr's __ASSERT_NO_MSG in firmware builds and to sof_panic() in the posix testbench. Do the same in stft_process-generic.c and stft_process-hifi3.c. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Enable the KPB, TFLM, MFCC, Gain and Volume modules on cAVS 2.5, Meteor Lake (ace15_mtpm) and PTL. cAVS 2.5 also gets its DP heap sized for the TFLM working set. PTL updates its rimage manifest to include the new components and shifts the library base address to make room. The tensorflow CMake wiring is parameterized to work on any xtensa target: hardcoded /home/lrg zephyr-sdk paths, mcpu names and include directories are replaced with SOC_TOOLCHAIN_NAME and ZEPHYR_SDK_INSTALL_DIR expansions, the -mllvm text-section-literals flag is feature-detected, HiFi4-only kernels are gated on TENSORFLOW_HAVE_NNLIB_HIFI4, and a small libc/libm archive-member shim provides the abs()/frexp()/round() symbols the static tflm build needs from newlib without pulling in the whole libc.a. The MTL board defconfig changes and the TWB thread stack bump that this feature also needs are kept in a separate [DNM] commit so they don't alter the default MTL build. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Document the TFLM Wake-on-Voice architecture: pipeline graph, module UUIDs, topology entry points, static vs LLEXT build variants, MFCC config blob layout, KPB trigger flow, and the on-device vs off-device verification loop. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add an end-to-end training pipeline for TFLM wake-word models under src/audio/mfcc/tune and src/audio/tensorflow/tune: - Piper-TTS driven keyword dataset generator with per-file Gaussian gain jitter capped 3 dB below full scale so the augmented WAVs directly reflect the level distribution the model will train on without clipping. - Batch feature extractor that runs the SOF testbench MFCC on the WAVs and writes raw mel40 features. - Silence and unknown-class preparation from Speech Commands v2 with the same level-jitter policy (silence exempt). - Python trainer that fits an int8-quantized DS-CNN on the mel features, emits the tflite/C-array pair, and auto-writes a sof_tflm_labels.h header so the runtime labels match the training set without hand-editing. - Off-device verifier that runs the tflite model against a directory of WAVs and reports per-class scores. Refreshes the tensorflow install and training recipe and fixes int8 calibration to use the correct scale and zero-point. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Previously binding two DP (Data Processing) scheduled components was rejected with IPC4_INVALID_REQUEST. This patch adds support for DP-to-DP binding by creating a dual ring buffer configuration where each DP module gets its own ring buffer on either side of the intermediate comp_buffer. Data flow for DP-to-DP: src_DP -> ring_buf_src -> comp_buffer -> ring_buf_sink -> sink_DP Changes in helper.c: - Remove the DP-to-DP bind rejection in ipc_comp_connect(). - Add src_is_dp, sink_is_dp, and dp_to_dp flags to detect the DP-to-DP case. - Create a second ring_buffer allocated from the source module's mod_alloc_ctx for the source side of the comp_buffer. - Refcount the DP vregion for each created ring_buffer via vregion_get(), with a NULL alloc guard. Changes in audio_buffer.c: - Change audio_buffer_attach_secondary_buffer() from a global rejection to per-side checks, allowing both secondary_buffer_sink and secondary_buffer_source to be set simultaneously. - Add a dual-secondary sync path in audio_buffer_sync_secondary_buffer() that cascades data through: input ring_buffer -> comp_buffer -> output ring_buffer, with rate-limiting applied on the output side. Changes in ring_buffer.c: - Release the DP vregion in ring_buffer_free() via vregion_put() and free the mod_alloc_ctx when the refcount reaches zero, matching the pattern used in comp_buffer_free(). Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
The existing sof_tflm_generate_keyword_dataset.sh is hard-wired to the English multi-speaker piper-sample-generator (LibriTTS-R). It cannot train a positive class in any other language because the phonemizer inside that checkpoint only knows English graphemes. Add a companion script that drives the regular piper-tts package against any single-speaker voice from rhasspy/piper-voices (Finnish fi_FI-harri, Swedish sv_SE-nst, etc.). To compensate for the missing speaker-embedding axis it (a) randomizes Piper noise-scale / noise-w per utterance while cycling several length-scale values and (b) fans each synthesized clip out into sox pitch-cents plus pitch-preserving tempo copies. Output layout, IR augmentation hook, and gain jitter match the English generator so sof_tflm_train_pipeline.sh consumes it unchanged. Update the tune README with a language-agnostic pipeline table entry, a non-English setup section, and Finnish and Swedish examples. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
This commit collects the MTL board defconfig, virtual heap and TWB stack changes that the TFLM Wake-on-Voice pipeline needs on Meteor Lake. It is kept separate so the default MTL build stays unchanged; apply it locally when testing the KPB + MFCC + TFLM pipeline on ace15_mtpm hardware. The MTL board defconfig gets the C++/staging switches TFLM needs (SOF_STAGING, CPP, STD_CPP17), builds tensorflow statically instead of as an llext module because zephyr-sdk GCC cannot link the -fPIC llext form, disables COMP_TESTER and COMP_GOOGLE_RTC for the same reason, bumps the kernel heap pool and EDF stack and adds a picolibc malloc arena for the static TFLM path. LLEXT_HEAP_DYNAMIC is turned off: the runtime-allocated LLEXT metadata heap broke D3 resume on ARL-S, where BASEFW sent FW_READY but never serviced the first post-resume IPC. The virtual heap default for MM_DRV_INTEL_ADSP_MTL_TLB and the default TWB thread stack are bumped globally so DP-scheduled TFLM inference and the KPB history buffer have enough working set. Both are default-value bumps that we do not want to land as-is, hence [DNM]. Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.