boot test and virtual heap testing by lyakh · Pull Request #7688 · thesofproject/sof

lyakh · 2023-05-25T12:26:52Z

No description provided.

lyakh · 2023-06-06T15:00:04Z

SOFCI TEST

lyakh · 2023-06-06T15:08:31Z

SOFCI TEST

lgirdwood

@dabekjakub is this test now passing with latest VMH PR ?

marc-hb

Do we really need yet another, brand new test framework? Genuine question. Can't Zephyr's ZTEST and twister help with this?

https://docs.zephyrproject.org/latest/develop/test/ztest.html

cc: @teburd , @dcpleung

lyakh · 2023-11-17T07:30:18Z

Do we really need yet another, brand new test framework? Genuine question. Can't Zephyr's ZTEST and twister help with this?
https://docs.zephyrproject.org/latest/develop/test/ztest.html
cc: @teburd , @dcpleung

@marc-hb as far as I understand - no, not easily, ztest seems to be (mostly) used with twister when building a separate image just for testing, whereas what we want is a kind of a boot self-test, similar to what the kernel can do. I'd much prefer to use an existing framework and if ztest or another Zephyr API can do that - I'd love to know about it and to switch to it! Or maybe we want to add boot-time self-testing to Zephyr?

lyakh · 2023-11-17T07:31:32Z

@dabekjakub is this test now passing with latest VMH PR ?

@lgirdwood no, I see a failure in the log, I'll double check whether I'm calling the API wrongly or if there's a problem in VMH.

lyakh · 2023-11-17T08:17:20Z

@dabekjakub is this test now passing with latest VMH PR ?
@lgirdwood no, I see a failure in the log, I'll double check whether I'm calling the API wrongly or if there's a problem in VMH.

Unfortunately all https://sof-ci.01.org/sofpr/PR7688/build14963/devicetest/index.html test results are empty, but in my local test I got this:

[ 0.135591] <inf> pipe: pipeline_new: pipeline new pipe_id 0 priority 0
[ 0.135743] <wrn> vmhtest: vmh_test_single: Ignoring failure to allocate 1616 in non-contiguous mode
[ 0.135750] <err> vmhtest: vmh_test_single: Failed to allocate 26
[ 0.135946] <err> vmhtest: vmh_test: Non-contiguous test error -12
[ 0.135951] <inf> boot_test: sof_boot_test: test 0xa0076534 returned -12

So either I'm testing wrongly or there's a problem with VMH, @dabekjakub ?

teburd · 2023-11-17T15:06:13Z

Do we really need yet another, brand new test framework? Genuine question. Can't Zephyr's ZTEST and twister help with this?
https://docs.zephyrproject.org/latest/develop/test/ztest.html
cc: @teburd , @dcpleung
@marc-hb as far as I understand - no, not easily, ztest seems to be (mostly) used with twister when building a separate image just for testing, whereas what we want is a kind of a boot self-test, similar to what the kernel can do. I'd much prefer to use an existing framework and if ztest or another Zephyr API can do that - I'd love to know about it and to switch to it! Or maybe we want to add boot-time self-testing to Zephyr?

ztest should be usable outside of twister, its a way of structuring test code and checks (asserts)

You can run any test in the zephyr/tests directory by building and flashing normally with west, nothing particularly special about twister doing this.

And you can decide when and how to run but by default it creates a main to do this for you.

lyakh · 2023-11-20T07:22:19Z

ztest should be usable outside of twister, its a way of structuring test code and checks (asserts)
You can run any test in the zephyr/tests directory buy building and flashing normally with west, nothing particularly special about twister doing this.
And you can decide when and how to run but by default it creates a main to do this for you.

@teburd right, thanks. My understanding is that maybe it can be used but it hardly would bring anything in this configuration. What we need is to run multiple tests at SOF startup time. So it should be an otherwise normal SOF build, having run those tests it will proceed and function normally as any other SOF firmware. And it seems to me that we need to run them at a specific point in SOF run-time sequence, long after all Zephyr initialisation has completed. What advantages would we still gain if using ztest - if at all possible? We'd need to make a new launch point inside of SOF too, right?

lgirdwood · 2023-11-24T16:34:59Z

This also need to be used by the SOF application so that after a normal FW boot we enter the Ztest logic and run thru all the UTs. The FW boot complete IPC messaging can then be used to indicate test results. The kernel would know about Ztest mode with an extended manifest flag.
@lgirdwood the way it's currently done - yes, this framework can be used from other SOF code paths too. Not sure what you mean by "all the UTs" - new or existing ones? You can write new ones or possibly adapt tests from the existing testbench / cmocka sets. As for the kernel knowing about that - so far it's decided by a Kconfig option and the kernel doesn't need to know anything about it. But sure, we can extend it in the future

Today we have very little UTs, so yes, we can add more and also add existing cmocka tests where appropriate (and if posible).

Kconfig needs to set a flag in the ext manifest so that kernel knows "this is a test"

lgirdwood

LGTM - ztest "hello world"
@lyakh dont see the extended manifest flag yet. How do we report failures when a test does not assert() or cause an exception ? e.g. will everything go out via trace

test 0: feature1 vmh.c:54 - pass
test 1: feature2 vmh.c:87 - failed - got 97 expected 78.
....
test: all test completed 89 passed 3 failed

lgirdwood · 2023-11-27T12:09:17Z

+config SOF_BOOT_TEST
+ bool "enable SOF run-time testing"
+ depends on ZTEST
+ help


Can we expand this more and tell the users that its for testing use only and should not be enabled for normal audio usage. i.e what this enables/disables and impact.

lgirdwood · 2023-11-27T12:11:41Z

+	if (!h)
+ return -ENOMEM;
+
+	return vmh_free_heap(h);


@lyakh @dabekjakub this is a simple test today i.e. "hello world", but lets add to it and stress it when we have the ztest infra understood and merged (as I think we all have good test ideas baking....).

lyakh · 2023-11-28T07:44:02Z

LGTM - ztest "hello world" @lyakh dont see the extended manifest flag yet.

@lgirdwood can we discuss and add manifest and IPC reporting as follow-up PRs? Manifest is needed to make sure that the kernel is expecting any eventual delays?

How do we report failures when a test does not assert() or cause an exception ? e.g. will everything go out via trace
test 0: feature1 vmh.c:54 - pass
test 1: feature2 vmh.c:87 - failed - got 97 expected 78.
....
test: all test completed 89 passed 3 failed

Yes, logging, unfortunately I don't see any CI device results in this PR, so copying from my local testing (possibly of a modified version):

Running TESTSUITE sof_boot
===================================================================
START - vmh
[ 0.201955] <inf> sof_boot_test: vmh_test: vmh init good
PASS - vmh in 0.001 seconds
===================================================================
TESTSUITE sof_boot succeeded

lgirdwood · 2023-11-29T15:57:26Z

Yes, logging, unfortunately I don't see any CI device results in this PR, so copying from my local testing (possibly of a modified version):
Running TESTSUITE sof_boot
===================================================================
START - vmh
[ 0.201955] <inf> sof_boot_test: vmh_test: vmh init good
PASS - vmh in 0.001 seconds
===================================================================
TESTSUITE sof_boot succeeded

I think we need to list each UT executed and its result as a 1st step (the total pass/fail can come later). Th point is that all tests should be visible and result recorded.

lyakh · 2023-11-30T08:51:01Z

Yes, logging, unfortunately I don't see any CI device results in this PR, so copying from my local testing (possibly of a modified version):
Running TESTSUITE sof_boot
===================================================================
START - vmh
[ 0.201955] <inf> sof_boot_test: vmh_test: vmh init good
PASS - vmh in 0.001 seconds
===================================================================
TESTSUITE sof_boot succeeded
I think we need to list each UT executed and its result as a 1st step (the total pass/fail can come later). Th point is that all tests should be visible and result recorded.

@lgirdwood you mean list them in the log? The above output is Zephyr's ZTEST() output, onle the "vmh init good" line is mine and it's actually not in this PR, so in fact imagine, that it isn't there - that's my local modification. But otherwise we want to use Zephyr standard testing, including their standard reporting, right? Of course, we can add any IPC reporting on top, in follow up PRs

kv2019i · 2023-12-01T14:24:01Z

@lyakh @lgirdwood I think the log output goes to logging subsystem as normal so it looks like this:
https://sof-ci.01.org/sofpr/PR7688/build423/devicetest/index.html?model=ADLP_RVP_NOCODEC-ipc4&testcase=check-playback-10sec

*** Booting Zephyr OS build zephyr-v3.5.0-2009-gefc32081893d ***
[ 0.034310] <inf> main: sof_app_main: SOF on intel_adsp_cavs25
[ 0.034321] <inf> main: sof_app_main: SOF initialized
[ 0.034770] <inf> ipc: ipc_cmd: rx	: 0x44000000|0x31400008
[ 0.035353] <inf> ipc: ipc_cmd: rx	: 0x44000000|0x3060004c
[ 0.037053] <inf> ipc: ipc_cmd: rx	: 0x11000005|0x0
[ 0.037066] <inf> pipe: pipeline_new: pipeline new pipe_id 0 priority 0
Running TESTSUITE sof_boot
===================================================================
TESTSUITE sof_boot succeeded
[ 0.037521] <inf> ipc: ipc_cmd: rx	: 0x40000004|0x15

I'm not sure what happens if the test fails, that should probably be tested. We need some means to observe what happens if the test fails. @lyakh ?

lyakh · 2023-12-01T15:00:03Z

I'm not sure what happens if the test fails, that should probably be tested. We need some means to observe what happens if the test fails. @lyakh ?

@kv2019i That would be a separate PR to sof-test? But I'm not sure if we rely on parsing the log, if not, we need a notification IPC and a debugfs entry in the kernel?

lyakh · 2023-12-01T15:08:17Z

I'm not sure what happens if the test fails

@kv2019i a failure looks like

Running TESTSUITE sof_boot
===================================================================
START - vmh
[ 0.488501] <wrn> sof_boot_test: vmh_test_single: Ignoring failure to allocate 1616 in non-contiguous mode
[ 0.488508] <err> sof_boot_test: vmh_test_single: Failed to allocate 26
[ 0.488630] <err> sof_boot_test: vmh_test: Non-contiguous test error -12
[ 0.488635] <err> sof_boot_test: sof_boot_vmh: vmh init failed: -12
FAIL - vmh in 0.001 seconds
===================================================================
TESTSUITE sof_boot failed.

lgirdwood · 2023-12-04T15:30:39Z

Yes, logging, unfortunately I don't see any CI device results in this PR, so copying from my local testing (possibly of a modified version):
Running TESTSUITE sof_boot
===================================================================
START - vmh
[ 0.201955] <inf> sof_boot_test: vmh_test: vmh init good
PASS - vmh in 0.001 seconds
===================================================================
TESTSUITE sof_boot succeeded
I think we need to list each UT executed and its result as a 1st step (the total pass/fail can come later). Th point is that all tests should be visible and result recorded.
@lgirdwood you mean list them in the log? The above output is Zephyr's ZTEST() output, onle the "vmh init good" line is mine and it's actually not in this PR, so in fact imagine, that it isn't there - that's my local modification. But otherwise we want to use Zephyr standard testing, including their standard reporting, right? Of course, we can add any IPC reporting on top, in follow up PRs

Yes, pls use ZTEST() API, but this needs to at least print the line number and file name otherwise its less helpful than using original log().

lyakh · 2023-12-05T07:22:26Z

Yes, logging, unfortunately I don't see any CI device results in this PR, so copying from my local testing (possibly of a modified version):
Running TESTSUITE sof_boot
===================================================================
START - vmh
[ 0.201955] <inf> sof_boot_test: vmh_test: vmh init good
PASS - vmh in 0.001 seconds
===================================================================
TESTSUITE sof_boot succeeded
I think we need to list each UT executed and its result as a 1st step (the total pass/fail can come later). Th point is that all tests should be visible and result recorded.
@lgirdwood you mean list them in the log? The above output is Zephyr's ZTEST() output, onle the "vmh init good" line is mine and it's actually not in this PR, so in fact imagine, that it isn't there - that's my local modification. But otherwise we want to use Zephyr standard testing, including their standard reporting, right? Of course, we can add any IPC reporting on top, in follow up PRs
Yes, pls use ZTEST() API, but this needs to at least print the line number and file name otherwise its less helpful than using original log().

@lgirdwood as you see above there's "vmh" in ZTEST() prints:

START - vmh
PASS - vmh in 0.001 seconds

that is the name of the test. We can agree to give more descriptive names, containing at least the file name, and if there are more than one tests in a file, then also an indication of which one this is. Would that be enough?

lyakh · 2023-12-05T07:27:57Z

+	} else {
+ ztest_test_pass();
+	}
+}


in fact, thinking about this now, this code would probably be common for all SOF boot tests, so we can have a macro to be called in this case something like SOF_ZTEST_BOOT(vmh, vmh_test); I can try to do this now or we can try to switch to one like this later when we get at least a second such boot test

lgirdwood · 2023-12-05T16:36:04Z

Yes, pls use ZTEST() API, but this needs to at least print the line number and file name otherwise its less helpful than using original log().
@lgirdwood as you see above there's "vmh" in ZTEST() prints:
START - vmh
PASS - vmh in 0.001 seconds
that is the name of the test. We can agree to give more descriptive names, containing at least the file name, and if there are more than one tests in a file, then also an indication of which one this is. Would that be enough?

As long as all tests are uniquely named with easy discovery then I'm good.

lgirdwood · 2023-12-05T16:39:42Z

@lyakh can you check CI. I assume this is disabled in default Kconfig as this would obviously fail a CI audio test.

marc-hb · 2023-12-05T19:29:54Z

+
+if (CONFIG_ACE_VERSION_1_5)
+ zephyr_library_sources_ifdef(CONFIG_SOF_BOOT_TEST
+ test/vmh.c


add_subdirectory(test/)

Add a ZTEST test-suite to run SOF boot-time self-tests at the time of the first FW_GEN_MSG IPC. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

This adds an initial Virtual Memory Heap test. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Add calls to vmh_alloc() and vmh_free() for testing. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

Test both cases - when spanning multiple blocks is allowed and when it isn't. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

lyakh · 2023-12-08T06:56:36Z

The only CI complaints are style - don't think using ret as a macro parameter can cause problems, and unrelated kernel error messages in main-ace

marc-hb · 2023-12-13T22:31:18Z

From the moment this PR enabled this test, it has always failed on MTL

[ 0.129560] <inf> ipc: ipc_cmd: rx	: 0x44000000|0x3060004c
[ 0.405513] <inf> ipc: ipc_cmd: rx	: 0x11000005|0x0
[ 0.405533] <inf> pipe: pipeline_new: pipeline new pipe_id 0 priority 0
Running TESTSUITE sof_boot
===================================================================
START - virtual_memory_heap
[ 0.505716] <wrn> sof_boot_test: vmh_test_single: Ignoring failure to allocate 1616 in non-contiguous mode
[ 0.505723] <err> sof_boot_test: vmh_test_single: Failed to allocate 26
[ 0.505845] <err> sof_boot_test: vmh_test: Non-contiguous test error -12
[ 0.505850] <err> sof_boot_test: sof_boot_virtual_memory_heap: virtual_memory_heap failed: -12
FAIL - virtual_memory_heap in 0.001 seconds
===================================================================
TESTSUITE sof_boot failed.
[ 0.606210] <inf> ipc: ipc_cmd: rx	: 0x40000004|0x15
[ 0.606253] <inf> dma: dma_get: dma_get() ID 0 sref = 1 busy channels 0
[ 0.607026] <inf> ipc: ipc_cmd: rx	: 0x40000006|0x10

(I think it always passed on cavs25)

We never noticed because we never paid attention to FW errors:

Catch firmware errors sof-test#1075

Now it's causing a DSP panic when combined with another change:

dai_zephyr: Silence benign warnings #8621

marc-hb · 2023-12-14T01:32:33Z

Temporarily disabling this test in:

debug_overlay.conf: temporarily disable SOF_BOOT_TEST #8624

lyakh mentioned this pull request May 25, 2023
memory management: Add virtual heap memory allocators #7276
Merged

lyakh force-pushed the testvmh branch from f0e8b25 to 09048a8Compare June 6, 2023 13:49

lyakh force-pushed the testvmh branch from 09048a8 to 0814a1cCompare June 13, 2023 13:46

lyakh force-pushed the testvmh branch 6 times, most recently from 2d2fc61 to 5a11244Compare July 27, 2023 10:33

lgirdwood reviewed Jul 31, 2023
View reviewed changes

lyakh force-pushed the testvmh branch from 5a11244 to 0c18896Compare November 16, 2023 17:00

lyakh changed the title ~~[DNM] boot test and virtual heap testing~~boot test and virtual heap testingNov 16, 2023

lyakh marked this pull request as ready for review November 16, 2023 17:00

lyakh requested review from abonislawski, dbaluta, iuliana-prodan, kv2019i, lbetlej, marc-hb, mmaka1 and plbossart as code owners November 16, 2023 17:00

marc-hb requested a review from dabekjakub November 16, 2023 19:02

marc-hb reviewed Nov 16, 2023
View reviewed changes

lyakh force-pushed the testvmh branch from 2a818ab to 1c28162Compare November 24, 2023 16:14

lyakh force-pushed the testvmh branch from 1c28162 to f20e491Compare November 27, 2023 08:59

lgirdwood reviewed Nov 27, 2023
View reviewed changes

lyakh commented Dec 5, 2023
View reviewed changes

kv2019i approved these changes Dec 5, 2023
View reviewed changes

lgirdwood approved these changes Dec 5, 2023
View reviewed changes

marc-hb reviewed Dec 5, 2023
View reviewed changes

lyakh added 4 commits December 6, 2023 17:46

test: add boot-time ztest-based testing
f6ed14c
Add a ZTEST test-suite to run SOF boot-time self-tests at the time of the first FW_GEN_MSG IPC. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

test: add a simple VMH test
3e2f073
This adds an initial Virtual Memory Heap test. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

test: extend VMH testing with an allocation
e94a998
Add calls to vmh_alloc() and vmh_free() for testing. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

vmh test: extend test for spanning
06350f3
Test both cases - when spanning multiple blocks is allowed and when it isn't. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>

lyakh force-pushed the testvmh branch from f20e491 to 06350f3Compare December 6, 2023 16:49

kv2019i mentioned this pull request Dec 7, 2023
west.yml: update Zephyr to efc32081893 #8582
Merged

kv2019i merged commit dba33cc into thesofproject:mainDec 8, 2023

lyakh deleted the testvmh branch December 8, 2023 14:15

marc-hb mentioned this pull request Dec 13, 2023
dai_zephyr: Silence benign warnings #8621
Closed

Uh oh!

Conversation

lyakh commented May 25, 2023

Uh oh!

lyakh commented Jun 6, 2023

Uh oh!

lyakh commented Jun 6, 2023

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

marc-hb left a comment

Choose a reason for hiding this comment

Uh oh!

lyakh commented Nov 17, 2023

Uh oh!

lyakh commented Nov 17, 2023

Uh oh!

lyakh commented Nov 17, 2023

Uh oh!

teburd commented Nov 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lyakh commented Nov 20, 2023

Uh oh!

lgirdwood commented Nov 24, 2023

Uh oh!

lgirdwood left a comment

Choose a reason for hiding this comment

Uh oh!

lgirdwoodNov 27, 2023

Choose a reason for hiding this comment

Uh oh!

lgirdwoodNov 27, 2023

Choose a reason for hiding this comment

Uh oh!

lyakh commented Nov 28, 2023

Uh oh!

lgirdwood commented Nov 29, 2023

Uh oh!

lyakh commented Nov 30, 2023

Uh oh!

kv2019i commented Dec 1, 2023

Uh oh!

lyakh commented Dec 1, 2023

Uh oh!

lyakh commented Dec 1, 2023

Uh oh!

lgirdwood commented Dec 4, 2023

Uh oh!

lyakh commented Dec 5, 2023

Uh oh!

lyakhDec 5, 2023

Choose a reason for hiding this comment

Uh oh!

lyakhDec 7, 2023

Choose a reason for hiding this comment

Uh oh!

lgirdwood commented Dec 5, 2023

Uh oh!

lgirdwood commented Dec 5, 2023

Uh oh!

marc-hbDec 5, 2023

Choose a reason for hiding this comment

Uh oh!

lyakhDec 7, 2023

Choose a reason for hiding this comment

Uh oh!

lyakh commented Dec 8, 2023

Uh oh!

marc-hb commented Dec 13, 2023

Uh oh!

marc-hb commented Dec 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

teburd commented Nov 17, 2023•
edited
Loading