Skip to content

Stop test-harness artefacts from losing LAVA results - #554

Draft
Ricardo Salveti (ricardosalveti) wants to merge 5 commits into
qualcomm-linux:mainfrom
ricardosalveti:claude-findings
Draft

Stop test-harness artefacts from losing LAVA results#554
Ricardo Salveti (ricardosalveti) wants to merge 5 commits into
qualcomm-linux:mainfrom
ricardosalveti:claude-findings

Conversation

@ricardosalveti

@ricardosalvetiRicardo Salveti (ricardosalveti) commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Analysis of all 12,658 meta-qcom LAVA jobs run between 19 and 27 Aug 2026
found that a large share of the red in CI is produced by the harness rather
than by the platforms under test: 62 of 287 recorded test-case failures had
no RESULT=FAIL in the log at all, and 616 jobs died without reporting any
result. This series fixes the six causes that are ours to fix.

  • send-to-lava: kernel printk splices into testcase signals because the
    quiet window closes before the line has left the UART. 36 of 200 sampled
    jobs were affected; cdsp_remoteproc lost 15 of its 51 failures this way.
  • lib_display: DRM hot-plug detection is sampled once, so boards that do
    have a panel skip 18-21% of the time (hamoa-iot-evk-04, lemans-hyd-01).
  • remoteproc: the boot check reads a snapshot taken moments after login,
    failing instances that are still coming up (cdsp1 on sa8775p, 9-21%).
  • AudioRecord: the capture watchdog does not allow for PipeWire start-up,
    truncating recordings below the duration gate.
  • lib_bluetooth: a duplicated 75s poll and a bootstrap that cannot succeed
    on an all-zero BD address burn the 300s bt budget, killing 66 whole jobs.
  • lib_gstreamer: a pipeline blocked in a V4L2 ioctl on a wedged iris codec
    ignores SIGINT and outlives its watchdog, killing 35 gstr-video jobs.

Each commit carries the measured failure rates it addresses. No test verdicts
change: every fix either lets a passing test be read, or makes a failing test
report as a failure instead of taking the job down with it.

All touched scripts pass shellcheck and sh -n.

@ricardosalveti
send-to-lava: drain the console before restoring printk
0e30657
LAVA parses testcase signals from the serial console, which also
carries kernel printk. The script already quiets printk around the
signal, but printf returns once the line is in the tty buffer, not once
the UART has sent it, so the loglevel is restored while the signal is
still on the wire and a message emitted in that gap splits it:
<<<LAVA_SIGNAL_TESTCASE TEST_CASE_ID=cdsp_remoteproc RESU[ 69.2]
Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
LAVA cannot parse that, so a test that reported PASS is recorded as a
failure. Over 12658 meta-qcom jobs run between 19 and 27 Aug 2026, 62
of 287 recorded test-case failures had no clean RESULT=FAIL in the log
at all, and 36 of 200 sampled jobs carried a spliced signal.
Hold the quiet window across the whole emission for one result file and
sleep either side of it, so kernel output already queued on the UART
finishes before the first printf and the signals themselves reach the
link before printk resumes. Make the delay LAVA_SIGNAL_DRAIN_SECONDS,
default 0.3s, with a whole-second fallback for shells without
fractional sleep.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
@ricardosalveti
lib_display: wait for a connected DRM connector before skipping
12fe313
display_log_snapshot_and_require_connector() reads /sys/class/drm once
and skips the test when it sees no connected connector. Hot-plug
detection is asynchronous, so a board that does have a panel can still
read disconnected when the first test of a job looks, seconds after
boot.
The pre-merge display plan only runs on boards tagged display, yet
between 19 and 27 Aug 2026 those boards skipped anyway and not
consistently: hamoa-iot-evk-04 42 of 196 runs, lemans-hyd-01 36 of 197,
kaanapali-mtp-06 1 of 12. A board either has a display or it does not,
so a rate between 0 and 100 percent is a race, and every skip is silent.
Poll for a connector instead of sampling once, on the monotonic clock so
an NTP step landing mid-job cannot end the wait early. Bound it with
DISPLAY_CONNECTOR_WAIT, default 20s, polled every
DISPLAY_CONNECTOR_POLL, default 2s. Boards with nothing attached reach
the same skip, only later.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
@ricardosalveti
remoteproc: wait for an instance to reach running before failing
db2230e
The boot check compares the state captured when the instance list was
built against running and fails on anything else. That snapshot is taken
moments after login, while remote processors are still coming up, so an
instance that boots a second later is recorded as a failure. On
lemans-evk it is the second CDSP that loses the race, reported offline
while remoteproc2 passes.
Between 19 and 27 Aug 2026 cdsp_remoteproc failed this way on every
board carrying more than one instance, at rates that rule out a
processor that never boots: kaanapali-mtp-06 8 of 17, lemans-02 9 of 89,
rb8-hyd-06 4 of 36. adsp_remoteproc shows the same pattern.
Re-read the state through the existing wait_remoteproc_state() before
deciding, so only a processor that never reaches running is reported.
Bound the wait with BOOT_TO or --boot-to, default 30s, 0 to keep the
previous single-shot behaviour, and apply it to all four remote
processor suites, which share the check verbatim. Measure the wait with
get_monotonic_seconds() rather than date, so an NTP step cannot cut it
short.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
@ricardosalveti
AudioRecord: give the capture watchdog start-up headroom
21fb80b
The capture is bounded by an external watchdog set to the requested
duration. pw-record has no duration flag on this path, so the watchdog
is what ends the recording, and its clock starts before PipeWire has
opened the device. Whatever the graph spends coming up is taken out of
the recording rather than added to it:
[record_16KHz_2ch] loop 1 FAILED (rc=124, 6s, ...
reason=duration-too-short:0.724<3.500)
The recorder never failed, it was stopped before it had captured enough.
Between 19 and 27 Aug 2026 this cost iq-x7181-evk 19 to 29 failures per
AudioRecord config out of roughly 207 runs each.
Run the watchdog for the requested duration plus
AUDIO_RECORD_START_GRACE, default 5s, so a slow start eats the headroom
instead of the recording. The duration validated against is unchanged,
an explicit --timeout still overrides the watchdog, and the ALSA paths
already pass -d to arecord so there the grace only widens the outer
watchdog.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
@ricardosalveti
lib_gstreamer: escalate the pipeline watchdog to SIGKILL
e98feeb
gstreamer_run_gstlaunch_timeout() bounds gst-launch with timeout
--signal=INT, chosen so -e can turn the signal into EOS and finalise the
container. Nothing escalates behind it, so when the pipeline cannot act
on a signal the watchdog expires and then waits with it.
That is what a wedged video codec does. On iq-x7181-evk the encode
pipeline is bounded at 40 seconds and the codec fails four seconds in
with "qcom-iris aa00000.video-codec: received system error of type
0x5000003". gst-launch is blocked in a V4L2 ioctl, never returns to
userspace to take the SIGINT, and the job runs until lava-test-shell
kills it at 600 seconds. Between 19 and 27 Aug 2026 that lost 35
gstr-video jobs, with no test-level record that the codec had failed.
Pass --kill-after so the watchdog escalates to SIGKILL, keeping SIGINT
first so a healthy pipeline still finalises its output. Make the grace
GSTLAUNCH_KILL_AFTER, default 10s, probe for the option since some
busybox builds lack it, and warn when running without it. The killed
pipeline returns 137, which every caller already treats as a failure.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Ricardo Salveti <ricardo.salveti@oss.qualcomm.com>
@ricardosalveti

Ricardo Salveti (ricardosalveti) commented Aug 27, 2026

Copy link
Copy Markdown
ContributorAuthor

This is the outcome of an investigation of all the lava jobs executed over the past 8 days.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ricardosalveti