Preview snapshot permanently fails when a stale /tmp/led_matrix_preview.png.tmp is owned by the other service's user (fs.protected_regular blocks even root)

[ChuckBuilds]

Summary

The display preview snapshot writes through a fixed temp path, /tmp/led_matrix_preview.png.tmp. /tmp is world-writable and sticky, and the display service (ledmatrix, User=root) and the web service (ledmatrix-web, User=hdpi) are different users. Once a stale .tmp owned by the other user is left behind, fs.protected_regular makes every subsequent write fail with EACCES — even for root — and nothing ever clears it.

On this rig the preview has been dead for 22.9 hours. The display itself is running fine.

Symptoms

$ journalctl -u ledmatrix | grep Snapshot
17:18:18 WARNING src.display_manager - Snapshot write failing (web preview/health mirror is stale): [Errno 13] Permission denied: '/tmp/led_matrix_preview.png.tmp'
17:23:18 WARNING (same)
17:28:18 WARNING (same)
...every 5 minutes, indefinitely

$ ls -la /tmp/led_matrix_preview*
-rw-rw-r-- 1 root root 10723 Sep 3 18:53 /tmp/led_matrix_preview.png
-rw-rw-r-- 1 hdpi hdpi 92 Sep 4 16:32 /tmp/led_matrix_preview.png.tmp <-- stale, wrong owner
-rw-r--r-- 1 hdpi hdpi 0 Sep 4 17:48 /tmp/led_matrix_preview_viewer
$ ls -ld /tmp
drwxrwxrwt 20 root root 2260 Sep 4 17:48 /tmp

/api/v3/health consequently reports the hardware as stale on a perfectly healthy rig:

"hardware": { "snapshot_age_seconds": 82484.0, "status": "stale" }

Root cause — it is not ordinary permissions

Root normally bypasses DAC, so EACCES for root looks impossible. It's fs.protected_regular, which refuses an O_CREAT open of an existing file in a world-writable sticky directory when the file's owner is neither the opener nor the directory owner. Demonstrated directly on the rig:

$ sudo python3 -c "open('/tmp/led_matrix_preview.png.tmp','wb').write(b'x')"
root FAILED: [Errno 13] Permission denied: '/tmp/led_matrix_preview.png.tmp'

So the atomic write-then-rename is safe against torn reads but not against a cross-user stale temp file in a shared sticky directory.

Why it never recovers

display_manager catches the error, logs it (throttled to every 5 minutes) and carries on. Nothing removes or works around the offending file, so the only fix is a human running rm /tmp/led_matrix_preview.png.tmp. The comment at display_manager.py:227 says the failure "is never silent" — true, but "never silent" and "never recovers" are different things, and the log line doesn't say what to delete.

Knock-on effect on the health check

src/common/snapshot_policy.py is explicit that the file's mtime is a liveness proxy:

"The health check ... uses the file's AGE as a liveness proxy: age >= 60s reads as degraded."
"TOUCH_INTERVAL must stay well under it."

TOUCH_INTERVAL is 20s and IDLE_INTERVAL is 30s, so the policy is sound — but both the WRITE and the TOUCH go through the same blocked path, so the liveness proxy inverts: a healthy, rendering display reports stale. Anyone using /api/v3/health for monitoring gets a permanent false alarm.

Suggested fix

Any one of these closes it; the first two are the smallest:

1. Unique temp name — tempfile.mkstemp(dir="/tmp", prefix="led_matrix_preview.", suffix=".png") then os.replace(). A stale file from another user can never collide, because the name is never reused.
2. Unlink-on-EACCES retry — try to remove the temp path and retry once before giving up. Note this fails too under protected_regular if the directory isn't owned by the caller, so (1) is more robust.
3. Move the mirror out of /tmp — a service-owned directory such as /var/cache/ledmatrix/ (already used for other state) has neither the sticky-bit hazard nor the cross-user ambiguity.

Worth also including the offending path in the warning so the manual fix is obvious:

Snapshot write failing (web preview/health mirror is stale): ...
 -> a stale /tmp/led_matrix_preview.png.tmp owned by another user blocks this; remove it

Environment

LEDMatrix v3.3.0-4-g0730d952, Raspberry Pi (Debian trixie, kernel 6.18.34), 256x64 panel.
ledmatrix.service runs as root, ledmatrix-web.service as hdpi. Found while validating plugins — I tried to capture the panel and found the preview 23 hours old.

[ChuckBuilds]

Confirmed the diagnosis by clearing the stale file on the rig:

$ rm -f /tmp/led_matrix_preview.png.tmp
$ sleep 35 && ls -l /tmp/led_matrix_preview.png
-rw-rw-r-- 1 root root 8387 2026-09-04 17:53:21 /tmp/led_matrix_preview.png # written again, by root
$ curl -s localhost:5000/api/v3/health
hardware: {'snapshot_age_seconds': 9.5, 'status': 'connected'}

Snapshot writes resumed immediately and the hardware check went from stale (82484s) to connected (9.5s) with no service restart — so the stale temp file was the only thing blocking it.

Worth noting the overall health status stayed degraded after this fix, for the unrelated reason in #529.

[ChuckBuilds]

Found the producer — this is reproducible on demand, and the trigger is running the plugin test suite.

Reproduction

plugins/hockey-scoreboard/test_hockey_emulator.py constructs a real DisplayManager. Run as a normal user on a rig whose display service runs as root:

$ rm -f /tmp/led_matrix_preview.png.tmp # start clean
$ ls /tmp/led_matrix_preview.png.tmp ; echo "absent"
absent

$ cd ~/LEDMatrix
$ PYTHONPATH=$PWD LEDMATRIX_CORE=$PWD python3 plugin-repos/hockey-scoreboard/test_hockey_emulator.py
 (rc=0, test passes)

$ ls -la /tmp/led_matrix_preview.png.tmp
-rw-rw-r-- 1 hdpi hdpi 128 Sep 4 19:40 /tmp/led_matrix_preview.png.tmp

The leftover is a truncated but valid PNG header — the snapshot write that never completed:

$ file /tmp/led_matrix_preview.png.tmp
/tmp/led_matrix_preview.png.tmp: PNG image data, 256 x 64, 8-bit/color RGB, non-interlaced
$ head -c 16 /tmp/led_matrix_preview.png.tmp | od -c
0000000 211 P N G \r \n 032 \n \0 \0 \0 \r I H D R

From that moment the root display service is locked out permanently:

$ sudo python3 -c "open('/tmp/led_matrix_preview.png.tmp','wb').write(b'x')"
root BLOCKED: [Errno 13] Permission denied: '/tmp/led_matrix_preview.png.tmp'

$ journalctl -u ledmatrix --since -1min | grep -c "Snapshot write failing"
1

and rm-ing the file restores it within one cycle (preview mtime updates immediately).

Why this matters more than a one-off stale file

The size and ownership match exactly what I found in the original report (92 bytes at 16:32, 128 bytes at 18:58 — both hdpi-owned). Both appeared during plugin-test runs. So this isn't a rare accident: running the plugin test suite as a normal user silently and permanently kills the live display's preview and health signal, and the only recovery is a manual rm that nobody would know to do.

It recurred on this rig within the same session after I cleared it once, which is how I noticed.

Ruled out along the way

For anyone retracing this: it is not the web service (ledmatrix-web runs as hdpi but has no DisplayManager — zero display_manager lines in its journal and no matching open fds), not calendar/test_display_does_not_blank_the_panel.py, and not baseball-scoreboard/test_baseball_plugin.py (that one imports the real DisplayManager too but leaves nothing behind). src/display_manager.py is the only writer of the .tmp path in the tree.

Bearing on the fix

This strengthens the case for the unique-temp-name option in the original report, since a colliding fixed name is what turns a harmless test artifact into a permanent outage of another process. Two further points:

• A DisplayManager constructed inside a test arguably shouldn't be mirroring frames to the shared production path at all — an env guard (LEDMATRIX_NO_SNAPSHOT=1, or skipping when sys.stdout isn't a tty) would stop tests touching live state regardless of the naming fix.
• The write should clean up after itself on failure. Leaving a partial file behind is what makes this permanent rather than transient.

Environment unchanged from the original report: LEDMatrix v3.3.0-4-g0730d952, Debian trixie, 256x64, ledmatrix.service as root and ledmatrix-web.service as hdpi.

[ChuckBuilds]

Possibly relevant history: #202 (closed 2026-01-27) was the inverse of the precondition here — LEDMatrix was leaving /tmp as drwxrwsr-x, i.e. without the sticky bit, which broke apt update.

/tmp on this rig is now correct:

$ stat -c "%a %A" /tmp
1777 drwxrwxrwt

That sticky bit is exactly what makes fs.protected_regular refuse the cross-user open described above. So the failure mode in this issue is only reachable on a system where /tmp is set up properly — which may be why it surfaces now and didn't before #202 was fixed.

Not suggesting anything should change about #202's fix; noting it only because "it worked before" is a reasonable thing to wonder here, and the answer may be that it worked before for the wrong reason.