perf(scroll): pace frames to the panel — 44→100 fps, stalls 14% → 0.02%

docs/SCROLL_PERFORMANCE.md

@@ -0,0 +1,266 @@
+# Scroll Performance
+
+How scrolling is paced on this hardware, what was wrong with it, and how to
+configure a plugin so its marquee is smooth.
+
+Measured on a Raspberry Pi 4 driving a 2×128×64 chain (256×64 logical) at
+`limit_refresh_rate_hz: 100`. Numbers below come from that panel.
+
+| | before | after |
+|---|---|---|
+| scroll frame rate | 44–46 fps | **100 fps, locked** |
+| frames ≥ 45 ms | 14–17% | none observed |
+| dominant frame time | 20 ms | **10 ms** |
+| disk cache write (~1 MB) | 14.8 ms | **5.4 ms** |
+
+---
+
+## The one rule that matters
+
+**Motion is smooth when the strip advances a whole number of pixels per panel
+refresh.**
+
+Advancing one pixel per refresh on a 100 Hz panel gives 100 px/s. Slower crisp
+speeds come from holding each frame for several refreshes -- 50 px/s is one
+pixel every second refresh -- which is covered under *Choosing a speed* below.
+A speed that lands on no such combination has to do one of two bad things:
+
+- **blend** two adjacent columns to render a half-step — on pixel-font text
+ this alternates crisp and smeared frames and reads as shimmer, or as the
+ text jumping a pixel ahead of itself;
+- **repeat** a frame — the strip stands still, then jumps, which reads as
+ judder.
+
+Neither is tunable away. Pick a speed that divides evenly.
+
+`src.common.scroll_config` solves this for you: `configure()` snaps a requested
+speed to the nearest one the panel can actually show in whole pixels, and
+`scripts/scroll_speeds.py` prints the full ladder for your hardware.
+
+## Choosing a speed
+
+The crisp speeds are not a fixed list -- they depend on how fast *your* panel
+refreshes, which depends on its size, `pwm_bits`, `gpio_slowdown` and the Pi
+model. A Pi Zero driving a long chain has a completely different set of good
+speeds from a Pi 4 driving a short one.
+
+```bash
+# what can this panel do? (reads your configured refresh rate)
+python3 scripts/scroll_speeds.py
+
+# what does it ACTUALLY manage, rather than what is configured?
+sudo systemctl stop ledmatrix
+sudo python3 scripts/scroll_speeds.py --measure
+sudo systemctl start ledmatrix
+
+# highlight the closest option to the speed you want
+python3 scripts/scroll_speeds.py --want 45
+
+# try one on the panel
+sudo systemctl stop ledmatrix
+sudo python3 scripts/scroll_speeds.py --demo 50
+sudo systemctl start ledmatrix
+```
+
+Sample ladder for a 100 Hz panel:
+
+```
+ 20.0 px/s (1px every 5 refreshes = 20.0 fps, slightly stepped)
+ 25.0 px/s (1px every 4 refreshes = 25.0 fps, slightly stepped)
+ 33.3 px/s (1px every 3 refreshes = 33.3 fps, smooth)
+ 50.0 px/s (1px every 2 refreshes = 50.0 fps, smooth)
+ 66.7 px/s (2px every 3 refreshes = 33.3 fps, smooth)
+ 100.0 px/s (1px every 1 refresh = 100.0 fps, smooth)
+```
+
+### How a slow speed stays crisp
+
+`SwapOnVSync(canvas, framerate_fraction)` holds each frame for N panel
+refreshes. **The panel keeps refreshing at its full rate either way**, so
+holding a frame costs nothing in flicker -- it only changes how often a *new*
+image is presented. That is what allows 50 px/s to be one whole pixel every
+second refresh, instead of half a pixel every refresh (which has no good
+rendering, only a choice between blur and judder).
+
+`scroll_config.configure()` snaps the requested speed to the nearest entry on
+the ladder and reports the hold that speed needs. It does **not** apply the
+hold: the hold belongs to a scroll, not to a plugin's lifetime, and plugins
+share one display manager -- one set at construction is reset the moment any
+other plugin finishes scrolling. Apply it yourself when the scroll starts:
+
+```python
+settings = scroll_config.configure(
+ self.scroll_helper,
+ plugin_config=self.config,
+ global_config=self.global_config,
+ display_manager=self.display_manager, # supplies the panel refresh rate
+)
+
+# ...then, each time this plugin begins scrolling:
+self.display_manager.set_scrolling_state(True, frame_hold=settings.frame_hold)
+```
+
+Passing `display_manager` only lets `configure` read the true refresh rate from
+`display.hardware`, which a plugin config cannot see. Skipping the
+`set_scrolling_state` call is the mistake that matters: the speed still
+resolves, but the panel keeps presenting a new frame every refresh, so a slow
+snapped speed falls back to fractional pixels. Pass `snap_to_crisp=False` to
+keep an exact requested speed and accept the artefacts.
+
+Speeds slower than about 20 px/s are stepped no matter what, because a 1-pixel
+advance at 20 fps is simply a coarse increment. That is the pixel pitch, not a
+software limit; the only way to move in smaller increments is sub-pixel
+blending, which this display does not tolerate (see above).
+
+## Configuring a plugin
+
+Use the shared resolver rather than reading config keys yourself:
+
+```python
+from src.common import scroll_config
+
+settings = scroll_config.configure(
+ self.scroll_helper,
+ plugin_config=self.config,
+ global_config=self.global_config,
+ refresh_hz=scroll_config.refresh_hz_from_config(self.global_config),
+ plugin_logger=self.logger,
+)
+```
+
+It resolves every config shape in one place, applies the speed, and returns
+what it did. Precedence, highest first:
+
+1. `display_options.scroll_speed` + `scroll_delay` — **the recommended form**
+2. `display.scroll_speed` + `scroll_delay` — deprecated shape
+3. `scroll_speed` + `scroll_delay` at the root — legacy flat
+4. `scroll_pixels_per_second` — deprecated
+5. the global `display` block
+6. the built-in default (100 px/s)
+
+`scroll_speed` is pixels per frame and `scroll_delay` is the frame period in
+seconds, so the pair means `scroll_speed / scroll_delay` px/s. The recommended
+config for a 100 Hz panel:
+
+```json
+"display_options": { "scroll_speed": 1.0, "scroll_delay": 0.01 }
+```
+
+### Why the deprecated key ranks below the explicit pair
+
+Because some plugins give `scroll_pixels_per_second` a **schema default**, and
+schema defaults are merged into plugin config. Ranking it above the pair means
+it is always present and always wins, so the documented settings become
+unreachable. That is a real, shipped bug — see
+[ledmatrix-plugins#408](https://github.com/ChuckBuilds/ledmatrix-plugins/issues/408).
+
+If you are writing a plugin: do not give a deprecated key a schema default.
+
+## What was actually wrong
+
+Four independent faults, each found by measurement.
+
+### 1. The frame loop slept on top of a wait it had already done
+
+`display_controller.py` ran the high-FPS loop as `render → SwapOnVSync (blocks
+to the panel's refresh) → time.sleep(0.008) → plugin ticks`. The sleep was
+unconditional and added to a wait that had already happened. Render work
+measured ~4 ms, so each iteration cost ~12 ms against a 10 ms refresh grid —
+every swap missed a refresh and landed on the next one. The loop settled at
+exactly 50 fps while asking for 125, with no headroom, so ~14% of frames
+slipped a further refresh.
+
+Now the loop sleeps only the remainder of the frame budget, with a 1 ms floor
+so plugin threads still get the GIL.
+
+### 2. `SwapOnVSync` held the GIL while blocking
+
+The rgbmatrix binding declares it without `nogil` (unlike `SetPixel`, `Clear`
+and `Fill` immediately above it in `cppinc.pxd`), so the render thread held the
+GIL for the entire vsync wait — most of every frame. Background threads were
+starved into long uninterruptible bursts; a 1.5 MB API response costs ~17 ms to
+parse and ~18 ms to re-encode for the cache, and `json.raw_decode` cannot be
+preempted mid-document. Those bursts are what the render loop then waited on.
+
+Fixed by rebuilding the binding: `scripts/build_rgbmatrix_nogil.sh`.
+
+### 3. Sub-pixel blending was wrong for this display
+
+Enabling it made things worse, not better — see the rule at the top. It is off
+by default and only Vegas mode opts in via `set_sub_pixel_scrolling(True)`.
+
+### 4. Frame-based stepping raced the vsync clock
+
+Frame-based mode gated motion on a wall clock at `1/scroll_delay` steps per
+second. Plugins set `scroll_delay` to the frame period, which puts that
+comparison exactly on its own threshold: a frame arriving a hair early moved
+zero pixels and rendered an identical frame, which dirty-tracking skipped, so
+it returned in ~2 ms and the beat repeated. No `scroll_delay` value tunes this
+out — a shorter delay just trades stalled frames for periodic double-steps.
+
+`ScrollHelper` now accumulates elapsed time in both modes at the same
+configured speed, so position stays proportional to real time.
+
+## Diagnosing a juddery scroller
+
+**`Avg FPS` will lie to you.** It is a 100-frame moving average, and a 2 ms
+duplicate frame plus a 21 ms double-wait average to exactly 10 ms. A ticker
+that is stalling on half its frames still reports a healthy `100.0`.
+
+Look at the **distribution** instead:
+
+```bash
+journalctl -u ledmatrix --since "-10min" --no-pager \
+ | grep -oE "Frame time: [0-9.]+ms" | awk '{print $3}' | sed 's/ms//' \
+ | awk '{printf "%.0f\n", $1}' | sort -n | uniq -c
+```
+
+Reading it, on a 100 Hz panel:
+
+| you see | it means |
+|---|---|
+| everything at 10 ms | healthy |
+| a mode at ~2 ms | **duplicate frames** — the swap was skipped because the image did not change. The scroller is advancing less than one pixel per frame. |
+| a mode at 20/30/50 ms | frames missing refreshes — per-frame work is overrunning, or a background thread is holding the GIL |
+| `Avg FPS` above 100 | duplicates present, unless the scroll cycle has completed and is idling |
+
+Then confirm what the plugin actually loaded — config edits do not always reach
+the running code:
+
+```bash
+journalctl -u ledmatrix --since "-5min" --no-pager | grep -iE "px/s|px/frame"
+```
+
+If a plugin logs its scroll config **twice** with different modes, the second
+line is what is running.
+
+## Rebuilding the binding
+
+```bash
+bash scripts/build_rgbmatrix_nogil.sh # build into a scratch dir
+sudo bash scripts/build_rgbmatrix_nogil.sh --install
+sudo bash scripts/build_rgbmatrix_nogil.sh --rollback
+```
+
+The build never touches the installed module. `--install` backs up the original
+to `~/rgbmatrix-core.so.ORIGINAL` first, and rolls back automatically if the
+service does not come back healthy. Requires `build-essential`; Cython is
+installed into a cached venv under `~/.cache/ledmatrix-cython`.
+
+Re-run it after upgrading `rpi-rgb-led-matrix`, since a library upgrade
+replaces the patched binding.
+
+## Faster JSON
+
+`src/cache/disk_cache.py` uses `orjson` when it is importable and falls back to
+the stdlib otherwise, so it is optional:
+
+```bash
+sudo pip3 install --break-system-packages orjson
+```
+
+Encoding is where it pays — about 7× on this hardware. Decoding gains far less
+(~1.3× on large payloads) because the cost there is building Python objects,
+not scanning text. That is also why moving parsing to a subprocess does not
+help: `pickle.loads` of the same payload costs 8.1 ms against `json.loads` at
+10.9 ms, so the work just moves rather than disappearing.

requirements.txt

@@ -55,6 +55,18 @@ packaging>=23.0,<27.0
 # range as a hard dependency — keep the two in sync.
 # pip install 'psutil>=6.0.0,<7.0.0'
 #
+# orjson — faster JSON for the disk cache
+# (src/cache/disk_cache.py). Encoding a ~1MB cache
+# record drops from ~12ms to ~1.6ms on a Pi 4, which
+# matters because that work holds the GIL and stalls
+# the render thread mid-scroll. Falls back to the
+# stdlib json when missing — see docs/SCROLL_PERFORMANCE.md.
+# The 3.11.6 floor is CVE-2025-67221: orjson.dumps did not
+# limit recursion on deeply nested documents, and the disk
+# cache encodes payloads parsed straight from third-party
+# APIs. 3.11.6 covers the Python range above.
+# pip install 'orjson>=3.11.6,<4.0'
+#
 # Flask-Limiter — request rate limiting in web_interface/app.py
 # (accidental-abuse protection, not security). The
 # web interface starts without rate limiting when